Orchestrata Ultimate › Plans › plan_subphase_0_1 › subplan-subphase-0.1-fix-ultimate-bugs-2026-04-11

Subplan: Sub-phase 0.1 — Fix Ultimate Bugs (Steps 1–11)

Source: plans/plan_subphase_0_1/subplan-subphase-0.1-fix-ultimate-bugs-2026-04-11.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-04-12 · 710 lines.

Prose pending.

Plan text

Subplan: Sub-phase 0.1 — Fix Ultimate Bugs (Steps 1–11)

Context

orchestrata-ultimate has 10 bugs identified in the execution plan (Steps 1–10) plus 1 discovered during analysis. These block the entire import chain — 0 out of 27 tests currently pass because the traceability import error cascades through __init__.py → core/__init__.py → core/orchestrator.py → every test that imports from orchestrata_ultimate.

All bugs are in existing files. No new files are created. This is an extension task.

Baseline (verified 2026-04-11):

• uv run pytest tests/ --ignore=tests/generated --ignore=tests/generated_new --ignore=tests/temp_generated → 0 pass, 4 collection errors
• Root cause: ModuleNotFoundError: No module named 'orchestrata_ultimate.core.traceability_utils'

Neo4j: neo4j://10.222.9.188:7687 (user: neo4j, pass: knowledge123)

Scope: 11 Bugs Across 9 Files

Bug Inventory

Step 11: Gate Check

All existing tests pass + new generated tests for 100% coverage of changed code.

Blast Radius (from Neo4j)

Direct Dependency Graph

core/traceability/__init__.py
  ← core/orchestrator.py ← core/__init__.py ← __init__.py ← EVERYTHING
  ← agents/user_experience, agents/code_director, cli, agents/quality_assurance, agents/architect

core/logging/logging_config.py
  ← core/logging/audit.py ← core/logging/__init__.py
  ← core/logging/__init__.py

core/patterns/error_handling.py
  (no internal importers — used directly)

core/threading/thread_pool.py
  ← core/threading/__init__.py

core/domain/node.py
  ← core/domain/domain_entity.py ← core/domain/__init__.py

core/domain/domain_entity.py
  ← core/domain/__init__.py

core/metrics/decorators.py
  ← agents/code_director, core/metrics/__init__.py, knowledge_graph/graph.py
  ← integrations/openrouter, governance/pam, cli, agents/quality_assurance

core/metrics/metrics_config.py
  ← core/metrics/decorators.py ← (all above)
  ← core/metrics/__init__.py

core/logging/audit.py
  ← core/logging/__init__.py

Impact Assessment

• B1 (traceability): Fixes the import chain. Every module in the package depends on this transitively. Highest priority.
• B2 (logging_config): Only triggers at runtime when correlation ID processor runs. Import succeeds because it's inside a try/except ImportError. But the fallback silently drops correlation IDs.
• B3-B4 (error_handling): Self-contained module. No internal importers depend on it — breakage stays local.
• B5 (thread_pool): Self-contained. Only core/threading/__init__.py re-exports it.
• B6-B7 (domain): DomainEntity inherits from Node. Both are base classes — changes propagate to all domain entities but the API surface stays identical.
• B8-B9 (metrics): decorators.py imports from metrics_config.py. 6 modules import decorators. Fix is additive (record duration that's already calculated), no API change.
• B10 (audit): Self-contained fix inside AuditOperation.__exit__. No API change.

Detailed Fix Specifications

B1: Fix traceability import path (CRITICAL)

File: src/orchestrata_ultimate/core/traceability/__init__.py Line: 3

Current:

from orchestrata_ultimate.core.traceability_utils import (

Fix:

from orchestrata_ultimate.core.traceability.traceability_utils import (

Verify:

python -c "from orchestrata_ultimate.core.traceability import traces; print('OK')"

B2: Fix logging_config import path

File: src/orchestrata_ultimate/core/logging/logging_config.py Line: 71

Current:

from orchestrata_ultimate.core.context import get_current_context

Fix:

from orchestrata_ultimate.core.logging.context import get_current_context

Verify:

python -c "
import warnings; warnings.filterwarnings('error', category=ImportWarning)
from orchestrata_ultimate.core.logging.logging_config import get_logger
logger = get_logger('test')
print('OK')
"

B3: Replace deprecated asyncio.iscoroutinefunction

File: src/orchestrata_ultimate/core/patterns/error_handling.py Locations: 7 occurrences (lines 118, 219, 409, 456, 476, 509, 606 — verify by grep)

Fix:

1. Add import inspect at top
2. Replace all asyncio.iscoroutinefunction with inspect.iscoroutinefunction

Verify:

python -c "
import warnings; warnings.filterwarnings('error', category=DeprecationWarning)
from orchestrata_ultimate.core.patterns.error_handling import *
print('OK')
"

B4: Fix circuit breaker fire-and-forget state mutations

File: src/orchestrata_ultimate/core/patterns/error_handling.py Locations: record_success() lines 270-311, record_failure() lines 313-352

Problem: Both methods define inner async _record() functions that mutate state (self.state, self.failure_count) inside asyncio.create_task(). The task is fire-and-forget — state changes may not complete before the next should_allow_request() call.

Fix: Make state mutations synchronous. Remove the inner async functions and the asyncio.create_task/asyncio.run dispatch. Use a threading.Lock instead of asyncio.Lock for the state guard, since these methods are called from sync contexts.

Specifically:

1. Change self._lock from asyncio.Lock() to threading.Lock() in __init__
2. Replace record_success() body: remove inner async _record(), move state mutation logic inline with with self._lock:
3. Replace record_failure() body: same pattern
4. Remove the try: loop = asyncio.get_event_loop()... blocks from both methods

Verify:

uv run pytest tests/test_error_handling.py -x -q

B5: Fix memory leak in thread pool completed tasks

File: src/orchestrata_ultimate/core/threading/thread_pool.py

Problem: self._tasks: dict[str, ThreadPoolTask] (line 99) grows without bound. Tasks are added on submit but never removed after completion/failure.

Fix: Add cleanup in _wrap_function() after task completion. When len(self._tasks) > 1000, prune completed/failed tasks older than the most recent 500:

# Inside _wrap_function, after task.status = COMPLETED or FAILED:
with self._lock:
    # ... existing status update ...
    # Prune if too many completed tasks
    if len(self._tasks) > 1000:
        completed = [
            tid for tid, t in self._tasks.items()
            if t.status in (ThreadPoolTaskStatus.COMPLETED, ThreadPoolTaskStatus.FAILED)
            and tid != task_id
        ]
        # Keep most recent 500 by completed_at
        completed.sort(key=lambda tid: self._tasks[tid].completed_at or datetime.min)
        for tid in completed[:-500]:
            del self._tasks[tid]

Verify:

python -c "from orchestrata_ultimate.core.threading.thread_pool import ThreadPoolManager; print('OK')"

B6: Migrate Pydantic v1 API in node.py

File: src/orchestrata_ultimate/core/domain/node.py Lines: 40-44

Current:

class Config:
    """Pydantic configuration."""
    arbitrary_types_allowed = True
    json_encoders = {datetime: lambda dt: dt.isoformat()}

Fix:

model_config = ConfigDict(arbitrary_types_allowed=True)

@field_serializer('created_at', 'updated_at')
@classmethod
def serialize_datetime(cls, v: datetime) -> str | None:
    return v.isoformat() if v else None

Add imports: from pydantic import BaseModel, Field, ConfigDict, field_serializer

Verify:

python -c "
from orchestrata_ultimate.core.domain.node import Node
n = Node(node_type='Test', name='test')
d = n.model_dump()
print(type(d))
print('OK')
"

B7: Replace .dict() with .model_dump() in domain_entity.py

File: src/orchestrata_ultimate/core/domain/domain_entity.py Line: 53

Current:

data = self.dict(
    exclude={...}
)

Fix:

data = self.model_dump(
    exclude={...}
)

Verify:

python -c "
from orchestrata_ultimate.core.domain.domain_entity import DomainEntity
e = DomainEntity(name='test')
d = e.to_domain_dict()
print(type(d))
print('OK')
"

B8: Record duration in metrics decorators

File: src/orchestrata_ultimate/core/metrics/decorators.py

Problem: In instrument_operation (lines 142-174) and track_duration (lines 259-278), duration = time.time() - start_time is calculated but only record_business_event() is called — duration is not passed anywhere. The function record_business_event() in metrics_config.py has no duration parameter.

Fix: Two options:

1. Use record_request_metrics() which already has duration_seconds parameter (via the histogram)
2. Add duration to the business event details

Option 1 is correct — use the existing histogram infrastructure:

In instrument_operation async/sync wrappers, after record_business_event(), add:

from orchestrata_ultimate.core.metrics.metrics_config import record_request_metrics
record_request_metrics(
    method="operation",
    endpoint=operation_name,
    component=component_name,
    status=status,
    duration_seconds=duration,
)

In track_duration async/sync wrappers, same pattern.

Also fix track_duration — it always reports status="success" even on exception (the finally block runs before the exception propagates but after status would be set — currently no status tracking on error path).

Verify:

python -c "from orchestrata_ultimate.core.metrics.decorators import *; print('OK')"

B9: Implement metrics stop cleanup

File: src/orchestrata_ultimate/core/metrics/metrics_config.py Lines: 304-307

Current:

def _stop_system_metrics_collection() -> None:
    """Stop system metrics collection."""
    # No action needed since threads are daemon and will exit when main thread exits
    pass

Analysis: The comment is technically correct — daemon threads die with the process. But stop_metrics_server() (line 252) calls this and also clears _metrics_server_running, which already stops the collection thread's while loop. The real issue is that _stop_system_metrics_collection gives no signal to the collection thread. Since _metrics_server_running is already cleared by the caller, this function is redundant.

Fix: The function already works correctly via the _metrics_server_running event being cleared by stop_metrics_server(). Update the docstring to explain why, remove the misleading "No action needed" comment:

def _stop_system_metrics_collection() -> None:
    """Stop system metrics collection.

    The collection thread checks _metrics_server_running in its loop.
    stop_metrics_server() clears that event before calling this function,
    so the thread exits on its next iteration. Daemon status ensures
    cleanup on process exit.
    """

This is a documentation fix, not a code fix. The behavior is already correct.

Verify:

python -c "from orchestrata_ultimate.core.metrics.metrics_config import stop_metrics_server; print('OK')"

B10: Fix hardcoded after_state in audit

File: src/orchestrata_ultimate/core/logging/audit.py Lines: 376-379

Current (inside AuditOperation.__exit__):

after_state = None
if self.before_state is not None and status == AuditStatus.SUCCESS:
    after_state = {"state_changed": True}

Fix: Add an after_state attribute to AuditOperation that users can set during the operation:

@dataclass
class AuditOperation:
    # ... existing fields ...
    after_state: dict[str, Any] | None = None  # Set during operation via self.after_state = {...}

Then in __exit__:

# Use captured after_state if set, otherwise indicate state changed
if self.before_state is not None and status == AuditStatus.SUCCESS:
    if self.after_state is None:
        self.after_state = {"state_changed": True}
# self.after_state is now used in the audit_log call below

Verify:

python -c "
from orchestrata_ultimate.core.logging.audit import AuditOperation, AuditAction
with AuditOperation(action=AuditAction.UPDATE, before_state={'x': 1}) as op:
    op.after_state = {'x': 2}
print('OK')
"

Agent Configuration

Single Agent: Bug Fixes (Steps 1–10)

All bugs are independent fixes on separate files. One agent handles all 9 files.

Task type: Extension

Sandbox layout:

/home/mohamed/workspace/orchestrata-ultimate-worktrees/subphase-0.1-bugfixes/
├── work/                                    <- WRITABLE
│   ├── traceability__init__.py              <- core/traceability/__init__.py
│   ├── logging_config.py                    <- core/logging/logging_config.py
│   ├── error_handling.py                    <- core/patterns/error_handling.py
│   ├── thread_pool.py                       <- core/threading/thread_pool.py
│   ├── node.py                              <- core/domain/node.py
│   ├── domain_entity.py                     <- core/domain/domain_entity.py
│   ├── decorators.py                        <- core/metrics/decorators.py
│   ├── metrics_config.py                    <- core/metrics/metrics_config.py
│   ├── audit.py                             <- core/logging/audit.py
│   └── AUDIT.md                             <- Agent MUST create this
├── context/                                 <- READ-ONLY
│   ├── spec.md                              <- This subplan's fix specifications
│   ├── standards.md                         <- From templates/standards.md
│   ├── traceability.md                      <- REQ mapping for all 11 bugs
│   ├── existing_design.mmd                  <- Mermaid diagram of current module relationships
│   ├── blast_radius.md                      <- Dependency graph from Neo4j
│   ├── ast-ultimate.json                    <- Full AST for reference
│   └── execution_plan_steps_1_11.md         <- Relevant execution plan excerpt
└── verify/                                  <- Orchestrator writes results here
    └── (test_output.txt after each verify cycle)

File mapping (work/ → repo):

Verify commands (run by orchestrator, not agent):

# 1. Import chain unblocked
python -c "from orchestrata_ultimate.core.traceability import traces; print('OK')"

# 2. All module imports succeed
python -c "
from orchestrata_ultimate.core.logging.logging_config import get_logger
from orchestrata_ultimate.core.patterns.error_handling import CircuitBreaker, retry_with_backoff
from orchestrata_ultimate.core.threading.thread_pool import ThreadPoolManager
from orchestrata_ultimate.core.domain.node import Node
from orchestrata_ultimate.core.domain.domain_entity import DomainEntity
from orchestrata_ultimate.core.metrics.decorators import instrument_request
from orchestrata_ultimate.core.metrics.metrics_config import configure_metrics
from orchestrata_ultimate.core.logging.audit import AuditLogger
print('ALL IMPORTS OK')
"

# 3. No deprecation warnings
python -W error::DeprecationWarning -c "
from orchestrata_ultimate.core.patterns.error_handling import *
print('NO DEPRECATION WARNINGS')
"

# 4. Existing tests pass
uv run pytest tests/test_error_handling.py tests/test_logging.py tests/test_metrics.py tests/test_cli.py -v --tb=short

# 5. Generated tests pass (run after generate_pytest.py)
uv run pytest tests/generated/ -v --tb=short

Test Generation Strategy

Before dispatching the agent, the orchestrator runs:

# Generate interface + validation tests for all affected files
python scripts/generate_pytest.py src/orchestrata_ultimate/core/traceability/__init__.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/logging/logging_config.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/patterns/error_handling.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/threading/thread_pool.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/domain/node.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/domain/domain_entity.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/metrics/decorators.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/metrics/metrics_config.py -o tests/generated
python scripts/generate_pytest.py src/orchestrata_ultimate/core/logging/audit.py -o tests/generated

These tests verify interface stability (method signatures, parameter types, return types). The agent's fixes must not break any generated interface test.

After the agent completes, the orchestrator runs:

# Full test suite
uv run pytest tests/ --ignore=tests/generated_new --ignore=tests/temp_generated -v --tb=short

# Coverage measurement using project's ParallelTestExecutor
python scripts/run_tests_parallel.py tests/ --workers auto

Coverage target: 100% of changed lines in the 9 modified files.

Requirements Traceability

AUDIT Document Requirements

The agent MUST produce work/AUDIT.md with these 6 sections:

1. Design Comparison

2. Changeset Report

3. Out-of-Scope Changes

Must be "None" or justified.

4. Traceability Matrix Coverage

All 10 REQs mapped to code locations.

5. Self-Audit Report (16 rows, all PASS)

Standard 16-row checklist from templates/standards.md.

6. Verification Notes

Agent's own verification of each fix (import tests, etc).

Pipeline Execution Checklist

Every step must be marked DONE with evidence before proceeding to the next. Do NOT skip steps. Do NOT proceed to the next phase until the current phase is complete.

Phase A: Pre-requisites

• [x] A1. Git working tree is clean (git status shows no modified tracked files)
• [x] A2. On master branch
• [x] A3. Orchestrator tool bugs fixed and committed SEPARATELY:
  • [x] A3a. create_sandbox.py list-vs-dict fix for _generate_existing_dataclasses
  • [x] A3b. generate_prompt.py missing basename Jinja2 filter
• [x] A4. agent_config.yaml validated: sandbox_base, subplan_path, spec_section_start, spec_section_end, destination (not dest), context_files as source/name dicts
• [x] A5. Neo4j is running and reachable

Evidence: ___

Phase B: Preflight

• [x] B1. Run: python scripts/orchestrator/preflight.py --config plans/plan_subphase_0_1/agent_config.yaml
• [x] B2. Result: env_vars PASS, neo4j PASS, redis PASS, neo4j_data PASS
• [x] B3. Warnings documented (imports WARN, ast_freshness WARN, git_clean WARN are expected)
• [x] B4. No unexpected FAILs beyond known import chain bug

Evidence: ___

Phase C: Generate Tests

• [x] C1. Run generate_pytest.py on ALL 10 affected source files
• [x] C2. Generated test files exist in tests/generated/ for each source file
• [x] C3. Count of generated test methods documented

Evidence: ___

Phase D: Create Sandbox

• [x] D1. Run: python scripts/orchestrator/create_sandbox.py --config plans/plan_subphase_0_1/agent_config.yaml
• [x] D2. Exit code 0
• [x] D3. work/ contains exactly 10 files (matching writable_files)
• [x] D4. context/ contains: spec.md, standards.md, traceability.md, existing_design.mmd, existing_dataclasses.md, ast-ultimate.json, execution-plan.md, subplan.md
• [x] D5. verify/ is empty

Evidence: ___

Phase E: Generate Prompt

• [x] E1. Run: python scripts/orchestrator/generate_prompt.py --config plans/plan_subphase_0_1/agent_config.yaml --output /tmp/subphase-0.1-prompt.md
• [x] E2. Exit code 0
• [x] E3. Prompt file exists and is non-empty

Evidence: ___

Phase F: Agent Execution in Sandbox

• [x] F1. All edits made ONLY to files in work/ directory
• [x] F2. No repo files modified directly (git status -- src/ is empty)
• [x] F3. All 10 bug fixes applied (B1-B10 + CLI)
• [x] F4. work/AUDIT.md created with all 6 required sections
• [x] F5. Each fix individually verified by reading the changed file in work/

Evidence: ___

Phase G: Verify Agent

• [x] G1. Run: python scripts/orchestrator/verify_agent.py --config plans/plan_subphase_0_1/agent_config.yaml
• [x] G2. All verify_commands PASS
• [x] G3. verify/results.md exists
• [x] G4. If any FAIL: agent fixes in work/, re-run (max 3 iterations)
• [x] G5. Final result: PASS

Evidence: ___

Phase H: Validate Audit

• [x] H1. Run: python scripts/orchestrator/validate_audit.py --config plans/plan_subphase_0_1/agent_config.yaml
• [x] H2. file_exists PASS
• [x] H3. sections PASS (all 6)
• [x] H4. self_audit_rows PASS (16 rows)
• [x] H5. no_fail_rows PASS
• [x] H6. requirements PASS (all 10 REQs)
• [x] H7. speculative_language PASS
• [x] H8. changeset_files PASS

Evidence: ___

Phase I: Human Review

• [x] I1. Full diff presented to user
• [x] I2. User explicitly approves

Evidence: User approved and instructed merge.

Phase J: Merge

• [x] J1. Run: python scripts/orchestrator/merge_agent.py --config plans/plan_subphase_0_1/agent_config.yaml --no-merge
• [x] J2. Commit created on branch fix/subphase-0.1-ultimate-bugs — aad18b9
• [x] J3. All tests pass on branch — 25/25
• [x] J4. Merged to master — 25/25 pass on master

Evidence: Merge commit on master. git merge --no-ff fix/subphase-0.1-ultimate-bugs

Files Summary

Total estimated change: ~136 lines across 10 files.

Sandbox Config

See plans/plan_subphase_0_1/agent_config.yaml for the authoritative config.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks