Orchestrata Ultimate › Plans › EFFICIENT_ANALYSIS_PLAN

Efficient Project Analysis Plan — Refined

Source: plans/EFFICIENT_ANALYSIS_PLAN.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-05-17 · 411 lines.

Prose pending.

Plan text

Efficient Project Analysis Plan — Refined

Orchestrata Ultimate — Post-Reconnaissance Edition

Date: 2026-05-17
Recon Status: ✅ COMPLETE
Goal: Thorough project analysis while minimizing context consumption via structured-data-first workflow.

1. Executive Summary (Actual Data)

Critical Findings

1. Architecture is the #1 problem. 85 layer violations. Core layer 0 imports from layers 1–4. Agents (layer 3) import from orchestrator (layer 4). Many concrete peer imports should be protocols.
2. Test purity is compromised. 171 violations including FakeProlog, FakeBridge, lifecycle_stub, Dummy() classes — all forbidden per AGENTS.md Test Standards (T1–T9).
3. Only 4 real code ERRORs (hardcoded Neo4j URI, 2 mypy untyped defs, 1 unused import). The 3,853 issues are 99.9% style/complexity/docstring warnings.
4. 13 orchestrata classes remain unported — mostly kernel pipeline mixins, CXP internal classes, and agent mixins.
5. All source files compile (py_compile clean). The "70+ syntax errors" from project-status.md appears stale.

2. Structured Data Sources (Read-First Priority)

2.1 Already-Generated Reports (Fresh from Recon)

2.2 AST Databases

2.3 Requirements Traceability

2.4 Spec Database

3. Tool Inventory — Verified Commands

Commands below were executed and verified during reconnaissance.

3.1 Tier 0: Global Intelligence (Run Once Per Session)

# 1. Architecture layer violations
python scripts/architecture_checker.py src/orchestrata_ultimate/ -v -o reports/architecture_report.md

# 2. Code quality snapshot (ruff + mypy + AST style)
python scripts/code-review.py src/orchestrata_ultimate/ --format json -o reports/code_review_latest.json

# 3. Test standards audit (fakes, stubs, skips, monkeypatch)
python scripts/test-review.py tests/ --format json -o reports/test_review_latest.json

# 4. Test purity scan (stricter; exits non-zero on violations)
python scripts/audit_test_purity.py

# 5. CI config validation
python scripts/validate_ci.py

3.2 Tier 1: Subsystem Deep Dives

# Scoped code review
python scripts/code-review.py src/orchestrata_ultimate/core/ --format json -o reports/code_review_core.json

# Scoped test review
python scripts/test-review.py tests/core/ --format json -o reports/test_review_core.json

# Unified review pipeline (shell script wrapper)
./scripts/run_review_pipeline.sh core

3.3 Tier 2: Spec & Requirements Analysis

# Validate a single spec against standards (REQUIRES spec.md path; no global mode)
python specs/spec-review/validate_spec_standards.py specs/004-cxp/spec.md --json

# Parse spec requirements to Prolog facts
python scripts/parse_spec_requirements.py 004 specs/004-cxp/checklists/requirements.md

# Scan implementation for spec evidence
python scripts/scan_implementation.py 004 src/orchestrata_ultimate/

Correction from original plan: validate_spec_standards.py does NOT have a global mode. It validates one spec at a time. For batch validation, loop over specs/*/spec.md.

3.4 Tier 3: AST Querying (JSON/SQLite)

# Regenerate ultimate AST (~30s)
python scripts/serialize_codebase_ast.py serialize \
    --source src/orchestrata_ultimate \
    --name orchestrata-ultimate \
    --output plans/ast-ultimate.json

# Regenerate deep comparison (~15s)
python scripts/serialize_codebase_ast.py deep-compare \
    plans/ast-ultimate.json plans/ast-orchestrata.json \
    --mapping plans/class-mapping.json \
    --output plans/ast-deep-comparison.json

# Generate SQLite for SQL queries
python -m scripts.ast_serializer src/orchestrata_ultimate sqlite --out analysis.db

Note: python -m scripts.ast_serializer fails with relative import errors when run directly. Use python scripts/ast_serializer/__main__.py or ensure PYTHONPATH includes scripts/.

3.5 Tier 4: Testing & Verification

# Parallel pytest with JSON output
python scripts/run_tests_parallel.py tests/core/ --json

# Test failure categorization
python scripts/run_test_audit.py core

# Real-time streaming audit
python scripts/run_test_audit_stream.py core

4. Verified Quick Queries (Copy-Paste)

All queries below were tested against actual data.

4.1 Porting Status Dashboard

jq '{
  total_pairs: .summary.total_pairs,
  mostly_complete: .summary.readiness.mostly_complete,
  equal: .summary.readiness.equal,
  unmapped_ultimate: (.unmapped.ultimate_only | length),
  unmapped_orchestrata_total: ([.unmapped.orchestrata_by_package[]] | add | length),
  unmapped_orchestrata_packages: (.unmapped.orchestrata_by_package | keys)
}' plans/ast-deep-comparison.json

4.2 Unmapped Orchestrata Classes (Still Need Porting)

jq '.unmapped.orchestrata_by_package | to_entries | map({package: .key, classes: .value, count: (.value | length)}) | sort_by(.count) | reverse' plans/ast-deep-comparison.json

Current result:

• orchestrata-kernel: PipelineEscalationMixin, PipelineKGMixin, PipelineRunner, PipelineSessionMixin (4)
• orchestrata-ux: JSScaffoldBuilder, UXJSMixin (2)
• orchestrata-cxp: _CXPEncoder, _HeaderProxy (2)
• orchestrata-agents: AnalystParsingMixin, AnalystStorageMixin (2)
• orchestrata-orchestrator: _BrokerMetrics (1)
• orchestrata-knowledge: CXPAuditor (1)
• orchestrata-codedirector: _ReviewContext (1)

4.3 Code Review Summary

jq '{
  files: .files_analyzed,
  total_issues,
  severity: .summary_by_severity,
  category: .summary_by_category,
  errors: [.issues[] | select(.severity == "ERROR")]
}' reports/code_review_latest.json

4.4 Test Review Summary

jq '{
  files: .files_analyzed,
  total_issues,
  severity: .summary_by_severity,
  category: .summary_by_category
}' reports/test_review_latest.json

4.5 Top 15 Files by Issue Count

jq '.issues | group_by(.file) | map({file: .[0].file, count: length}) | sort_by(.count) | reverse | .[:15]' reports/code_review_latest.json

Current leaders:

1. core/interfaces/__init__.py — 193 issues
2. core/protocols/__init__.py — 187 issues
3. kernel/performance_manager.py — 140 issues
4. cxp/proto/cxp_pb2_grpc.py — 104 issues (auto-generated protobuf)
5. agents/volks/qa/volk.py — 102 issues

4.6 Architecture Violation Summary

# Count violations by source package
grep "VIOLATION:" reports/architecture_report.md | sed 's/.*orchestrata_ultimate\.\([a-z_]*\).*/\1/' | sort | uniq -c | sort -rn | head -15

4.7 Test Purity Violation Summary

# Already generated by audit_test_purity.py; just re-run:
python scripts/audit_test_purity.py 2>&1 | grep -E "^\[|^SUMMARY:|^  \["

4.8 Classes with Method Signature Mismatches

jq '.pairs[]
  | select(.ultimate_exists and .orchestrata_exists)
  | {
      name: .ultimate_name,
      mismatches: [
        .method_diff.shared[]
        | select(
            .async_match == false or
            .return_match == false or
            (.param_diffs | length > 0)
          )
        | {method: .name, issues: [
            (if .async_match == false then "async_mismatch" else empty end),
            (if .return_match == false then "return_mismatch" else empty end),
            (if (.param_diffs | length > 0) then "param_mismatch" else empty end)
          ]}
      ]
    }
  | select((.mismatches | length) > 0)' plans/ast-deep-comparison.json | head -60

5. Priority-Ranked Action Items (Derived from Recon)

🔴 P0 — Fix Before Any Major Porting

🟡 P1 — High Impact, Medium Urgency

🟢 P2 — Cleanup & Polish

6. The Efficient Analysis Workflow (Refined)

Phase A: Reconnaissance (5–10 min) — MANDATORY

Run these in parallel:

# Terminal 1: Architecture + Code quality
python scripts/architecture_checker.py src/orchestrata_ultimate/ -v -o reports/architecture_report.md
python scripts/code-review.py src/orchestrata_ultimate/ --format json -o reports/code_review_latest.json

# Terminal 2: Test audits
python scripts/test-review.py tests/ --format json -o reports/test_review_latest.json
python scripts/audit_test_purity.py

# Terminal 3: AST status (if source changed since Apr 25)
python scripts/serialize_codebase_ast.py serialize --source src/orchestrata_ultimate --name orchestrata-ultimate --output plans/ast-ultimate.json
python scripts/serialize_codebase_ast.py deep-compare plans/ast-ultimate.json plans/ast-orchestrata.json --mapping plans/class-mapping.json --output plans/ast-deep-comparison.json

Decision gate: If audit_test_purity.py or architecture_checker.py output grew significantly since last run, stop and fix before proceeding.

Phase B: Query-First Investigation

Rule: Query JSON before reading source. Use this priority:

1. Project-level questions → jq on ast-deep-comparison.json, rtm.json, code_review_latest.json
2. Dependency questions → architecture_checker.py or grep on violation reports
3. Specific class/function questions → jq filter on AST JSON to find file/module, THEN Grep for exact line, THEN ReadFile with line_offset
4. Never read a whole module to "understand it" — the AST JSON already has signatures, inheritance, imports, and decorators

Phase C: Spec Compliance Checks

# Batch validate all specs
for spec in specs/*/spec.md; do
  echo "=== $spec ==="
  python specs/spec-review/validate_spec_standards.py "$spec" --json | jq '{spec_id, passed, failed, coverage_pct}'
done > reports/spec_validation_summary.txt

Phase D: Surgical Fixes

Only after JSON queries pinpoint the exact file and line:

# Example workflow:
# 1. Find file with issue
jq '.issues[] | select(.code == "HARDCODED_URI") | .file' reports/code_review_latest.json
# 2. Grep for context
 grep -n "bolt://" src/orchestrata_ultimate/core/infra_config.py
# 3. Read only relevant lines
# ReadFile path=src/... line_offset=60 n_lines=20

7. Context Conservation Rules (Updated)

✅ DO

• Run tools and read their small outputs (<100KB reports)
• Use jq to extract sub-5KB slices from AST JSON
• Use Grep to find exact file/line references
• Use ReadFile with line_offset and n_lines to read ≤50 lines at a time
• Batch spec validation via shell loop, capturing only jq-filtered summaries

❌ DON'T

• Do NOT ReadFile on plans/ast-ultimate.json (14MB) or plans/ast-deep-comparison.json (3.2MB)
• Do NOT ReadFile on any generated protobuf file (*_pb2_grpc.py, *_pb2.py)
• Do NOT read entire test files to understand failures (use run_test_audit.py categorization)
• Do NOT read core/interfaces/__init__.py or core/protocols/__init__.py whole (193 and 187 issues each; use tool output)
• Do NOT manually scan for imports (architecture checker + AST JSON have them)
• Do NOT validate specs one-by-one interactively (batch them in a shell loop)

8. Maintenance: Keeping Data Fresh

After any significant code change:

# Regenerate all recon artifacts
python scripts/serialize_codebase_ast.py serialize --source src/orchestrata_ultimate --name orchestrata-ultimate --output plans/ast-ultimate.json

python scripts/serialize_codebase_ast.py deep-compare \
    plans/ast-ultimate.json plans/ast-orchestrata.json \
    --mapping plans/class-mapping.json \
    --output plans/ast-deep-comparison.json

python scripts/architecture_checker.py src/orchestrata_ultimate/ -v -o reports/architecture_report.md
python scripts/code-review.py src/orchestrata_ultimate/ --format json -o reports/code_review_latest.json
python scripts/test-review.py tests/ --format json -o reports/test_review_latest.json

Runtime budget: ~3 minutes total on this hardware.

9. Summary Checklist

Before starting any analysis task:

• [ ] reports/architecture_report.md exists and is ≤24h old
• [ ] reports/code_review_latest.json exists
• [ ] reports/test_review_latest.json exists
• [ ] plans/ast-deep-comparison.json exists and is ≤24h old (if source changed)
• [ ] jq is available for JSON filtering

During analysis:

• [ ] Query JSON first — never read source for information already in AST/reports
• [ ] Use jq filters — extract only relevant slices (<5KB)
• [ ] Use Grep — to find exact file paths from filtered results
• [ ] Use ReadFile with offsets — read only the relevant function/class
• [ ] Use tools, not eyes — let automated tools find issues; human reviews only for architecture/design decisions

After analysis:

• [ ] Update plans/project-status.md with key metrics
• [ ] Append new findings to reports/ directory
• [ ] If code changes made, regenerate all recon artifacts

This plan is based on actual reconnaissance data from 2026-05-17. All commands were verified. All metrics are real.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks