Orchestrata Ultimate › Plans › tools_useful_to_plan
Tools Useful to Plan
Source: plans/tools_useful_to_plan.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-04-11 · 795 lines.
Prose pending.
Plan text
Tools Useful to Plan
Reference guide for Claude and other agents working on the Orchestrata-Ultimate port. Every tool lives in scripts/ unless noted. All commands assume the project root as CWD.
1. serialize_codebase_ast.py (1451 LOC)
Purpose: Source of truth for what exists in both codebases. Serializes every module, class, method, signature, import, and inheritance chain to JSON. Performs semantic comparison between orchestrata and orchestrata-ultimate.
Key classes: CodebaseAST, ClassInfo, FunctionInfo, ParamInfo, ImportInfo, ModuleInfo, ClassVarInfo
Subcommands
serialize — Produce a full AST JSON for one codebase
python scripts/serialize_codebase_ast.py serialize \
--source src/orchestrata_ultimate \
--name orchestrata-ultimate \
--output plans/ast-ultimate.json
python scripts/serialize_codebase_ast.py serialize \
--source /home/mohamed/workspace/orchestrata/packages \
--name orchestrata \
--output plans/ast-orchestrata.json
--sourcemust be the root Python package directory (contains__init__.pyor submodules)--nameis a label embedded in the JSON--outputis the destination file
deep-compare — Semantic comparison using class mapping
python scripts/serialize_codebase_ast.py deep-compare \
plans/ast-ultimate.json \
plans/ast-orchestrata.json \
--mapping plans/class-mapping.json \
--output plans/ast-deep-comparison.json
Produces the 336K JSON the execution plan was built from. For each class pair it reports readiness (stub, partial, mostly_complete, needs_creation, equal), method-level signature diffs, inheritance chains, and class variable diffs. Also reports per-package coverage, import translations, and dependency diffs.
compare — Shallow name-based comparison
python scripts/serialize_codebase_ast.py compare \
plans/ast-ultimate.json \
plans/ast-orchestrata.json \
--output plans/ast-comparison.json
Name-only match without semantic pairing. Less useful than deep-compare.
When to use
- After completing a tier of the execution plan, re-serialize ultimate and deep-compare to measure progress
- When adding a new class mapping, add it to
plans/class-mapping.jsonthen re-rundeep-compare - When checking if a method signature was ported correctly
2. diagram_generator.py (1585 LOC)
Purpose: Bidirectional Code ↔ Mermaid diagram generator. Uses the full CodebaseAST from serialize_codebase_ast.py (not a watered-down model). Generates class diagrams from source, generates Python skeleton code from diagrams, and verifies round-trip fidelity.
Key classes: CodeToDiagram, DiagramToCode, CodebaseAnalyzer, SpecUpdater
Subcommands
code2diagram — Generate Mermaid from source
python scripts/diagram_generator.py code2diagram \
--source src/orchestrata_ultimate \
--output-dir specs/001-kernel
Options:
--source— Root of the Python package to analyze--output-dir— Directory to write the.mmdfile--filter-modules— Comma-separated list of module paths to include (e.g.,core.orchestrator,core.domain)--filter-classes— Comma-separated list of class names to include
diagram2code — Generate Python skeleton from Mermaid
python scripts/diagram_generator.py diagram2code \
--diagram specs/001-kernel/kernel.mmd \
--output-dir src/orchestrata_ultimate/core
verify — Round-trip verification (code → diagram → code)
python scripts/diagram_generator.py verify \
--source src/orchestrata_ultimate \
--output-dir /tmp/verify_output
Reports class count matches, method set matches, and any mismatches.
Legacy modes (for spec updating)
# Update a single spec with diagrams
python scripts/diagram_generator.py --spec specs/001-kernel/spec.md
# Update all specs
python scripts/diagram_generator.py --all
# Dry run (show what would change)
python scripts/diagram_generator.py --all --dry-run
# Validate existing diagrams only
python scripts/diagram_generator.py --validate
When to use
- After porting classes, regenerate diagrams for the relevant spec
- When verifying that the codebase and its documentation are in sync
- When checking round-trip fidelity (62/62 classes and method sets should match)
3. spec-review.py (1080 LOC)
Purpose: Reviews all 55+ spec files in specs/*/spec.md against the 20 standards defined in specs/spec-review/spec-standards.yaml. Reports missing sections, incomplete data models, missing API contracts, etc.
# Review all specs
python scripts/spec-review.py
# Review a specific spec
python scripts/spec-review.py specs/001-kernel/spec.md
Output includes total specs reviewed, pass/fail counts, issues by section, and per-spec issue counts with criticality levels.
When to use
- After updating a spec, re-run to check compliance
- To identify which specs need the most work before a milestone
4. generate_comprehensive_audit.py (752 LOC)
Purpose: Generates a compliance-audit.md for each spec, comparing the spec against the actual codebase using CodebaseAST + CodeToDiagram. Reports which spec requirements are implemented, partially implemented, or missing.
Note: This script must be run from the scripts/ directory (uses local imports).
cd scripts && python generate_comprehensive_audit.py
The script currently has a hardcoded source path that needs the codebase at scripts/src/orchestrata_ultimate. Adjust the path or symlink if needed.
When to use
- After completing a tier, generate audits to see which requirements are now covered
- To produce compliance reports for stakeholder reviews
5. code-review.py (1163 LOC)
Purpose: Unified code review tool that runs ruff, mypy, and AST-based style checks. Produces reports in text, JSON, or markdown.
# Review specific files
python scripts/code-review.py src/orchestrata_ultimate/core/orchestrator.py
# Review a directory
python scripts/code-review.py src/orchestrata_ultimate/core/ --format markdown -o review.md
# Skip specific checks
python scripts/code-review.py src/orchestrata_ultimate/ --skip-mypy --skip-ruff
# Set minimum severity
python scripts/code-review.py src/orchestrata_ultimate/ --min-severity WARNING
Options:
--output, -o— Output file for the report--format {text,json,markdown}— Output format--min-severity {ERROR,WARNING,INFO}— Filter by severity--skip-ruff/--skip-mypy/--skip-style/--skip-checklist— Disable specific checks--max-files— Limit number of files processed
When to use
- Before committing ported code, run a review to catch style and type issues
- To generate a review report for a PR
6. architecture_checker.py (461 LOC)
Purpose: Validates project architecture — enforces import rules, module boundaries, dependency direction, and layering constraints.
# Check entire project
python scripts/architecture_checker.py src/orchestrata_ultimate/
# Check specific paths with verbose output
python scripts/architecture_checker.py src/orchestrata_ultimate/core/ -v
# Output to file
python scripts/architecture_checker.py src/orchestrata_ultimate/ -o arch_report.md
Options:
--verbose, -v— Show per-violation details--output, -o— Write report to file
When to use
- After porting a module, verify it doesn't violate architecture constraints (e.g., core depending on agents)
- As a pre-commit gate for architectural hygiene
7. auto_fix.py (241 LOC)
Purpose: Auto-fixer for common LLM-generated code defects. Runs a pipeline of fixers over Python files.
Available fixers:
| Fixer | What it does |
|---|---|
relative_imports |
Rewrites relative imports (from .. → absolute) |
line_length |
Splits lines exceeding 88 characters |
import_order |
Reorders imports: stdlib, third-party, local (separated by blank lines) |
redis_host |
Replaces hardcoded "localhost" in Redis constructors with os.environ.get() |
missing_datetime |
Adds missing datetime import when datetime is used bare |
# Fix all Python files in a directory
python scripts/auto_fix.py src/orchestrata_ultimate/core/
# Fix specific files with verbose output
python scripts/auto_fix.py src/orchestrata_ultimate/core/orchestrator.py -v
# List available fixers
python scripts/auto_fix.py --list-fixers
# Disable specific fixers
python scripts/auto_fix.py src/orchestrata_ultimate/ --disable redis_host --disable line_length
When to use
- After porting a file from orchestrata, run auto_fix to fix imports and common issues
- Before running mypy/ruff, run auto_fix to clean up relative imports and line length
8. fix_relative_imports.py (116 LOC) / fix_relative_imports_ast.py (139 LOC)
Purpose: Two scripts that rewrite relative imports to absolute imports. fix_relative_imports_ast.py uses AST-based parsing (more reliable); fix_relative_imports.py uses regex.
# Fix a specific file
python scripts/fix_relative_imports_ast.py src/orchestrata_ultimate/core/traceability/__init__.py
# Fix all files in a directory (no args = current dir)
python scripts/fix_relative_imports_ast.py
Prefer fix_relative_imports_ast.py — it is more accurate. Both modify files in-place and remove backup files.
When to use
- After copying a file from orchestrata that uses relative imports
- When
auto_fix.pywithrelative_importsfixer is insufficient
9. generate_pytest.py (917 LOC)
Purpose: Generates pytest test files from AST analysis of source code. Creates test stubs with proper imports, fixtures, and test function signatures based on the classes and methods found.
# Generate tests for a single module
python scripts/generate_pytest.py src/orchestrata_ultimate/core/orchestrator.py -o tests/
# Generate tests for a directory
python scripts/generate_pytest.py src/orchestrata_ultimate/core/ -o tests/
# Use custom template directory
python scripts/generate_pytest.py src/orchestrata_ultimate/core/ -o tests/ -t templates/
Options:
--output, -o— Output directory for generated test files--template-dir, -t— Directory containing Jinja2 templates
When to use
- After porting a new module, generate initial test stubs
- Before writing manual tests, generate the scaffold and fill in assertions
10. generate_module.py (258 LOC)
Purpose: Generates Python module scaffolds from templates. Supports three module types: task, service, agent.
# Generate an agent module
python scripts/generate_module.py agent qa_volk \
--class-name QualityAssuranceAgent \
--description "QA verification agent" \
--output src/orchestrata_ultimate/agents/volks/
# Generate a service module
python scripts/generate_module.py service cache \
--class-name CacheService \
--output src/orchestrata_ultimate/core/
# Generate a task module
python scripts/generate_module.py task validation \
--output src/orchestrata_ultimate/kernel/
Options:
--output, -o— Output directory (default: current)--class-name, -c— PascalCase class name (defaults to derived from module name)--description, -d— Description for the module/class
When to use
- When starting a new module from scratch during porting
- When the execution plan says "create new file" for a class that doesn't exist yet
11. generate_docs.py (383 LOC)
Purpose: Generates markdown or README documentation from source code using AST analysis.
# Generate markdown docs for a module
python scripts/generate_docs.py src/orchestrata_ultimate/core/orchestrator.py -o docs/core.md
# Generate README format for a directory
python scripts/generate_docs.py src/orchestrata_ultimate/core/ -f readme -o docs/core_README.md
# Use a custom template
python scripts/generate_docs.py src/orchestrata_ultimate/core/ -t custom_template.md
Options:
--output, -o— Output file path--format, -f {markdown,readme}— Output format--template, -t— Custom template file
When to use
- When generating API documentation for ported modules
- When a spec requires documentation artifacts
12. enhance_spec.py (462 LOC)
Purpose: Analyzes spec files and adds missing sections based on the standards defined in specs/spec-review/spec-standards.yaml. Inserts placeholder sections for required content.
# Preview what would be added to a spec (dry run, default)
python scripts/enhance_spec.py specs/001-kernel/spec.md
# Actually apply changes
python scripts/enhance_spec.py specs/001-kernel/spec.md --apply
# Process all specs
python scripts/enhance_spec.py --all --apply
# JSON output
python scripts/enhance_spec.py specs/001-kernel/spec.md --output json
Options:
--all— Process all spec.md files inspecs/--dry-run— Show what would be added without modifying files (default)--apply— Actually modify the spec files--output {text,json}— Output format
When to use
- After creating a new spec that is missing required sections
- Before running
spec-review.pyto fill in missing sections first
13. generate_rtm.py (145 LOC) / generate_complete_rtm.py (189 LOC) / analyze_orchestrata_rtm.py (205 LOC)
Purpose: Requirements Traceability Matrix tools. They parse requirements from specs, find @traces annotations in source code, and map requirements to implementation.
generate_rtm.py
Basic RTM generation from source code trace annotations.
python scripts/generate_rtm.py
Outputs rtm/rtm.json and rtm/rtm.md.
generate_complete_rtm.py
Extended RTM that also includes test file mappings and integrates with orchestrata's RTM data.
python scripts/generate_complete_rtm.py
Outputs rtm/rtm.json, rtm/rtm.md, rtm/comprehensive_mapping.json, and rtm/integrated_rtm.json.
analyze_orchestrata_rtm.py
Analyzes the orchestrata source RTM entries and maps them to orchestrata-ultimate paths. Produces a coverage report.
python scripts/analyze_orchestrata_rtm.py
Outputs rtm/orchestrata_coverage_report.md and rtm/orchestrata_rtm_sample_transformed.json.
When to use
- After porting classes with
@tracesannotations, regenerate RTM to track coverage - To see which requirements have been verified vs. still missing
- To map orchestrata requirements to ultimate implementation paths
14. parse_requirements.py (162 LOC)
Purpose: Parses requirements from spec files and produces structured JSON. Used by RTM generators.
Note: Currently broken due to an import error (ModuleNotFoundError: No module named 'orchestrata_ultimate.core.traceability_utils'). This needs a fix in src/orchestrata_ultimate/core/traceability/__init__.py before it can run.
python scripts/parse_requirements.py
When to use
- As a dependency for the RTM pipeline (once the import is fixed)
15. run_tests.sh (264 LOC) / run_tests_parallel.py (155 LOC) / run_prolog_tests.py (241 LOC) / run_plunit_tests.py (319 LOC)
Purpose: Test runners with different specializations.
run_tests.sh
Parallel per-module test runner. Each module gets a fresh Prolog engine to avoid pyswip's global singleton issues.
# Run all tests in parallel
./scripts/run_tests.sh
# Force sequential execution
./scripts/run_tests.sh --serial
# Use Python runner instead of shell
./scripts/run_tests.sh --python
# Pass extra pytest args
./scripts/run_tests.sh -v --tb=short
Note: Shell runner is deprecated in favor of the Python runner. Use --python flag or call run_tests_parallel.py directly.
run_tests_parallel.py
Python-based parallel test runner. Currently broken due to import error.
python scripts/run_tests_parallel.py
python scripts/run_tests_parallel.py --serial
run_prolog_tests.py
Runs Prolog test files per-file with crash recovery and JUnit XML output.
# Run all Prolog tests
python scripts/run_prolog_tests.py
# Run with coverage
python scripts/run_prolog_tests.py --cov
# Run with pytest marker filter
python scripts/run_prolog_tests.py --marker governance
# Specify test directories
python scripts/run_prolog_tests.py --dirs tests/knowledge tests/governance
# Output JUnit XML
python scripts/run_prolog_tests.py --junit-xml test-prolog.xml
run_plunit_tests.py
Runs SWI-Prolog plunit tests and emits JUnit XML.
# Run all plunit tests
python scripts/run_plunit_tests.py
# Filter by file pattern
python scripts/run_plunit_tests.py --file test_ethics.plt
# Custom plt directory
python scripts/run_plunit_tests.py --plt-dir prolog/tests/
# JUnit XML output
python scripts/run_plunit_tests.py --junit-xml junit.xml
When to use
run_tests.sh/run_tests_parallel.py— Python unit tests (once import error is fixed)run_prolog_tests.py— Prolog governance and knowledge testsrun_plunit_tests.py— SWI-Prolog plunit tests specifically- Preferred:
uv run pytest tests/ -n auto -x --timeout=120for Python tests (per AGENTS.md)
16. check_test_infrastructure.sh (60 LOC) / setup_test_infrastructure.sh (671 LOC)
Purpose: Infrastructure health check and setup for running real (non-mock) tests against Redis, Neo4j, and SWI-Prolog.
check_test_infrastructure.sh
Verifies that test infrastructure is running:
bash scripts/check_test_infrastructure.sh
Checks: Redis (port 6380, DB 15), Neo4j (port 7688), SWI-Prolog, test directories.
setup_test_infrastructure.sh
Sets up the full test infrastructure (Docker containers, DBs, message queues):
bash scripts/setup_test_infrastructure.sh
Creates: Redis test instance (port 6380, DB 15), Neo4j test instance (port 7688), Prolog test server (port 8080), temp directories.
When to use
- Before running integration tests, run
check_test_infrastructure.sh - When setting up a fresh environment, run
setup_test_infrastructure.sh
17. debug_mypy.py (34 LOC)
Purpose: Runs mypy on the project and parses the output into structured error reports.
python scripts/debug_mypy.py
Parses mypy errors and prints them in a more readable format. Currently shows errors in thread_pool.py (union-attr, no-any-return, var-annotated).
When to use
- When investigating mypy type errors in the codebase
- As a quick type-check diagnostic
18. update_opencode_font.py (35 LOC)
Purpose: Updates the opencode font configuration to use MesloLGL Nerd Font Mono.
python scripts/update_opencode_font.py
Sets: fontSize=14, mono=MesloLGL Nerd Font Mono, font=ibm-plex-mono.
When to use
- When setting up a new development environment
- When opencode font rendering is broken
19. session_cli.py (94 LOC)
Purpose: CLI for session management — tracking context, achievements, and session state.
Note: Currently broken due to the same traceability_utils import error.
python scripts/session_cli.py status
python scripts/session_cli.py start
python scripts/session_cli.py achievement <name>
When to use
- To track work session state and achievements
- Currently non-functional until import error is fixed
20. test_diagram.py (57 LOC) / test_diagram2.py (74 LOC)
Purpose: Legacy test scripts for the diagram generator. Both are currently broken due to importing old class names (ASTAnalyzer, MermaidGenerator) that no longer exist in the refactored diagram_generator.py.
Status: Broken — should be deleted or updated to use the current CodeToDiagram/DiagramToCode API.
When to use
- Not currently usable. Replace with proper pytest tests or update imports.
Key Data Files
| File | Purpose |
|---|---|
plans/ast-ultimate.json |
Full AST of orchestrata-ultimate (397K) |
plans/ast-orchestrata.json |
Full AST of orchestrata source (8.0MB) |
plans/ast-deep-comparison.json |
Semantic comparison with all diffs (336K) |
plans/ast-comparison.json |
Shallow name-based comparison |
plans/class-mapping.json |
Explicit class name mappings + regex transforms + module mappings + tier assignments |
plans/execution-plan.md |
238-step execution plan for the port |
specs/spec-review/spec-standards.yaml |
20 standards for spec compliance |
rtm/rtm.json |
Requirements traceability matrix |
Common Workflows
After completing a tier of the execution plan
# 1. Re-serialize the ultimate codebase
python scripts/serialize_codebase_ast.py serialize \
--source src/orchestrata_ultimate \
--name orchestrata-ultimate \
--output plans/ast-ultimate.json
# 2. Deep-compare against orchestrata
python scripts/serialize_codebase_ast.py deep-compare \
plans/ast-ultimate.json \
plans/ast-orchestrata.json \
--mapping plans/class-mapping.json \
--output plans/ast-deep-comparison.json
# 3. Update spec diagrams
python scripts/diagram_generator.py --all
# 4. Review specs
python scripts/spec-review.py
# 5. Run tests
uv run pytest tests/ -n auto -x --timeout=120
After porting a single file
# 1. Fix imports
python scripts/auto_fix.py src/orchestrata_ultimate/<path>/
# 2. Code review
python scripts/code-review.py src/orchestrata_ultimate/<path>/
# 3. Architecture check
python scripts/architecture_checker.py src/orchestrata_ultimate/<path>/ -v
# 4. Generate tests
python scripts/generate_pytest.py src/orchestrata_ultimate/<path>/ -o tests/<path>/
# 5. Run relevant tests
uv run pytest tests/<path>/ -n auto -x --timeout=120
Checking overall porting progress
# Re-serialize and deep-compare (see above), then inspect the output:
python -c "
import json
d = json.load(open('plans/ast-deep-comparison.json'))
for tier, info in d.get('tier_analysis', {}).items():
print(f'{tier}: {info.get(\"coverage_pct\", 0):.1f}% coverage')
"
21. sync_ast_to_neo4j.py (306 LOC) — CLI wrapper
Purpose: Syncs the serialized AST JSON to Neo4j via Neo4jSyncer engine (src/orchestrata_ultimate/knowledge_graph/sync.py, 1,986 LOC). 12 node types, 18 relationship types.
python scripts/sync_ast_to_neo4j.py schema # Create constraints + indexes
python scripts/sync_ast_to_neo4j.py full --ast plans/ast-ultimate.json --project orchestrata-ultimate # Full sync
python scripts/sync_ast_to_neo4j.py stats # Node/edge counts
python scripts/sync_ast_to_neo4j.py verify --project orchestrata-ultimate # 8 verification checks
python scripts/sync_ast_to_neo4j.py incremental --ast plans/ast-ultimate.json --project orchestrata-ultimate --modules agents.base_agent # Re-sync specific modules
python scripts/sync_ast_to_neo4j.py map --mapping plans/class-mapping.json # Cross-project MAPS_TO edges
Connection via env vars (NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD) or CLI flags (--uri, --user, --password).
22. perplexity_search.py (89 LOC) — Perplexity API Integration
Purpose: Command-line interface to query the Perplexity AI API for web search and information retrieval.
# Basic usage
python scripts/perplexity_search.py "What is the latest Python version?"
# Specify model
python scripts/perplexity_search.py --model pplx-70b-online "Advanced query"
# Output raw JSON response
python scripts/perplexity_search.py --raw "Debug query"
# Use specific API key
python scripts/perplexity_search.py --api-key your-key-here "Query"
Environment Variables:
PPLX_API_KEY: Your Perplexity API key (required if not using--api-keyflag)
Available Models:
pplx-7b-online(default)pplx-70b-onlinepplx-7b-chatpplx-70b-chatmixtral-8x7b-instructllama-2-70b-chat
When to use
- When you need real-time web search capabilities during development
- For research queries that require up-to-date information
- As an alternative to manual web searching during coding tasks
23. scripts/orchestrator/ (7 files, 2,110 LOC) — Agent Dispatch Pipeline
Purpose: Automates the sandboxed agent workflow. Agents work in isolated directories with only designated files writable. Orchestrator handles context injection, test execution, verification, and git operations.
Config-driven via YAML. See scripts/orchestrator/README.md for config format and docs/ORCHESTRATOR.md for architecture.
python scripts/orchestrator/preflight.py --config agent_config.yaml # 8 env checks
python scripts/orchestrator/create_sandbox.py --config agent_config.yaml # Build sandbox + context
python scripts/orchestrator/generate_prompt.py --config agent_config.yaml --output /tmp/prompt.md # Render prompt
python scripts/orchestrator/verify_agent.py --config agent_config.yaml # Copy→test→diff
python scripts/orchestrator/validate_audit.py --config agent_config.yaml # Validate AUDIT.md
python scripts/orchestrator/merge_agent.py --config agent_config.yaml # Branch→commit→merge (no push)
Pipeline: preflight → create_sandbox → generate_prompt → [dispatch agent] → verify_agent → validate_audit → [human review] → merge_agent
Known Issues
-
Import error in
src/orchestrata_ultimate/core/traceability/__init__.py— referencesorchestrata_ultimate.core.traceability_utilswhich doesn't exist. This breaksparse_requirements.py,run_tests_parallel.py,session_cli.py, andgenerate_comprehensive_audit.py. -
test_diagram.pyandtest_diagram2.py— import old class names that no longer exist. Should be deleted or updated. -
generate_comprehensive_audit.py— must be run fromscripts/directory and has a hardcoded source path. Needs path adjustment. -
run_tests.sh— marked as deprecated in favor ofrun_tests_parallel.py(or plainuv run pytest). -
run_tests_parallel.py— broken due to the traceability import error.