Orchestrata Ultimate › Plans › spec-audit › J-standards-arch

Cluster J — Standards & Architecture (SPEC-097 … SPEC-101)

Source: plans/spec-audit/J-standards-arch.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-04 · 305 lines.

Prose pending.

Plan text

Cluster J — Standards & Architecture (SPEC-097 … SPEC-101)

Investigation agent J. READ-ONLY audit of src/orchestrata_ultimate/ against the five cross-cutting standards/architecture specs. Evidence by code only (checklist - [ ] markers ignored per _METHOD.md). These are conventions enforced across the whole codebase, so conformance is judged by systemic patterns + the standards/check-deps gates, not one module.

Headline: the AST constitutional gates are green — standards → 0 Key-Rule violations, check-deps --include-tests → all imports declared. The real remaining work is not in those gates; it is that three of these specs (098, 099, and large parts of 100/097) describe an aspirational SaaS-shaped system (REST API, MFA/OAuth/encryption, structlog+OpenTelemetry, mock-based test templates) that the project deliberately is not. The project is a local multi-agent dev tool: WebSocket JSON-RPC + gRPC transports, JWT+PAM++ auth, stdlib-JSON logging, real-infra no-mock tests. So the dominant finding is divergence + unbuilt aspirational capability, not half-finished ports.

SPEC-097 Testing Standards — PARTIAL (core enforcement DONE; spec text largely DIVERGENT)

• Specifies: test templates, coverage thresholds, quality gates, naming, markers, faker/hypothesis test data, unit/integration/e2e layout, CI.
• Verified:
  • ✅ Quality-gate enforcement is real and complete — scripts/test-review.py implements T1–T15 (all 15 rules present: titles at test-review.py:63–79, dispatch :108). These are stricter than the spec: T1 "No mocking", T2 "No stubs/fakes", T4 "Function-scope fixtures only", T15 "No pytest-xdist".
  • ✅ No-xdist enforced two ways — PreToolUse hook .claude/hooks/no-xdist.sh (.claude/settings.json:9) + test-review.py:_check_no_xdist (:206, regex :211).
  • ✅ Batched runner exists and is canonical — scripts/run_tests.py dispatches 4 subprocesses via ThreadPoolExecutor, each running ~244 modules as one pytest session (run_tests.py:7–14); injects PYTEST_XDIST_WORKER per subprocess for Redis-DB isolation. Second runner scripts/run_tests_threadpool.py also present (pre-push).
  • ✅ Markers registered & strict — pyproject.toml:171–185 registers unit, integration, e2e, requires_redis/neo4j/prolog, isolation, etc.; addopts has --strict-markers --strict-config (:161–162).
  • ✅ RTM/traceability data-models from the class diagram exist (RTMReport, ValidationReport) — core/traceability/__init__.py:69, core/traceability_utils.py:143.
  • ⚠️ DIVERGENCE — templates contradict the project. Every spec test template (§2.1–2.3, Example 1) uses unittest.mock.Mock/@patch, freezegun, faker — all forbidden by T1/T8 and the "No mocking — real implementations only" rule. The spec is out of date relative to the project's actual (stricter) philosophy.
  • ⚠️ DIVERGENCE — layout. Spec FR-003 mandates tests/unit/, tests/e2e/ and pytest tests/unit/. Actual layout is by domain (tests/agents, tests/core, tests/cxp, …) + markers. tests/unit/ and tests/e2e/ are absent; tests/integration/ has 1 file. Commands in FR-003 do not match reality.
  • ❌ GAP — property-based / test-data libs. hypothesis, faker, freezegun are not declared in pyproject.toml (only pytest-cov, pytest-xdist, pytest-asyncio, fakeredis). Spec §1.4 (hypothesis) + §5 (faker) unmet. faker/freezegun are deliberately-not-used (mocking-adjacent, banned); hypothesis is a genuine small gap.
  • ❌ GAP — coverage thresholds unenforced. [tool.coverage.run/report] exist (pyproject.toml:188,198) but there is no fail_under gate and run_tests.py does not --cov. The 90/85/80/70% targets (FR-002) are aspirational, not gated.
  • ⚠️ ORPHAN — ParallelTestExecutor (core/threading/test_executor.py:183, the spec's "Test Runner Interface" run_tests) has only 2 in-file uses; the real runner (run_tests.py) rolls its own subprocess batching and never calls it. refs ParallelTestExecutor → 1 def, 1 import, 2 use (all internal).
  • Minor inconsistency: pyproject.toml addopts:169 carries --dist=loadfile — an xdist flag T15 forbids; inert without -n, but the project's own default config would fail its own T15 if it were in a test file.
• Remaining work / plan (small):
  • Scope: (a) rewrite SPEC-097 to the actual philosophy (no-mock, real-infra, domain layout + markers, T1–T15, batched runner) — this shrinks the "gap" to spec-debt; (b) optionally add a coverage fail_under gate; (c) optionally add hypothesis for property tests; (d) decide ParallelTestExecutor keep-vs-delete.
  • Target files: specs/097-testing-standards/spec.md (rewrite); pyproject.toml ([tool.coverage.report] fail_under, add hypothesis); scripts/run_tests.py (cov).
  • Approach: this is documentation reconciliation + two additive tooling knobs. No behaviour change to test enforcement (already complete).
  • Dependencies: constrains every other cluster — T1–T15 are the rules all clusters' test plans must obey. No inbound deps.
  • Effort: S. Risk: low (spec-only + additive).
• Evidence log: test-review.py T1–T15 grep; .claude/settings.json hook; run_tests.py header; pyproject.toml markers/addopts/coverage; ls tests/{unit,integration,e2e}; dep grep (hypothesis/faker absent); refs ParallelTestExecutor.

SPEC-098 API Standards — DIVERGENCE / GAP (system has no REST API; interfaces are WS JSON-RPC + gRPC + minimal FastAPI health)

• Specifies: REST versioning (/api/v{n}, X-API-Version), categorized error codes (AUTH/AUTHZ/VAL/RES/RATE/SYS), rate-limit headers, pagination, idempotency, security headers, ETag/cache, WebSocket message types, OpenAPI 3.x, /api/v1/health.
• Verified (systemic greps over src/…/gateway/ and whole src/):
  • ❌ No REST API matching the spec. The gateway is a pure JSON-RPC 2.0 over WebSocket server (gateway/server.py:1–4 docstring). Zero hits in gateway/ for: X-RateLimit* (0), /api/v1/X-API-Version (0), AUTH-/VAL-/RES-/RATE-/SYS- codes (0), /health|/ready|/metrics (0), X-Content-Type-Options|X-Frame-Options (0), per_page|pagination|has_next (0), Idempotency-Key (0).
  • ✅ Health endpoint exists but divergent path — FastAPI router in postman/health.py (create_health_router:167, routes /health/live, /health/ready, /health/checks), wired into a FastAPI app at cli.py:43 (app.include_router(create_health_router())). Satisfies the intent of FR-018 but not the shape (/api/v1/health w/ version+uptime+ per-dep checks). refs create_health_router → def+import+2 use (wired).
  • ⚠️ Rate limiting exists but different concern — providers/rate_limiter.py, providers/router.py throttle outbound LLM provider calls, not API ingress. Not FR-008/009.
  • ⚠️ Error codes — the gateway uses JSON-RPC error codes (gateway/exceptions.py RPCError), a structured scheme, but not the spec's HTTP AUTH-/VAL-/RES- taxonomy.
  • ⚠️ fastapi, uvicorn, websockets, prometheus-client are declared (pyproject.toml), so the substrate for a REST/metrics surface exists but is unused for the spec's purposes (fastapi only in cli.py, postman/{health,admin}.py, kernel/cli_main.py).
• Remaining work / plan:
  • Scope: decide product intent. Path A (recommended): rewrite SPEC-098 to document the real transports — WS JSON-RPC (gateway), gRPC (VizInt pipeline), and the FastAPI health/admin surface — and drop the REST/versioning/idempotency/pagination sections as not-applicable. Path B (if a public REST API is a real goal): large greenfield — versioned FastAPI app, error-code middleware, ingress rate-limit middleware + X-RateLimit-*, security-header middleware, idempotency store, OpenAPI.
  • Target files: specs/098-api-standards/spec.md (Path A); or new src/…/api/ package (Path B).
  • Approach: Path A is spec reconciliation (aligns with SPEC-101/014's WS+gRPC design). Path B duplicates transports the system already has — only justified if external partners need REST.
  • Dependencies: couples to SPEC-014 (gateway/JWT), SPEC-099 (auth codes), SPEC-100 (/metrics, error schema), VizInt gRPC cluster. Security headers would land on the existing FastAPI health app.
  • Effort: S (Path A) / L (Path B). Risk: Path A low; Path B high (new attack surface, auth, duplication).
• Evidence log: gateway pattern greps (all 0); gateway/server.py docstring; postman/health.py routes + cli.py:43 wiring; refs create_health_router; providers/rate_limiter.py; fastapi dep + usage grep.

SPEC-099 Security & Auth Framework — GAP (only JWT token-auth + PAM++ authz + basic audit exist; the framework is unbuilt)

• Specifies: centralized auth (API-key/OAuth2/session-JWT), MFA (TOTP+backup codes), session mgmt (24h/7d refresh), token blacklist/revocation (Redis), AES-256-GCM encryption
  • monthly key rotation, bcrypt password hashing + policy, account lockout + progressive delay, tamper-resistant audit hash chains + 7yr retention, PAM++ authorization.
• Verified (systemic greps over whole src/):
  • ✅ JWT token auth — gateway/auth.py TokenAuthenticator (:16) create/validate HS256 with expiry (create_token:46, validate_token:91). This is the SPEC-014 gateway slice (@traces spec="014"), the only realized authentication.
  • ✅ PAM++ authorization / deny-by-default — governance/resource_access.py:304–309 ("Default deny", reason="default_deny"), governance/ethical.py:104 check(). Satisfies FR-008/009/011 via the governance subsystem (Prolog via janus).
  • ✅ Audit logging exists — kernel/audit/logger.py AuditLogger:81 (Redis-queued, log_message_routing/policy_query/escalation); gateway/audit.py AuditWriter; orchestrator/neo4j_audit.py AuditEvent:104. Covers FR-027/028 partially.
  • ❌ MFA/TOTP (FR-004/005/037–040): 0 hits for pyotp|TOTP|mfa; pyotp/qrcode not declared.
  • ❌ Password hashing + policy (FR-018/032/033/036): 0 hits for bcrypt|passlib|password_hash; passlib/bcrypt not declared (spec's own deps table lists them).
  • ❌ Encryption at rest / key rotation (FR-021/022): 0 hits for AES|Fernet|encrypt|cryptography; cryptography not declared.
  • ❌ Token blacklist / revocation (FR-019/020/035): 0 hits for blacklist|revoke.
  • ❌ Account lockout / progressive delay (FR-025/026/034): 0 hits for failed_login|lockout|failed_attempts.
  • ❌ OAuth2 / refresh tokens / server sessions (FR-002/003/013–017): 0 hits for OAuth|oauth2|refresh_token; no session store as specified.
  • ❌ Audit hash chain (FR-029): 0 hits for previous_hash|calculate_hash|verify_hash in the audit modules — no tamper-resistance. (The class-diagram calculate_hash is source-derived, not ported.)
• Remaining work / plan (large, mostly greenfield — surface a scoping decision first):
  • Scope: this is a full subsystem, not a port gap. Realistic minimal slice for a local tool: (1) API-key issuance+bcrypt-hash+revocation on the JWT path; (2) audit hash-chain (cheap, high-value tamper-evidence on the existing AuditLogger). MFA/OAuth/encryption/ key-rotation/session-mgmt are only justified if the tool becomes multi-user/hosted.
  • Target files: extend gateway/auth.py + a new security/ package; kernel/audit/ (hash chain); pyproject.toml (add passlib/bcrypt, cryptography, pyotp only when building the relevant slice).
  • Approach: do not blind-build the whole spec. Confirm the deployment model (local-single-user vs hosted) before investing. Then build API-key+audit-chain first.
  • Dependencies: SPEC-014 (owns the JWT authenticator — 099 must not fork it), SPEC-098 (auth error codes), governance cluster (authz already done), SPEC-100 (audit/security logging). Cross-check: gap-register already records SPEC-014 as a divergence ("single JWT TokenAuthenticator, not pluggable IAuthStrategy") — 099's strategy/MFA ambitions inherit that same unbuilt-structure reality.
  • Effort: L (full spec) / M (API-key+audit-chain slice). Risk: high — security code, must not ship half a framework (a partial lockout/MFA is worse than none).
• Evidence log: gateway/auth.py full read; capability greps (all 0); governance/resource_access.py:304; kernel/audit/logger.py; dep-absence checks (pyotp/passlib/cryptography/bcrypt/qrcode all ABSENT).

SPEC-100 Logging & Exception Handling — PARTIAL (structured logging + BaseError + circuit-breaker DONE; OTel tracing, retry/bulkhead, unified taxonomy, /metrics endpoint MISSING; structlog+UltimateBaseError are DIVERGENCES)

• Specifies: structlog JSON logging, UltimateBaseError hierarchy + full taxonomy, OpenTelemetry distributed tracing, Prometheus metrics at /metrics, error-handling patterns (retry+backoff, circuit breaker, graceful degradation), contextvars propagation, env-var config (LOG_LEVEL, TRACING_ENABLED, METRICS_PORT).
• Verified (systemic):
  • ✅ Structured JSON logging + correlation IDs + contextvars — core/logging.py StructuredFormatter (:53), configure_logging (:114), contextvars correlation_id/session_id/volk_id (:17–19). Satisfies FR-001/006. 324 files use stdlib logging.
  • ⚠️ DIVERGENCE — stdlib logging, not structlog. Spec Impl-Req §1 "MUST use structlog." structlog>=24.1.0 is declared but imported by 0 files; the project uses stdlib logging + a custom JSON StructuredFormatter. The code author mapped SPEC-100 to this (@traces spec="SPEC-100" on StructuredFormatter.format and BaseError). structlog is a declared-but-unused dependency (check-deps only flags undeclared, so it passes).
  • ⚠️ DIVERGENCE — no unified UltimateBaseError. core/base/error.py BaseError:10 provides the capability (message/error_code/context/timestamp/to_dict w/ __cause__ chain) and is @traces-mapped to SPEC-100. But the top-level exceptions.py is empty by design ("Legacy re-exports removed to eliminate upward layer imports"), and each subpackage defines its own base inheriting Exception directly — VolkRuntimeError, CXPError, GatewayError, GovernanceError, PostmanError, SessionError, KnowledgeGraphError, LLMClientError, … — not BaseError. The spec's named taxonomy is largely absent: of {Configuration, Validation, Authentication, Authorization, Resource, Integration, NotFound, Conflict, RateLimit, Timeout, CircuitBreakerOpen, System}Error, only 3 exist and each subclasses its package base (cxp ValidationError(CXPError), gateway AuthenticationError(GatewayError), postman ConfigurationError(PostmanError)) — coincidental names, not the unified tree. This decentralization is intentional (avoids upward coupling).
  • ✅ Circuit breaker (FR-005) — postman/error_handler.py:89 + providers/circuit_breaker.py:31 (Redis-backed). Two implementations → duplicate/consolidation candidate.
  • ❌ GAP — OpenTelemetry distributed tracing (FR-003). opentelemetry-* (4 packages) declared but imported by 0 files in the entire repo (src+tests+scripts). core/tracing/ is spec-traceability (@traces decorators, RTM), not runtime distributed tracing. No trace/span propagation, no trace_id injection into logs.
  • ❌ GAP — retry+backoff & graceful-degradation & Bulkhead (FR-005). No retry_with_backoff, no graceful_degradation decorator, no RetryStrategy, no Bulkhead (0 symbols each). No tenacity/backoff lib. Only a GRACEFUL_DEGRADATION enum value in kernel/performance/sla.py:23.
  • ⚠️ PARTIAL — Prometheus metrics (FR-004). Collection exists — postman/metrics.py (Counter/Gauge/Summary, generate_latest():313) — but it is not exposed at a /metrics HTTP endpoint; only create_health_router is mounted (cli.py:43). Metrics text is returned by a method, never served.
  • ⚠️ DIVERGENCE — env-var names. Config uses ORCHESTRATA_LOG_LEVEL (gateway/cli.py:89), not the spec's LOG_LEVEL; no TRACING_ENABLED/METRICS_PORT.
• Remaining work / plan (this spec has the most actionable systemic conformance work):
  • Scope: (1) spec reconciliation — record structlog→stdlib + UltimateBaseError→ per-package-BaseError as deliberate divergences (or, if a unified tree is wanted, reparent every package base onto BaseError — a wide but mechanical change gated by refs); (2) remove or wire the 4 dead OTel deps — either delete from pyproject.toml or build real tracing; (3) expose /metrics on the existing FastAPI app (mount alongside create_health_router); (4) optionally add retry_with_backoff/graceful_degradation helpers + consolidate the two CircuitBreakers.
  • Target files: specs/100-*/spec.md; pyproject.toml (drop unused otel/structlog or add tracing); cli.py + postman/metrics.py (/metrics route); a new core/patterns/retry.py; consolidate postman/error_handler.py ↔ providers/circuit_breaker.py.
  • Approach: cheapest high-value = mount /metrics (S) + prune dead deps (S). Unified-taxonomy reparenting is M and must use refs/rename (token-precise) — do NOT hand-edit.
  • Dependencies: constrains every cluster — all new code must log via core.logging and raise *Error (package base), NOT structlog/UltimateBaseError. Couples to SPEC-098 (/metrics, error schema) and SPEC-099 (security logging).
  • Effort: M overall (S for /metrics + dep-prune; M for taxonomy/retry). Risk: low–med (taxonomy reparent touches many files but is mechanical).
• Evidence log: core/logging.py read; structlog/otel import greps (0 in repo); core/base/error.py read; per-package class *Error(Exception) grep; spec-exception existence counts; postman/metrics.py:9,313; cli.py:43; retry/graceful greps; providers/circuit_breaker.py + postman/error_handler.py.

SPEC-101 Ultimate Architecture — DONE (structural/constitutional invariants hold; component feature-depth delegated to per-cluster specs)

• Specifies: the umbrella hybrid architecture — CheetahClaws TUI (U:Cd) + Orchestrata kernel (CXP) + VizInt web UI + OpenRouter + Todoist + checkpoints + Neo4j KG + PAM++ governance; constitutional structural rules (absolute imports, layered structure, deny-by-default governance, audit trail).
• Verified:
  • ✅ Core components exist and are class-verified — UltimateOrchestrator (orchestrator/orchestrator.py:214, extends Orchestrator), OrchestrataKernel (kernel/kernel.py:185), CodeDirectorAgent (agents/volks/codedirector/volk.py:394, wraps CodeDirectorVolk), QualityAssuranceAgent (agents/volks/qa/volk.py:1029), UserExperienceAgent (agents/volks/ux/volk.py:238); ArchitectAgent (U:Ar) present under agents/volks/architect/. The layered packages all exist: agents, kernel, cxp, orchestrator, governance, knowledge_graph, gateway, providers, integrations.
  • ✅ Constitutional rules enforced & passing — python scripts/serialize_codebase_ast.py standards → 0 Key-Rule violations; check-deps --include-tests → all declared. The canonical checklist lives in core/standards.py (injected into codegen prompts): rule 1 absolute-imports-only, rule 2 no-mocking, etc. (core/standards.py:14–20+).
  • ✅ Governance deny-by-default (Rule Precedence / Governance Flow) — governance/resource_access.py:304 "Default deny". PAM++ on Prolog-via-janus (per project memory) — matches SPEC-101 "SWI-Prolog (PAM++)" intent (mechanism = janus adapter, not pyswip; documented divergence).
  • ⚠️ Note (not a J-cluster gap): SPEC-101 is an umbrella — depth of individual components (Todoist bidirectional sync, VizInt UI, OpenRouter routing, checkpoint/rollback, multi-agent-edit rollback FR-005.1, large/binary-file handling FR-001.1/.2) is owned by other specs/clusters. Several of those are known-partial in docs/spec-gap-register.md (e.g. SPEC-012 PM budget, SPEC-023 goal-system consolidation, gateway health/plugin CLI). The architecture-level invariants themselves hold.
  • ⚠️ The X-API-Version/REST versioning strategy in SPEC-101 §"API Versioning Strategy" inherits SPEC-098's divergence (no REST API) — update alongside 098.
• Remaining work / plan: none at the constitutional/structural level. Action = keep the standards+check-deps gates green (already CI-shaped) and let per-cluster specs carry component feature work. Optionally reconcile the embedded REST-versioning/pyswip mentions with reality (rolls into SPEC-098/governance spec edits).
• Evidence log: symbols for the 6 core classes; standards (0 violations) + check-deps (clean); core/standards.py head; governance/resource_access.py:304; cross-ref docs/spec-gap-register.md.

Cluster summary table

Note on gates: standards (0 violations) and check-deps (clean) are green — but they gate structure/undeclared-imports, not the aspirational capabilities above. Two declared- but-unused dependency families (structlog, opentelemetry-*) pass check-deps yet represent unbuilt SPEC-100 features — a blind spot the gates do not catch.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks