Orchestrata Ultimate › Plans › spec-audit › reverify-2026-10-18 › J-standards-arch

Cluster J — Standards & Architecture (SPEC-097 … SPEC-101) — Re-verification 2026-10-18

Source: plans/spec-audit/reverify-2026-10-18/J-standards-arch.md at 8a2cb42ad · last changed 2026-10-18 · first committed 2026-10-18 · 319 lines.

Prose pending.

Plan text

Cluster J — Standards & Architecture (SPEC-097 … SPEC-101) — Re-verification 2026-10-18

Re-verification agent J. READ-ONLY, evidence-by-code-only, against the CURRENT tree (components branch, 337 commits past the 3-month-old session-195 audit). Prior audit confirmed/updated, not redone: plans/spec-audit/J-standards-arch.md (session 195), plans/spec-audit/00-MASTER-PLAN.md §2/§6 (S3 Gatekeeper, S8 Scribe, D4/D5).

Method note: Python structure read exclusively via the AST toolkit (scripts/serialize_codebase_ast.py symbols/refs/source/check-deps/trace-lint), per the repo mandate. check-deps and trace-lint parse the live tree. The Key-Rule gate.py run gate reads a Prolog fact base and is currently INVALID (stale snapshot — src/orchestrata_ultimate/tools/__init__.py not_in_snapshot), so its "0 violations" verdict from session 195 could NOT be re-confirmed without an infra write (uv run ast-sync, out of scope for this read-only pass).

Headline (what changed since session 195):

1. K1 landed. The gateway now has a real HTTP surface (gateway/http_surface.py: /health, /api/v1/health, /ready, /metrics, CORS, security headers, X-Request-ID), wired in gateway/server.py:916–923. This closes SPEC-098's "no HTTP surface" finding and SPEC-100's "/metrics not exposed" finding.
2. SPEC-098 was re-written in-tree. Its spec.md now carries a 2026-08-09 APPLICABILITY ruling: 9 FRs APPLIED (error taxonomy, security headers, request-id, health, CORS, token auth, message types) and 13 REST-only FRs marked N/A. So 098 moved from "DIVERGENCE/GAP" to a recorded, implemented divergence.
3. D4 landed. structlog + the four opentelemetry-* packages are gone from pyproject.toml; check-deps --include-tests is green, which proves zero imports of them anywhere in src+tests+scripts.
4. SPEC-099 is untouched. Spec text still the full aspirational SaaS framework (MFA/OAuth/AES-256-GCM/bcrypt/lockout/hash-chain/7-yr retention). checklists/ still MISSING. Zero @traces(spec="099") exist (task #142's dangling-id situation was resolved by removing/retargeting the traces — JWT now traces SPEC-014 — not by adding the checklist). D5's re-spec (S8 Scribe) is still owed.
5. Test-review.py grew T1→T20 (was T1–T15 in session 195; T15/T16 xdist now split into usage + import). The stale --dist=loadfile addopt from the session-195 audit is gone.

SPEC-097 Testing Standards — PARTIAL (enforcement DONE+expanded; spec text DIVERGENT)

• Specifies: test templates (mock/faker/freezegun-based), coverage thresholds, quality gates, tests/unit|integration|e2e/ layout, CI.
• Verified:
  • ✅ Quality-gate enforcement real and STRICTER than the spec — scripts/test-review.py enumerates T1–T19 in a single rule dict at test-review.py:228–246 and a separate T20 (@pytest.mark.anyio forbidden) at test-review.py:865–878. Rules include T1 "No mocking" (:228), T2 "No stubs/fakes" (:229), T4 "Function-scope fixtures only" (:231), T15 "No pytest-xdist" (:242), T16 "No banned-library imports" (:243). Dispatch: _check_no_xdist at test-review.py:664.
  • ✅ No-xdist enforced at the config layer too — pyproject.toml:388 carries the comment "pytest-xdist is BANNED"; it is absent from the dev-dependency list (pyproject.toml:382–390 lists pytest/pytest-asyncio/pytest-cov/pytest-timeout only). The session-195 --dist=loadfile addopt is gone from addopts (pyproject.toml:268–277).
  • ✅ Batched runner canonical — scripts/run_tests.py docstring (:1–24): "ThreadPoolExecutor dispatches 4 subprocesses … ~244 test modules … 4 times instead of 978", ORCHESTRATA_WORKER_SLOT per subprocess for Redis isolation.
  • ✅ Markers registered & strict — pyproject.toml:284–299: unit, integration, e2e, requires_redis, requires_graph, requires_prolog, isolation, slow, etc.; addopts = --strict-markers --strict-config (:268–270).
  • ⚠️ DIVERGENCE — spec templates contradict the enforced rules. Spec §2.1–2.3 (specs/097-testing-standards/spec.md:855–1030) use unittest.mock.Mock/@patch, freezegun, faker — all forbidden by T1/T8/T16. Spec §1.4/§5 mandate hypothesis/faker (spec.md:844–847, spec.md:1084–1094).
  • ❌ GAP — coverage thresholds unenforced. pytest-cov is declared (pyproject.toml:385) but there is no fail_under gate; run_tests.py does not --cov. FR-002's 90/85/80/70% targets are aspirational, not gated.
  • ❌ GAP — property-based libs. hypothesis/faker/freezegun are not declared (pyproject dependency grep returns only fastapi/uvicorn/prometheus-client of the probe set). faker/freezegun are deliberately-banned-adjacent; hypothesis is a genuine small gap.
  • ⚠️ ORPHAN — ParallelTestExecutor (core/threading/test_executor.py:1435; re-exported core/threading/__init__.py:4). refs ParallelTestExecutor → 1 def, 5 imports, 15 uses — all in-file (test_executor.py:1740,1769) + tests; 0 production callers outside its own module, and scripts/run_tests.py rolls its own batching. Still the spec's "Test Runner Interface" run_tests that nothing calls.
• Remaining work / plan:
  • Scope: (a) rewrite SPEC-097 to the actual philosophy (no-mock, real-infra, domain layout + markers, T1–T20, batched runner) — this collapses the "gap" to spec-debt; (b) optional fail_under coverage gate; (c) optional hypothesis; (d) decide ParallelTestExecutor keep-vs-delete.
  • Target files: specs/097-testing-standards/spec.md; pyproject.toml ([tool.coverage.report] fail_under, add hypothesis); scripts/run_tests.py (--cov).
  • Approach: documentation reconciliation + two additive tooling knobs. No behaviour change to enforcement (already complete).
  • Dependencies: constrains every other cluster (T1–T20 are the rules all clusters' tests obey). No inbound deps.
  • Effort: S. Risk: low.
• Evidence log: source test-review --source scripts (T1–T19 @228–246, T20 @865–878, _check_no_xdist @664); sed pyproject.toml addopts/markers/xdist-ban; source run_tests --source scripts header; symbols --pattern 'hypothesis|faker|freezegun' (0); refs ParallelTestExecutor; grep pyproject.toml for hypothesis/faker/freezegun (absent).

SPEC-098 API Standards — DIVERGENCE (spec re-written; 9 FRs implemented+tested; 13 N/A recorded)

• Specifies: REST versioning/error codes/rate limits; re-scoped 2026-08-09 to the gateway's WS-JSON-RPC + 3-HTTP-endpoint reality.
• Verified:
  • ✅ HTTP surface exists & wired — gateway/http_surface.py build_gateway_http_app (:32) exposes /health, /api/v1/health, /ready, /metrics; refs build_gateway_http_app → USE at gateway/server.py:916 (production-wired).
  • ✅ FR-006/FR-007 error taxonomy — gateway/error_taxonomy.py ErrorCategory(StrEnum) with AUTH/AUTHZ/VAL/RES/RATE/SYS (:29–46) + error_data() builder (:72).
  • ✅ FR-016 security headers — gateway/http_surface.py _spec098_headers sets X-Content-Type-Options: nosniff, X-Frame-Options: DENY (:77–99).
  • ✅ FR-017 request tracing — same middleware echoes/mints X-Request-ID (:86–92).
  • ✅ FR-018 versioned health — /api/v1/health alias route (:100–111).
  • ✅ FR-022 CORS — CORSMiddleware with allow_origins/allow_headers incl. X-Request-ID (:60–76).
  • ✅ FR-019 token auth — gateway/server.py:1285 user_id = await authenticate(websocket, …); JWT TokenAuthenticator at gateway/auth.py:19 (create_token:41, validate_token:81).
  • ✅ FR-020 message types — gateway/models.py:27 MessageType enum documents the FR-020 mapping (event/error identical; ping/pong = HEARTBEAT; subscribe/unsubscribe are METHODS) at :27–75.
  • ✅ FR-010 request auth — same TokenAuthenticator path.
  • ✅ Tested — tests/gateway/test_spec098_api_standards.py exists (spec-098-specific suite).
  • ⚠️ 13 FRs marked N/A in the spec itself (specs/098-api-standards/spec.md:40–68): version-in-path/header, MAJOR-version compat, Deprecation/Sunset, X-RateLimit-*, JSON-schema POST validation, data+metadata envelope, pagination, Idempotency-Key, ETag, OpenAPI. Recorded decision, not an omission.
• Remaining work / plan:
  • Scope: none at the code level. The divergence is recorded in-tree and the applicable FRs are implemented and tested. Optional S: reconcile the stale class-diagram tail of the spec (it still shows a REST-shaped ParallelTestExecutor/CircuitBreaker/Bulkhead diagram) and the REST-only examples.
  • Target files: specs/098-api-standards/spec.md (optional diagram/examples trim).
  • Approach: doc-only.
  • Dependencies: couples SPEC-014 (owns http_surface + JWT), SPEC-099 (auth codes), SPEC-100 (/metrics, error schema).
  • Effort: S (optional) / effectively done. Risk: low.
• Evidence log: source gateway.http_surface (full read); source gateway.error_taxonomy; refs build_gateway_http_app; symbols --pattern MessageType; source gateway.models (:27–75); source gateway.server (:1285, :916–923); symbols --pattern TokenAuthenticator; spec.md APPLICABILITY block read.

SPEC-099 Security & Auth Framework — GAP (unchanged; only JWT+PAM++ authz + audit exist; checklist missing; D5 re-spec not done)

• Specifies: centralized auth (API-key/OAuth2/session-JWT), MFA (TOTP+SMS+backup codes), session mgmt (24h/7d), token blacklist/revocation, AES-256-GCM + monthly rotation, bcrypt hashing + policy, lockout + progressive delay, hash-chained 7-yr audit, PAM++ authz.
• Verified:
  • ✅ JWT token auth — gateway/auth.py:19 TokenAuthenticator (HS256, expiry, create_token:41, validate_token:81). This is the SPEC-014 slice; the class carries no @traces(spec="099").
  • ✅ PAM++ deny-by-default authz — governance/resource_access.py:250–255 # Default deny → reason="default_deny"; ResourceAccessController (:35) + protocol IResourceAccessController (governance/interfaces.py:146).
  • ✅ Audit logging — kernel/audit/logger.py:44 AuditLogger (Redis-queued, log_message_routing/log_policy_query/log_escalation/log_workflow_event, query APIs); gateway/audit.py:72 AuditWriter.
  • ❌ MFA/TOTP (FR-004/005/037–040) — symbols --pattern 'TOTP|MFA|pyotp' → 0 src hits; pyotp/qrcode not declared.
  • ❌ Password hashing + policy (FR-018/032/033/036) — symbols --pattern 'bcrypt' → 0; passlib/bcrypt not declared.
  • ❌ Encryption at rest / key rotation (FR-021/022) — symbols --pattern 'AES|Fernet' → 0; cryptography not declared.
  • ❌ Token blacklist / revocation (FR-019/020/035) — symbols --pattern 'blacklist|revoke_token|refresh_token' → only IntentSecurity.blacklist (orchestrator/intent_security.py:363, a requester blocklist, NOT the spec's revoked:{token_hash} Redis blacklist). GAP.
  • ❌ OAuth2 / refresh tokens / server sessions (FR-002/003/013–017) — symbols --pattern 'OAuth|refresh_token' → 0.
  • ❌ Account lockout / progressive delay (FR-025/026/034) — symbols --pattern 'lockout|failed_login' → 0.
  • ❌ Audit hash chain (FR-029/031) — symbols --pattern 'calculate_hash|verify_hash' → 0.
  • ⚠️ Checklist MISSING (task #142 confirmed) — specs/099-security-auth-framework/ contains only spec.md + compliance-audit.md; no checklists/ directory.
  • ⚠️ Zero @traces(spec="099") in source — trace-lint --source src/orchestrata_ultimate reports 13 findings (12 shotgun warnings + 1 shotgun error), no dangling-id, no unknown-spec, no spec-mismatch. Since 099 has no requirements.md, any REQ-099-* citation would be a dangling error → none exist. The earlier dangling-id state was resolved by retargeting traces (JWT→014, governance→007/060, audit→001/002), not by creating the checklist.
  • ⚠️ Spec text NOT re-scoped — D5 ("re-spec 099 down to JWT+PAM++ per D5", S8 Scribe) is unexecuted: specs/099-security-auth-framework/spec.md is still the 1.1.0 aspirational framework (OAuth/MFA/SQL tables/Redis-cluster/Vault/7-yr retention intact).
• Remaining work / plan:
  • Scope: D5 removed the security build, but the re-spec is still owed. Minimal work: (1) rewrite SPEC-099 to the realized JWT+PAM++ + audit slice and mark MFA/OAuth/crypto/ lockout/hash-chain as explicitly-out-of-scope for a local trusted tool; (2) create specs/099-*/checklists/requirements.md; (3) attach accurate @traces(spec="099") to the JWT/authz/audit code that genuinely implements 099 (or record it as living under 014/007). Optional S-M slice if multi-user/hosted is ever in scope: API-key issuance+bcrypt+revocation, audit hash-chain.
  • Target files: specs/099-security-auth-framework/spec.md (rewrite); specs/099-security-auth-framework/checklists/requirements.md (new); optionally extend gateway/auth.py + new security/ package.
  • Approach: spec reconciliation (S) first; any security code is a separate gated build, never a half-framework.
  • Dependencies: SPEC-014 (owns the JWT authenticator — 099 must not fork it), SPEC-098 (auth error codes), governance cluster (authz done), SPEC-100 (audit/security logging).
  • Effort: S (re-spec + checklist + traces) / M (API-key+audit-chain slice) / L (full spec — rejected by D5). Risk: low for the re-spec; high for any security build.
• Evidence log: ls specs/099-security-auth-framework/ (no checklists/); trace-lint --source src/orchestrata_ultimate (no dangling 099); symbols --pattern 'TOTP|MFA|pyotp|bcrypt|AES|Fernet|blacklist|revoke_token|refresh_token| lockout|failed_login|OAuth|calculate_hash|verify_hash' (all 0 except IntentSecurity.blacklist); source gateway.auth; source governance.resource_access (:250–255); symbols --pattern AuditLogger|AuditWriter; spec.md read (still 1.1.0 full framework).

SPEC-100 Logging & Exception Handling — PARTIAL (/metrics now exposed + dead deps pruned; UltimateBaseError/LOG_LEVEL divergence + retry/bulkhead/graceful gap remain)

• Specifies: structlog JSON logging, UltimateBaseError taxonomy, OTel tracing, Prometheus /metrics, retry+CB+graceful-degradation, contextvars, env-var config (LOG_LEVEL/TRACING_ENABLED/METRICS_PORT).
• Verified:
  • ✅ Structured JSON logging + contextvars — core/logging.py:56 StructuredFormatter (format:67); contextvars correlation_id/session_id/volk_id (:19–21); configure_logging (:111), new_correlation_id (:155), get_logger (:167).
  • ✅ /metrics now exposed (was a session-195 gap) — gateway/http_surface.py metrics() returns prometheus_client.generate_latest() (:124–130), served by serve_http and wired via build_gateway_http_app at gateway/server.py:917. Collector lives in postman/metrics.py (module present; tests/postman/test_metrics.py covers export_prometheus/export_json).
  • ✅ D4 dep-prune landed — structlog + opentelemetry-* absent from pyproject.toml; check-deps --include-tests → "All third-party imports are declared" (proves 0 structlog/ OTel imports anywhere).
  • ✅ Circuit breaker — postman/error_handler.py:33 CircuitBreaker + providers/circuit_breaker.py:47 CircuitBreaker (two implementations → consolidation candidate).
  • ⚠️ Retry+backoff partial — postman/error_handler.py:158 _calculate_backoff (exponential
    • jitter) + _get_circuit (:150), traced to SPEC-002. No standalone generic retry_with_backoff/RetryStrategy helper.
  • ⚠️ DIVERGENCE — stdlib logging, not structlog. configure_logging uses stdlib logging + StructuredFormatter; REQ-100-051 "MUST use structlog" is unmet-by-design (structlog now removed from deps, so this is a spec-text reconcile, not a dead dep).
  • ⚠️ DIVERGENCE — no UltimateBaseError. symbols --pattern 'UltimateBaseError' → 0. Capability lives in core/base/error.py:7 BaseError; each package defines its own base inheriting Exception: VolkRuntimeError (agents/exceptions.py:15), CXPError (core/packet/exceptions.py:26), GatewayError (gateway/exceptions.py:6), AuthenticationError(GatewayError) (gateway/exceptions.py:10), GovernanceError (governance/exceptions.py:8), PostmanError (postman/exceptions.py:26), ConfigurationError(PostmanError) (postman/exceptions.py:292). Named taxonomy is coincidental, not the unified tree.
  • ⚠️ DIVERGENCE — env-var names. gateway/cli.py:96–98 reads ORCHESTRATA_LOG_LEVEL + ORCHESTRATA_JSON_LOGS, not REQ-100-034's LOG_LEVEL/LOG_FORMAT; no TRACING_ENABLED/METRICS_PORT.
  • ❌ Tracing GAP (recorded) — D4 removed OTel; core/tracing/ is spec-traceability (@traces), not runtime distributed tracing. No trace/span propagation. (Recorded decision to prune, not a silent gap.)
  • ❌ Graceful-degradation + Bulkhead GAP — symbols --pattern 'graceful_degradation| RetryStrategy|Bulkhead' → 0 src hits (only a test name + the GRACEFUL_DEGRADATION enum value in kernel/performance/sla.py).
• Remaining work / plan:
  • Scope: (1) spec reconcile — record stdlib-logging + BaseError-not-UltimateBaseError + ORCHESTRATA_LOG_LEVEL as deliberate divergences (or reparent package bases onto BaseError — wide, mechanical, must use refs); (2) optionally add generic retry_with_backoff/ graceful_degradation helpers + consolidate the two CircuitBreakers; (3) tracing stays deferred per D4 (re-add only when wired).
  • Target files: specs/100-logging-exception-handling/spec.md; a new core/patterns/retry.py; consolidate postman/error_handler.py ↔ providers/circuit_breaker.py.
  • Approach: cheapest high-value = spec reconcile (S). Taxonomy reparent = M, refs-gated.
  • Dependencies: constrains every cluster (log via core.logging, raise per-package *Error); couples SPEC-098 (/metrics, error schema), SPEC-099 (security logging).
  • Effort: M overall (S reconcile; M taxonomy/retry). Risk: low–med (taxonomy reparent is wide but mechanical).
• Evidence log: source core.logging (:19–21,:56,:111); source gateway.http_surface (:124–130); refs build_gateway_http_app; check-deps --include-tests (green); grep pyproject.toml (no structlog/opentelemetry); symbols --pattern 'UltimateBaseError| RetryStrategy|Bulkhead|graceful_degradation' (0); symbols --pattern 'GatewayError|CXPError| GovernanceError|PostmanError|VolkRuntimeError|AuthenticationError' (per-package bases); source gateway.cli (:96–98); source postman.error_handler (:150,:158).

SPEC-101 Ultimate Architecture — DONE (constitutional invariants hold; component depth delegated)

• Specifies: umbrella hybrid — CheetahClaws TUI + Orchestrata kernel (CXP) + VizInt UI + OpenRouter + Todoist + checkpoints + Neo4j KG + PAM++; constitutional structural rules.
• Verified:
  • ✅ Core components exist — UltimateOrchestrator (orchestrator/orchestrator.py:207, extends Orchestrator), OrchestrataKernel (kernel/kernel.py:131), CodeDirectorAgent (agents/volks/codedirector/volk.py:628), QualityAssuranceAgent (agents/volks/qa/volk.py:1232), UserExperienceAgent (agents/volks/ux/volk.py:236). Layered packages all present (agents, kernel, cxp, orchestrator, governance, knowledge_graph, gateway, providers, integrations).
  • ✅ Constitutional rules enumerated in-tree — core/standards.py CODING_STANDARDS (:12–20+): absolute-imports-only, NO_MOCKING, dependency declaration, flat src-layout, etc. (injected into codegen prompts).
  • ✅ Governance deny-by-default — governance/resource_access.py:250–255 reason="default_deny".
  • ✅ Live gates green where live-checkable — check-deps --include-tests → all declared; trace-lint → 1 shotgun error + 12 shotgun warnings, no dangling/unknown-spec/ spec-mismatch (constitutional trace hygiene holds).
  • ⚠️ gate.py run (Key-Rule) currently INVALID, not green — fact base stale: src/orchestrata_ultimate/tools/__init__.py (not_in_snapshot). Needs uv run ast-sync (infra write, out of scope read-only). Session-195's "0 Key-Rule violations" could not be re-confirmed this pass; the stale-1-file delta is the only blocker, no rule violation shown.
  • ⚠️ Umbrella note (not a J gap): component feature-depth (Todoist sync, VizInt UI, OpenRouter routing, checkpoint/rollback FR-005.1, large/binary-file FR-001.1/.2) is owned by other clusters. The embedded REST-versioning/X-API-Version mention in §"API Versioning" inherits SPEC-098's divergence (recorded, now N/A).
• Remaining work / plan: none at the constitutional/structural level. Action = keep check-deps/trace-lint green, re-sync the fact base so gate.py run returns to a green verdict (maintenance, not a spec gap), and let per-cluster specs carry component feature work.
• Evidence log: symbols --pattern 'UltimateOrchestrator|OrchestrataKernel|CodeDirectorAgent| QualityAssuranceAgent|UserExperienceAgent'; source core.standards; check-deps --include-tests (green); trace-lint (no constitutional errors); gate.py run (INVALID — stale fact base); source governance.resource_access (:250–255).

Cluster summary table

Gates note: check-deps and trace-lint are green live. gate.py run (Key-Rule) is INVALID pending an ast-sync fact-base refresh (tools/__init__.py delta) — a maintenance item, not a rule violation. The two dead-dependency families that session 195 flagged (structlog, opentelemetry-*) are now removed per D4, so the "declared-but-unused passes check-deps" blind spot is closed for them.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks