Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate / Specs / 031-qa-volk

031-qa-volk: QA Volk (U:Qa)

The QA Volk (U:Qa) is the quality assurance agent in the Orchestrata multi-agent pipeline. It receives code artifacts from U:Cd (CodeDirector) and performs a full review-and-test pipeline: deterministic auto-fixing, three sequential review gates, LLM-based code review against a 31-item Google-style checklist, test generation, test execution, and fix delegation back to U:Cd on any failure. The cycle runs without a cap until all issues are resolved (G226), with escalating context on repeat...

At a glance

Item Value
Spec directory specs/031-qa-volk/
Status line in the spec Implemented
Requirement IDs defined 55
With a @traces citation in src/ 42 (76.4%)
With a @traces citation in tests/ 6 (10.9%)
Tracked files 7
RTM page 031-qa-volk

Reading this spec

The QA Volk (U:Qa) is the quality assurance agent of the pipeline. It receives code artifacts from the CodeDirector and runs a full review-and-test pipeline: deterministic auto-fixing, three sequential review gates, an LLM code review against a 31-item Google-style checklist, test generation and test execution. The spec's reasoning is that LLM-generated code has predictable defects (wrong async idioms, aioredis API misuse, relative imports, hardcoded Redis hosts, missing stdlib imports) that are fixed mechanically before an LLM review spends tokens on them, that structural gates must be deterministic rather than delegated to an LLM, and that test generation is separate from code generation so one agent cannot mark its own work complete.

The pipeline has nine steps. First, 13 deterministic AST and regex fixers run. Gate 1 checks syntax (every file compiles under Python 3.13). Gate 2 runs flake8 critical codes (E9xx and F8xx). Gate 3 is the checklist of traceability, import order, docstring and structure checks. Import validation imports each source module in the workspace virtual environment to catch hallucinated APIs. The LLM review follows, then test generation (a pytest suite with zero mocking, @traces decorators and real infrastructure fixtures) and test execution. Any failure bounces back to U:Cd; on cycle 3 and later, the history of failed attempts is included so the same broken strategy is not repeated. When everything passes, the work is committed in git and forwarded to U:Cp with the subject compliance_check. The pipeline does not end until tests pass. Requirement IDs use the REQ-031- prefix.

In the code, the largest tracers are orchestrata_ultimate/agents/volks/qa/volk and orchestrata_ultimate/agents/volks/qa/_autofix, with the fixers in orchestrata_ultimate/verification/autofix/redis_fixes and orchestrata_ultimate/verification/autofix/import_fixes, and the checks in orchestrata_ultimate/agents/volks/codedirector/review_checks. The per-requirement view is on the RTM page.

The spec's status line reads Implemented. It tracks 7 files and defines 55 requirement IDs. The full text follows below.

Measured coverage is 42 of 55 requirements cited by at least one @traces in src/ (76.4%), spread over 9 modules; the module citing the most of them is orchestrata_ultimate/agents/volks/qa/volk (19 requirements). In tests/, 6 requirements are cited by a test-side @traces.

Coverage gaps: 13 requirements have no citing symbol. The largest untraced groups are REQ-031-* (13). Untraced means no @traces names the ID; the behaviour may still exist in code that was never annotated. Test-side citations are thinner than code-side ones (6 against 42).

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID Done Requirement Traced by
REQ-031-001 no _autofix_aioredis() rewrites import aioredis to import redis.asyncio as aioredis 3
REQ-031-002 no _autofix_aioredis() rewrites aioredis.from_url() to redis.asyncio.from_url() 3
REQ-031-003 no _autofix_sync_redis() rewrites synchronous redis.Redis() to redis.asyncio.Redis() in async contexts 4
REQ-031-004 no _autofix_f821_datetime() adds from datetime import datetime, timedelta when datetime is used as a bare name without an import 5
REQ-031-005 no _autofix_missing_stdlib_imports() detects missing stdlib imports by scanning for known stdlib names and adds the correct import statements 4
REQ-031-006 no _autofix_missing_barename_imports() adds missing third-party imports for known packages used as bare names 5
REQ-031-007 no _autofix_xadd_dict() serializes nested dict values in xadd calls using json.dumps() to produce flat string fields 2
REQ-031-008 no _autofix_hset_dict() rewrites hset(name, mapping={...}) values to JSON-serialized strings 2
REQ-031-009 no _autofix_redis_decode() adds decode_responses=True to Redis client constructors that omit it 2
REQ-031-010 no _autofix_redis_host() replaces hardcoded "localhost" in Redis constructors with core.redis_factory.resolve_redis_host() (raises when unset — RUL-003) 5
REQ-031-011 no _autofix_relative_imports() rewrites from .module import X to absolute import paths 5
REQ-031-012 no _autofix_line_length() splits lines exceeding 80 characters at logical boundaries without breaking semantics 3
REQ-031-013 no _autofix_import_order() reorders imports to: stdlib first, third-party second, local last, separated by blank lines 3
REQ-031-014 no _autofix_unused_imports() removes import statements for names that are never referenced in the file body 1
REQ-031-015 no All 13 auto-fixers operate on all .py files in self.workspace 11
REQ-031-016 no Auto-fix pass runs before Gate 1 (Syntax) on every cycle, including test-fix return cycles 2
REQ-031-017 no review_gate_syntax() (from review_checks.py) compiles every .py file in files_created using Python AST 1
REQ-031-018 no review_gate_syntax() returns a list of (code, filepath, description) tuples for each syntax error 1
REQ-031-019 no Gate 1 failure immediately calls _bounce_for_fix() with label "GATE 1 FAIL: Syntax errors" 1
REQ-031-020 no Gate 1 failure skips Gate 2, Gate 3, and all subsequent pipeline phases 1
REQ-031-021 no review_gate_lint() (from review_checks.py) runs flake8 with critical error codes on all files_created 2
REQ-031-022 no Gate 2 targets E9xx (syntax at runtime) and F8xx (undefined names, unused imports) flake8 codes none
REQ-031-023 no Gate 2 failure calls _bounce_for_fix() with label "GATE 2 FAIL: Lint/style errors" 1
REQ-031-024 no Gate 2 failure skips Gate 3 and all subsequent pipeline phases 1
REQ-031-025 no review_gate_checklist() (from review_checks.py) runs deterministic structural checks on all files_created 1
REQ-031-026 no Checklist gate verifies: traceability decorators present, import order correct, public functions have docstrings, no relative imports 4
REQ-031-027 no Gate 3 failure calls _bounce_for_fix() with label "GATE 3 FAIL: Checklist violations" 1
REQ-031-028 no get_fix_instructions() converts issue tuples to a structured, human-readable fix prompt 1
REQ-031-029 no All three gates log status to stdout: " [U:Qa] Gate N/3: Syntax/Lint/Checklist..." and " [U:Qa] Gate N/3: PASS" 1
REQ-031-030 no When all three gates return empty lists, the pipeline proceeds to import validation 2
REQ-031-031 no CODE_REVIEW_PROMPT contains exactly 31 numbered checklist items covering all specified categories none
REQ-031-032 no Items 1-6 cover Correctness: IMPORTS, NO_MOCKING, API_CORRECTNESS, FILE_COMPLETENESS, ENTRY_POINT, DEPENDENCY_DECLARATION none
REQ-031-033 no Items 7-9 cover Design: DESIGN, COMPLEXITY, NO_DUPLICATION none
REQ-031-034 no Items 10-12 cover Functions: FUNCTION_LENGTH, FUNCTION_ARGS, NO_PLACEHOLDER none
REQ-031-035 no Items 13-14 cover Error Handling: ERROR_HANDLING, EXCEPTIONS none
REQ-031-036 no Items 15-20 cover: RESOURCE_MANAGEMENT, NAMING, TYPE_HINTS, DEFAULT_VALUES, DOCSTRINGS, COMMENTS none
REQ-031-037 no Items 21-29 cover Style and Python Best Practices: LINE_LENGTH, FORMATTING, STRINGS, MUTABLE_GLOBAL_STATE, COMPREHENSIONS, TRUE_FALSE, ITERATORS, DECORATORS, CONCURRENCY none
REQ-031-038 no Items 30-31 cover SECURITY and TESTS none
REQ-031-039 no LLM review call uses temperature=0.0, volk="U:Qa", and injects self._graph_context 1
REQ-031-040 no _parse_review_verdict() extracts "PASS" or "FAIL" from the REVIEW_VERDICT: line in the LLM response 2
REQ-031-041 no _parse_review_issues() extracts actionable issues from the ISSUES: block when verdict is FAIL 2
REQ-031-042 no FAIL verdict triggers _bounce_for_fix() to U:Cd with parsed issues 1
REQ-031-043 no Review runs on every cycle, including when packet carries from_test_fix=True (G226) 2
REQ-031-044 no QA_SYSTEM_PROMPT forbids unittest.mock, MagicMock, patch, AsyncMock, Mock imports none
REQ-031-045 no QA_SYSTEM_PROMPT forbids custom Mock/Fake/Stub/Dummy classes none
REQ-031-046 no QA_SYSTEM_PROMPT requires pytest framework (no unittest.TestCase) none
REQ-031-047 no QA_SYSTEM_PROMPT forbids asyncio.Awaitable and asyncio.Coroutine; requires collections.abc.Awaitable none
REQ-031-048 no QA_SYSTEM_PROMPT requires @traces decorator on every test function 1
REQ-031-049 no Test prompt includes 6-phase structure: Analyze, Plan, Infrastructure Rules, Source Code, Output Constraints, Traceability 2
REQ-031-050 no Generated tests use real Redis with the host from core.redis_factory.resolve_redis_host() (raises when unset — RUL-003) and db=15 2
REQ-031-051 no _extract_and_write_tests() parses TEST_FILE blocks from LLM response and writes to self.workspace 2
REQ-031-052 no _validate_no_mocks() scans written test files for forbidden mock patterns 3
REQ-031-053 no _retry_without_mocks() re-runs LLM generation with anti-mock instruction when mock usage is detected 3
REQ-031-054 no Each test file is capped at MAX 120 lines; longer output is split across multiple files 1
REQ-031-055 no _run_tests() executes pytest in self.workspace using workspace venv Python and returns (passed: bool, output: str) 10

Code that traces this spec

Module Symbols Requirements
orchestrata_ultimate/agents/volks/codedirector/review_checks 9 7
orchestrata_ultimate/agents/volks/qa/_autofix 22 15
orchestrata_ultimate/agents/volks/qa/_testgen 10 4
orchestrata_ultimate/agents/volks/qa/volk 16 19
orchestrata_ultimate/verification/autofix/engine 2 3
orchestrata_ultimate/verification/autofix/format_fixes 6 3
orchestrata_ultimate/verification/autofix/helpers 1 1
orchestrata_ultimate/verification/autofix/import_fixes 11 5
orchestrata_ultimate/verification/autofix/redis_fixes 12 8

Checklists

checklists/README.md

QA Volk (U:Qa) Checklists

Project: Orchestrata QA Volk (Spec 031) Component: U:Qa (QA Volk) Status: Implemented Owner: Mohamed Package: orchestrata-qa

Overview

The QA Volk (U:Qa) is the quality assurance agent in the Orchestrata multi-agent code generation pipeline. It receives code artifacts from U:Cd (CodeDirector) via CXP review_and_test packets and executes a full review-and-test pipeline before forwarding to U:Cp (Compliance Volk).

The pipeline is deterministic-first: 13 auto-fixers correct mechanical LLM defects before any gate runs; three sequential gates (Syntax, Lint, Checklist) catch structural issues without spending LLM tokens; LLM review evaluates the code against a 31-item Google-style checklist; test generation produces a zero-mocking pytest suite; test execution validates real behavior. Any failure bounces back to U:Cd with escalating context (G229). The loop runs without a cap (G226) until all gates pass and all tests pass.

Checklist Summary
Checklist Total Items Completed Progress
Requirements 55 0 0%
Constitutional Compliance 20 0 0%
Testing Plan 55 0 0%
TOTAL 130 0 0%
Key Components
1. QaVolk Class
2. Auto-Fix Pass (13 fixers)

Deterministic, zero-LLM corrections run before all gates:

Fixer Defect Corrected
_autofix_aioredis aioredis import and from_url API
_autofix_sync_redis Synchronous redis.Redis in async code
_autofix_f821_datetime Missing datetime import
_autofix_xadd_dict Nested dicts in xadd calls
_autofix_hset_dict Nested dicts in hset mapping
_autofix_redis_decode Missing decode_responses=True
_autofix_missing_stdlib_imports Missing stdlib imports by name
_autofix_missing_barename_imports Missing third-party imports
_autofix_redis_host Hardcoded "localhost" in Redis
_autofix_relative_imports Relative import paths
_autofix_line_length Lines exceeding 80 chars
_autofix_import_order Out-of-order import blocks
_autofix_unused_imports Unreferenced import statements
3. Three-Gate Review Pipeline (G227)

Sequential deterministic gates; first failure bounces immediately:

Gate Mechanism Bounce Label
Gate 1: Syntax review_gate_syntax() -- AST compile check "GATE 1 FAIL: Syntax errors"
Gate 2: Lint review_gate_lint() -- E9xx/F8xx flake8 "GATE 2 FAIL: Lint/style errors"
Gate 3: Checklist review_gate_checklist() -- structural rules "GATE 3 FAIL: Checklist violations"

Note: review_gate_syntax, review_gate_lint, review_gate_checklist, and get_fix_instructions are imported from orchestrata.volks.codedirector.review_checks (SPEC-029 package).

4. LLM Code Review
5. Test Generation
6. Test Execution
7. Escalating Fix Context (G229)
Cycle Additional Context Injected
1-2 Current issues + fix instructions only
3+ History of last 3 failed attempts
5+ Root-cause analysis + strategy change instructions
8+ Full file contents for affected files
8. Escalation Routing
Condition Target Subject
Any gate fails U:Cd fix_issues
Test suite fails U:Cd fix_issues (with from_test_fix=True)
All gates pass + tests pass U:Cp compliance_check
Checklist Files
1. requirements.md

Functional requirements organized by user story:

2. constitutional.md

Compliance with the Orchestrata constitutional framework (20 items):

3. testing.md

Comprehensive testing strategy (55 tests):

Performance Targets
Metric Target
Auto-fix pass duration <2 seconds for 10-file workspace
Gate 1 (Syntax) <1 second per file
Gate 2 (Lint) <5 seconds for 10-file workspace
Gate 3 (Checklist) <3 seconds for 10-file workspace
LLM review call Governed by SPEC-028 retry/backoff
Test generation LLM call Governed by SPEC-028 retry/backoff
Test execution Bounded by pytest timeout in workspace
Integration Points
Upstream
Downstream
Governance Rules Referenced
Rule Description
G100 Do NOT regenerate test files on test-fix return -- re-run existing ones
G218 Capture requirement IDs from packet payload for @traces injection
G226 No cycle cap -- review loops until ALL issues resolved
G227 Three-gate sequential pipeline: Syntax, Lint, Checklist
G229 Fix attempt history tracking for escalation context
References

Last Updated: 2026-04-04 Version: 1.0.0 Maintained By: Mohamed

checklists/constitutional.md

QA Volk (U:Qa) -- Constitutional Compliance

Component: U:Qa (QA Volk) Spec: 031-qa-volk Status: Compliance Review Total Compliance Items: 20

Overview

This document tracks compliance of the QA Volk (U:Qa) with the Orchestrata constitutional framework. U:Qa operates at the quality gate of the code generation pipeline: it enforces correctness, tests real behavior, and escalates failures deterministically. Constitutional compliance focuses on autonomy preservation (it reviews but does not override), governance separation (gates are deterministic rules, not LLM opinions), real-system testing (zero mocking), audit trail completeness, and graceful degradation when LLM or infrastructure is unavailable.

Article I: Agent Autonomy

Principle: Volks retain autonomy; QA enforces standards without overriding agent decisions.

Compliance Requirements

Priority: Critical Verification: Code review of auto-fixer methods

Article II: Prolog as Single Source of Truth

Principle: All governance rules are in Prolog; QA does not duplicate or override governance logic.

Compliance Requirements

Priority: High Verification: Confirm no pyswip calls in qa/volk.py

Article VI: Python 3.14t Implementation

Principle: All Orchestrata code targets Python 3.14t (free-threaded).

Compliance Requirements

Priority: Critical Verification: Runtime import check in CI

Article VII: Testing Without Mocks

Principle: All tests use real implementations; mocking is not permitted.

Compliance Requirements

Priority: Critical Verification: NO_MOCKING checklist item + _validate_no_mocks() coverage

Article VIII: Metrics and Observability

Principle: All components emit observable state; QA must be inspectable.

Compliance Requirements

Priority: High Verification: Conversation transcript inspection from spike iterations

Summary
Article Items Status
I -- Agent Autonomy 5 To verify
II -- Prolog Governance 4 To verify
VI -- Python 3.14t 3 To verify
VII -- No Mocking 4 To verify
VIII -- Observability 4 To verify
TOTAL 20

checklists/requirements.md

The requirement list above comes from this file; 0 of 55 checkbox items are ticked.

checklists/testing.md

QA Volk (U:Qa) -- Testing Plan

Component: U:Qa (QA Volk) Spec: 031-qa-volk Status: Test Planning Phase Total Test Cases: 55

Overview

This document defines the comprehensive testing strategy for the QA Volk (U:Qa). All tests follow the strict no-mocking policy: real file I/O in tmp_path fixtures, real flake8 subprocess invocations, real pytest executions, and real Redis where infrastructure tests are involved. Tests verify the QA pipeline from auto-fixers through gate logic, LLM review parsing, test extraction, and CXP packet construction.

Test Categories Summary
Category Test Count Purpose Priority
Auto-Fixer Tests 15 Verify each auto-fixer transforms code correctly Critical
Review Gate Tests 12 Gate pass/fail logic and bounce packet construction Critical
LLM Review Parsing Tests 8 Verdict and issue extraction from CODE_REVIEW_PROMPT responses High
Test Generation Tests 10 QA_SYSTEM_PROMPT output parsing and mock detection Critical
Escalation Context Tests 6 G229 escalating context across fix cycles High
Integration Tests 4 Full pipeline from CXP packet to bounce or handoff High
TOTAL 55
Test Infrastructure
Workspace Fixture

All tests that invoke auto-fixers or gates use a real temporary workspace:

@pytest.fixture
def workspace(tmp_path):
    ws = tmp_path / "project"
    ws.mkdir()
    return ws
QaVolk Fixture
@pytest.fixture
def qa(workspace):
    volk = QaVolk()
    volk.workspace = str(workspace)
    return volk
Minimal CXP Packet
def make_packet(workspace, files, summary="test task"):
    return CXPPacket(
        from_agent="U:Cd",
        to_agent="U:Qa",
        thread_id="thread-001",
        subject="review_and_test",
        payload={
            "files_created": files,
            "task_summary": summary,
            "project_dir": workspace,
        },
        work_status=WorkStatus.IN_PROGRESS,
    )
Auto-Fixer Tests (15)
aioredis Migration (3 tests)
Missing Import Injection (3 tests)
Redis API Fixes (4 tests)
Code Style Fixes (5 tests)
Review Gate Tests (12)
Gate 1: Syntax (3 tests)
Gate 2: Lint (3 tests)
Gate 3: Checklist (3 tests)
Bounce Packet Structure (3 tests)
LLM Review Parsing Tests (8)
Verdict Extraction (3 tests)
Issue Extraction (3 tests)
CODE_REVIEW_PROMPT Completeness (2 tests)
Test Generation Tests (10)
QA_SYSTEM_PROMPT Rules (3 tests)
Test File Extraction (4 tests)
Mock Detection and Retry (3 tests)
Escalation Context Tests (6)
Escalation Level Boundaries
History Tracking
Integration Tests (4)
Full Pipeline: Review Pass to U:Cp
Full Pipeline: Gate Bounce
Full Pipeline: Test Failure Bounce
G226: No Cycle Cap

Other tracked files

Full spec text

Verbatim from specs/031-qa-volk/spec.md; headings are pushed down two levels.

SPEC-031: QA Volk (U:Qa)

Metadata

Overview

What

The QA Volk (U:Qa) is the quality assurance agent in the Orchestrata multi-agent pipeline. It receives code artifacts from U:Cd (CodeDirector) and performs a full review-and-test pipeline: deterministic auto-fixing, three sequential review gates, LLM-based code review against a 31-item Google-style checklist, test generation, test execution, and fix delegation back to U:Cd on any failure. The cycle runs without a cap until all issues are resolved (G226), with escalating context on repeat failures (G229).

Why
How
  1. Auto-fix pass -- 13 deterministic AST/regex fixers run on workspace files before any gate, eliminating known LLM-generated defects.
  2. Gate 1: Syntax -- every file in the workspace compiles under Python 3.13; failures bounce to U:Cd immediately.
  3. Gate 2: Lint -- flake8 critical codes (E9xx, F8xx) pass; failures bounce to U:Cd.
  4. Gate 3: Checklist -- deterministic traceability, import-order, docstring, and structural checks; failures bounce to U:Cd.
  5. Import validation -- each source module is imported in the workspace venv to catch hallucinated APIs.
  6. LLM code review -- CODE_REVIEW_PROMPT drives a 31-item structured review; FAIL verdict triggers bounce to U:Cd.
  7. Test generation -- QA_SYSTEM_PROMPT generates a pytest suite with zero mocking, @traces decorators, and real infrastructure fixtures.
  8. Test execution -- tests run in the workspace venv; failures trigger bounce to U:Cd with from_test_fix=True.
  9. Pass path -- all gates pass, tests pass, git commit, forward to U:Cp with compliance_check subject.

Terminology

Term Definition
QaVolk The QA Volk implementation class; alias U:Qa, role "QA Engineer". Extends BaseVolk.
review_and_test The CXP subject that triggers the full QA pipeline in process().
Auto-fix pass 13 deterministic fixers that rewrite known LLM defects in-place before gates run.
Gate 1 -- Syntax AST compilation check: every .py file must parse without SyntaxError.
Gate 2 -- Lint Flake8 critical-code check: E9xx (runtime errors) and F8xx (undefined names, imports) must be zero.
Gate 3 -- Checklist Deterministic structural checks: traceability decorators, import order, docstrings, no relative imports.
review_gate_syntax Function from orchestrata-codedirector review_checks.py; returns list of (code, filepath, description) tuples.
review_gate_lint Function from orchestrata-codedirector review_checks.py; returns lint issues as tuples.
review_gate_checklist Function from orchestrata-codedirector review_checks.py; returns structural quality issues.
get_fix_instructions Function from orchestrata-codedirector review_checks.py; converts issues list to a structured fix prompt.
CODE_REVIEW_PROMPT 31-item Google-style checklist prompt driving LLM-based code review.
QA_SYSTEM_PROMPT System prompt for test generation: zero mocking, pytest only, @traces required, Python 3.13 idioms.
G226 Governance rule: no cycle cap -- review loops until ALL issues resolved.
G229 Governance rule: fix attempt history tracked and injected into bounce packets for escalation context.
_build_escalating_context Builds progressively richer fix context: issues only (cycles 1-2), history (cycles 3+), root-cause hints (cycles 5+), full file contents (cycles 8+).
_bounce_for_fix Emits a CXP fix_issues packet to U:Cd with escalating context and gate label.
from_test_fix Payload flag set on bounce packets after test failures; causes QA to skip re-generation and rerun existing tests (G100).
compliance_check CXP subject sent to U:Cp (Compliance Volk) when all review gates pass and tests pass.
fix_issues CXP subject sent to U:Cd when any review gate or test suite fails.
_test_fix_count Counter of test-fix cycles in the current thread; used for escalation context.
review_cycle Counter of full review cycles in the current thread; printed in status messages.
WORKSPACE_DIR The filesystem path of the generated project being reviewed; configurable from packet payload.

User Stories

US-031-001: Deterministic Auto-Fix Before Review

As a QA Volk receiving generated source code, I want known LLM-generated defects corrected automatically before any gates run, So that review cycles are not wasted on mechanical, predictable errors.

Acceptance Criteria:

US-031-002: Three-Gate Sequential Review Pipeline

As a QA pipeline enforcing quality standards, I want source code to pass three deterministic gates in strict sequence before LLM review, So that structural defects are caught cheaply without spending LLM tokens on broken code.

Acceptance Criteria:

US-031-003: LLM-Based Code Review with 31-Item Checklist

As a QA Volk performing code review, I want to drive an LLM through a structured 31-item checklist, So that code is evaluated against Google engineering standards before test generation.

Acceptance Criteria:

US-031-004: Test Generation with Zero-Mocking Policy

As a QA Volk generating test suites, I want to produce pytest suites that use zero mocking and real infrastructure, So that tests validate actual behavior rather than mock contracts.

Acceptance Criteria:

US-031-005: Test Execution and Fix Delegation

As a QA Volk running generated tests, I want to execute the test suite in the workspace and delegate failures to U:Cd, So that code is not forwarded to U:Cp until tests actually pass.

Acceptance Criteria:

US-031-006: Escalating Fix Context (G229)

As a QA Volk managing multi-cycle fix loops, I want to inject progressively richer context into bounce packets on repeated failures, So that U:Cd receives the history of failed attempts and cannot repeat broken strategies.

Acceptance Criteria:

US-031-007: Successful QA Handoff to Compliance

As a QA Volk completing a successful review-and-test cycle, I want to commit the workspace and forward a compliance_check packet to U:Cp, So that the pipeline continues with formal compliance verification.

Acceptance Criteria:

Data Models

CXP Packet: review_and_test
{
  "subject": "review_and_test",
  "sender": "U:Cd",
  "recipient": "U:Qa",
  "payload": {
    "workspace_dir": "/path/to/workspace",
    "task_summary": "Build user authentication module",
    "source_files": ["src/auth.py", "src/models.py"],
    "thread_id": "thread-123"
  }
}
CXP Packet: fix_issues
{
  "subject": "fix_issues",
  "sender": "U:Qa",
  "recipient": "U:Cd",
  "payload": {
    "workspace_dir": "/path/to/workspace",
    "from_test_fix": false,
    "escalating_context": "Cycle 1 issues only...",
    "gate_label": "GATE 1 FAIL: Syntax errors",
    "issues": [{"code": "E902", "file": "src/auth.py", "description": "SyntaxError"}],
    "fix_instructions": "Fix the following syntax errors..."
  }
}
CXP Packet: compliance_check
{
  "subject": "compliance_check",
  "sender": "U:Qa",
  "recipient": "U:Cp",
  "payload": {
    "test_passed": true,
    "test_files": ["test_auth.py", "test_models.py"],
    "test_output": "50 passed in 2.3s",
    "review": {"verdict": "PASS", "issues": []},
    "review_cycles": 1,
    "project_dir": "/path/to/workspace",
    "task_summary": "Build user authentication module"
  }
}
ReviewGateResult
class ReviewGateResult:
    gate_name: str           # "Syntax", "Lint", "Checklist"
    passed: bool
    issues: list[tuple[str, str, str]]  # (code, filepath, description)
    fix_instructions: str
EscalatingContext Levels
Cycle Range Context Level Content
1-2 Basic Current issues + fix instructions
3-7 History + Last 3 failed attempts
5+ Root Cause + Persistent issue analysis
8+ Full Content + Full file contents

Functional Requirements

FR-001: QaVolk MUST extend BaseVolk with alias "U:Qa" and role "QA Engineer".

FR-002: QaVolk process() MUST route "review_and_test" subject to _handle_review() method.

FR-003: All 13 auto-fixers MUST run on every .py file in workspace before any review gate executes.

FR-004: Gate 1 (Syntax) MUST compile every .py file using Python AST; failures MUST bounce to U:Cd immediately.

FR-005: Gate 2 (Lint) MUST run flake8 with E9xx/F8xx codes; failures MUST bounce to U:Cd.

FR-006: Gate 3 (Checklist) MUST run deterministic structural checks; failures MUST bounce to U:Cd.

FR-007: Import validation MUST attempt to import each source module in workspace venv; import errors bounce to U:Cd.

FR-008: LLM code review MUST use CODE_REVIEW_PROMPT with exactly 31 checklist items.

FR-009: LLM review MUST parse REVIEW_VERDICT line and extract PASS/FAIL status.

FR-010: FAIL verdict MUST trigger bounce to U:Cd with parsed issues and fix instructions.

FR-011: Test generation MUST use QA_SYSTEM_PROMPT with zero-mocking enforcement.

FR-012: Generated tests MUST use pytest, @traces decorators, and real Redis infrastructure.

FR-013: Test execution MUST run pytest in workspace venv; failures MUST loop to U:Cd with from_test_fix=True.

FR-014: G226 MUST enforce no cycle cap; review loops MUST continue until all issues resolved.

FR-015: G229 MUST track fix history and inject escalating context into bounce packets.

FR-016: Cycles 3+ MUST include last 3 failed attempts in bounce packet with warning not to repeat strategies.

FR-017: Cycles 5+ MUST include root-cause analysis for persistent issues.

FR-018: Cycles 8+ MUST include full file contents for referenced files.

FR-019: On successful path, _git_commit() MUST be called with "QA complete: tests PASS" message.

FR-020: On success, compliance_check CXP packet MUST be emitted to U:Cp with complete test and review metadata.

FR-021: _test_fix_count MUST increment on each test-fix cycle within same thread.

FR-022: review_cycle counter MUST be printed in status messages.

Error Conditions

ERR-ID Condition Handling
ERR-031-001 SyntaxError in any .py file Bounce to U:Cd via fix_issues with gate_label="GATE 1 FAIL: Syntax errors"
ERR-031-002 Flake8 E9xx (runtime error) detected Bounce to U:Cd via fix_issues with gate_label="GATE 2 FAIL: Runtime errors"
ERR-031-003 Flake8 F8xx (undefined name/import) detected Bounce to U:Cd via fix_issues with gate_label="GATE 2 FAIL: Undefined names"
ERR-031-004 Checklist structural violation detected Bounce to U:Cd via fix_issues with gate_label="GATE 3 FAIL: Structural violations"
ERR-031-005 Import validation failure (module not found) Bounce to U:Cd via fix_issues with gate_label="IMPORT VALIDATION FAIL"
ERR-031-006 LLM review returns FAIL verdict Bounce to U:Cd via fix_issues with parsed issues from LLM response
ERR-031-007 Test execution returns non-zero exit code Bounce to U:Cd via fix_issues with from_test_fix=True
ERR-031-008 Generated tests contain mock usage Re-run LLM generation with anti-mock instruction via _retry_without_mocks()
ERR-031-009 Test file exceeds 120 line limit Split into multiple test files automatically
ERR-031-010 LLM client returns invalid JSON response Retry request with temperature=0.0 and explicit format instructions
ERR-031-011 Git commit fails Log error, continue pipeline without commit
ERR-031-012 CXP packet send fails Log error, set work_status to WorkStatus.ERROR
ERR-031-013 Workspace directory not accessible Return error response with ERR_WORKSPACE_INACCESSIBLE
ERR-031-014 Thread ID changes mid-review Reset _fix_history and start fresh escalation context

API Contracts

class QaVolk(BaseVolk)
Methods

init(self, llm_client, workspace_dir, thread_id=None)

process(subject, payload)

_handle_review(payload)

_autofix_aioredis()

_autofix_sync_redis()

_run_review_gates()

_llm_code_review(source_files)

_generate_tests()

_run_tests()

_bounce_for_fix(gate_label, issues, fix_instructions, from_test_fix=False)

_build_escalating_context(issues, cycle)

_git_commit()

Non-Functional Requirements

Performance
Reliability
Maintainability

Technical Design

Architecture
U:Cd (CodeDirector) 
    |
    v
[review_and_test packet]
    |
    v
QaVolk._handle_review()
    |
    +---> [Auto-fix Pass: 13 deterministic fixers]
    |
    +---> [Gate 1: Syntax]
    |
    +---> [Gate 2: Lint]
    |
    +---> [Gate 3: Checklist]
    |
    +---> [Import Validation]
    |
    +---> [LLM Code Review: 31-item checklist]
    |
    +---> [Test Generation]
    |
    +---> [Test Execution]
    |
    v
[Pass] --> U:Cp (Compliance) [compliance_check]
[Fail] --> U:Cd (CodeDirector) [fix_issues]
Escalation Flow

Examples

Example 1: Syntax Gate Failure
[U:Qa] Auto-fix: 13 fixers applied
[U:Qa] Gate 1/3: Syntax... FAILED
  src/auth.py: SyntaxError: invalid syntax (line 42)
[U:Qa] Bouncing to U:Cd with escalating context (cycle 1)
Example 2: Successful Review Path
[U:Qa] Auto-fix: 13 fixers applied
[U:Qa] Gate 1/3: Syntax... PASS
[U:Qa] Gate 2/3: Lint... PASS
[U:Qa] Gate 3/3: Checklist... PASS
[U:Qa] Import validation... PASS
[U:Qa] LLM Code Review... PASS
[U:Qa] Generating tests...
[U:Qa] Running tests...
[U:Qa] Tests: 50 passed in 2.3s
[U:Qa] Git commit: QA complete: tests PASS
[U:Qa] Forwarding to U:Cp with compliance_check
Example 3: Escalating Context (Cycle 5)
[U:Qa] Test execution failed (cycle 5)
[U:Qa] Building escalating context:
  - Current issues: test_auth.py::test_login fails
  - History (last 3):
    - Cycle 2: 3 test failures, same test_login
    - Cycle 3: 2 test failures, test_login + test_logout
    - Cycle 4: 1 test failure, test_login
  - Root cause: test_login mocks Redis instead of using real instance
  - Strategy: Read entire auth.py, trace data flow, rewrite without mocks
[U:Qa] Bouncing to U:Cd with full escalation

Security

Governance

Visual Architecture

Class Diagram

Diagram source shown as text: Mermaid could not parse it (Error: Parse error on line 15:).

classDiagram-direction LR
    class ArchitectAgent {
        $agent_type: ClassVar[str] = 'U:Ar'
        ---
        +__init__(model: str = 'claude-3-5-haiku', agent_name: str = '')
        +async analyze_impact(file_path: str, old_string: str, new_string: str, context: dict[str, Any] | None = None): dict[str, Any]
        +async design_solution(requirements: dict[str, Any]): dict[str, Any]
        +async review_architecture(architecture: dict[str, Any]): dict[str, Any]
    }
    class BaseAgent <<pydantic>> {
        -model: str = Field(default='claude-3-5-haiku', description='Model identifier for the agent', min_length=1, strict=True)
        -agent_id: str = Field(default='', description='Unique agent identifier', strict=True)
        -agent_name: str = Field(default='', description='Human-readable agent name', strict=True)
        $agent_type: ClassVar[str] = 'U:Base'
        $_instance_counters: ClassVar[dict[str, int]] = {}
        ---
        {@classmethod} +validate_model(v: str): str
        +__init__(**data: Any): None
        #_generate_unique_id(): str
        +model_post_init(__context: Any): None
        +async receive_message(message: Any, sender: str | None = None): Any
    }
    class CodeDirectorAgent {
        $agent_type: ClassVar[str] = 'U:Cd'
        ---
        +__init__(model: str = 'claude-3-5-sonnet', agent_name: str = '')
        +async execute_edit(file_path: str, old_string: str, new_string: str, context: dict[str, Any] | None = None): dict[str, Any]
        +async review_code(code: str, requirements: dict[str, Any]): dict[str, Any]
        +async implement_feature(spec: dict[str, Any]): dict[str, Any]
    }
    class QualityAssuranceAgent {
        $agent_type: ClassVar[str] = 'U:Qa'
        ---
        +__init__(model: str = 'claude-3-5-sonnet', agent_name: str = '')
        +async review_edit(file_path: str, old_string: str, new_string: str, context: dict[str, Any] | None = None): dict[str, Any]
        +async run_tests(test_suite: str): dict[str, Any]
        +async audit_security(code: str): dict[str, Any]
    }
    class UserExperienceAgent {
        $agent_type: ClassVar[str] = 'U:Ux'
        ---
        +__init__(model: str = 'claude-3-5-haiku', agent_name: str = '')
        +async review_ux(file_path: str, old_string: str, new_string: str, context: dict[str, Any] | None = None): dict[str, Any]
        +async design_ui(requirements: dict[str, Any]): dict[str, Any]
        +async evaluate_accessibility(ui: dict[str, Any]): dict[str, Any]
    }
    BaseAgent <|-- ArchitectAgent
    BaseModel <|-- BaseAgent
    BaseAgent <|-- CodeDirectorAgent
    BaseAgent <|-- QualityAssuranceAgent
    BaseAgent <|-- UserExperienceAgent
Unit Tests
Integration Tests
Coverage Targets

Integration Points

U:Cd (CodeDirector)
U:Cp (Compliance Volk)
orchestrata-codedirector
orchestrata-llm (SPEC-028)

Success Criteria

Observability

Logging Requirements
  1. Component Logging
    • Log all major operations with appropriate log levels
    • Include correlation IDs for request tracing
    • Log performance metrics for critical operations
    • Log errors with full context for debugging
Metrics Requirements
  1. Performance Metrics
    • Operation duration histograms
    • Request/error rate counters
    • Resource utilization gauges
    • Business metrics relevant to domain
Tracing Requirements
  1. Distributed Tracing
    • Trace propagation across component boundaries
    • Span attributes for domain-specific context
    • Trace sampling based on operational needs
Alerting Requirements
  1. Critical Alerts
    • Error rate spikes
    • Performance degradation
    • Resource exhaustion
    • Business metric anomalies
Dashboard Requirements
  1. Operational Dashboard
    • Real-time system health
    • Performance trends
    • Error analysis
    • Resource monitoring

Revision History

Version Date Author Changes
1.0.0 2026-04-04 Mohamed Initial spec
1.0.1 2026-04-09 Mohamed Added missing sections per SPEC_STANDARD.md: Data Models, Functional Requirements (FR-001-022), Error Conditions (ERR-031-001 to ERR-031-014), API Contracts, Non-Functional Requirements, Technical Design, Examples, Security, Governance, Dependencies, Testing, Integration Points, Success Criteria, Revision History

Acceptance Criteria Summary