Orchestrata Ultimate / RTM / 035-verification
RTM: 035-verification
Requirement to symbol to test rows for 035-verification: Verification Agent (U:Vfy).
At a glance
| Item | Value |
|---|---|
| Requirement IDs | 230 |
Traced from src/ |
188 (81.7%) |
| Untraced | 42 |
Cited from tests/ |
8 (3.5%) |
Traced in src/ but not cited from tests/ |
182 |
Reading this matrix
This matrix lists each of the 230 requirement IDs of spec 035-verification with the src/ symbols that cite it and the tests that cite it. 188 of them (81.7%) have at least one citing symbol and 42 have none.
8 requirements are also cited from tests/, and 182 are traced in code but never named by a test. A test citation is a @traces on a test function; it shows intent to cover, not that the test runs or reaches the symbol.
Rows showing none in the symbols column are the gaps to close first. The densest untraced groups are REQ-035-US-011 (15), REQ-035-US-010 (14), REQ-035-US-003 (4).
Requirement to symbols to tests
| Requirement | Symbols in src/ |
Tests citing it |
|---|---|---|
REQ-035-US-001-001 Parse simple requirement statements ("create a file named test.py") |
orchestrata_ultimate/verification/config/verification_config ModelConfig L15orchestrata_ultimate/verification/config/verification_config VerificationConfig L27orchestrata_ultimate/verification/constraints/definitions filter_by_type L36+11 more |
none |
REQ-035-US-001-002 Parse compound requirements ("read config.yaml and update database") |
orchestrata_ultimate/verification/requirements/requirements_parser parse L71 |
none |
REQ-035-US-001-003 Extract action verbs (create, read, update, delete, execute) |
orchestrata_ultimate/verification/requirements/requirements_parser extract_functional L160 |
none |
REQ-035-US-001-004 Extract target nouns (file, database, api, system) |
orchestrata_ultimate/verification/requirements/requirements_parser extract_functional L160 |
none |
REQ-035-US-001-005 Identify constraints (permissions, limits, conditions) |
orchestrata_ultimate/verification/requirements/requirements_parser RequirementsParser L59orchestrata_ultimate/verification/requirements/requirements_parser extract_quality L119 |
none |
REQ-035-US-001-006 Handle ambiguous requirements with clarification questions |
orchestrata_ultimate/verification/requirements/requirements_parser parse L71 |
none |
REQ-035-US-001-007 Support multi-language input (English primary) |
none | none |
REQ-035-US-001-008 Generate valid Prolog predicates from requirements |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator generate_predicate L48 |
none |
REQ-035-US-001-009 Validate generated Prolog syntax |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator validate_syntax L73 |
none |
REQ-035-US-001-010 Store constraints for session persistence |
orchestrata_ultimate/orchestrator/verification_gate _persist L741orchestrata_ultimate/orchestrator/verification_gate open_session L518orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20+4 more |
none |
REQ-035-US-001-011 Create RequirementsParser class |
orchestrata_ultimate/verification/requirements/requirements_parser RequirementsParser L59 |
none |
REQ-035-US-001-012 Implement NLP parsing for requirements |
orchestrata_ultimate/verification/requirements/requirements_parser parse L71 |
none |
REQ-035-US-001-013 Create action verb dictionary (CREATE, READ, UPDATE, DELETE, EXECUTE, VALIDATE) |
orchestrata_ultimate/verification/requirements/requirements_parser extract_functional L160 |
none |
REQ-035-US-001-014 Create target noun dictionary (FILE, DATABASE, API, SYSTEM, SERVICE) |
orchestrata_ultimate/verification/requirements/requirements_parser extract_functional L160 |
none |
REQ-035-US-001-015 Implement constraint extraction using pattern matching |
orchestrata_ultimate/verification/requirements/requirements_parser RequirementsParser L59orchestrata_ultimate/verification/requirements/requirements_parser extract_performance L93orchestrata_ultimate/verification/requirements/requirements_parser extract_quality L119+1 more |
none |
REQ-035-US-001-016 Create Prolog predicate generator |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13 |
none |
REQ-035-US-001-017 Implement Prolog syntax validator |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator validate_syntax L73 |
none |
REQ-035-US-001-018 Create constraint storage backend (Redis) |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20orchestrata_ultimate/verification/integrations/redis_manager available L113 |
none |
REQ-035-US-001-019 Add requirement session management |
none | none |
REQ-035-US-001-020 Implement requirement versioning |
none | none |
REQ-035-US-001-021 Unit test: Parse "create file test.py" → constraint(file_exists, ['test.py']) |
orchestrata_ultimate/verification/requirements/requirements_parser parse L71 |
none |
REQ-035-US-001-022 Unit test: Parse "API must respond in 200ms" → constraint(response_time_max, [200]) |
orchestrata_ultimate/verification/requirements/requirements_parser extract_performance L93 |
none |
REQ-035-US-001-023 Integration test: Parse complex requirement generates valid Prolog |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator generate L31orchestrata_ultimate/verification/requirements/requirements_parser parse L71 |
none |
REQ-035-US-001-024 Performance test: Parse 100 requirements in <500ms average |
orchestrata_ultimate/verification/requirements/requirements_parser parse L71 |
none |
REQ-035-US-002-001 Define functional constraints (file exists, API responds) |
orchestrata_ultimate/verification/agents/execution_agent ExecutionAgent L17orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types Constraint L99 |
none |
REQ-035-US-002-002 Define quality constraints (test coverage, documentation) |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-002-003 Define performance constraints (response time, throughput) |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-002-004 Define security constraints (no vulnerabilities, authentication) |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-002-005 Support constraint priorities (required vs. optional) |
orchestrata_ultimate/verification/constraints/definitions sort_by_priority L26orchestrata_ultimate/verification/types Constraint L99 |
none |
REQ-035-US-002-006 Support constraint dependencies (A must satisfy before B) |
orchestrata_ultimate/verification/constraints/evaluator eval_one L194orchestrata_ultimate/verification/constraints/evaluator evaluate_with_dependencies L177 |
none |
REQ-035-US-002-007 Support constraint groups (AND/OR logic) |
orchestrata_ultimate/verification/constraints/definitions group_by_type L15 |
none |
REQ-035-US-002-008 Validate constraint syntax before activation |
orchestrata_ultimate/verification/constraints/definitions is_blocking L9orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator validate_syntax L73 |
none |
REQ-035-US-002-009 Persist constraint definitions |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20orchestrata_ultimate/verification/integrations/redis_manager save_state L45 |
none |
REQ-035-US-002-010 Create constraint definition schema |
orchestrata_ultimate/verification/types Constraint L99orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-002-011 Implement constraint parser (YAML/JSON → Prolog) |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator generate_predicate L48 |
none |
REQ-035-US-002-012 Create constraint priority system |
orchestrata_ultimate/verification/constraints/definitions sort_by_priority L26 |
none |
REQ-035-US-002-013 Implement constraint dependency resolution |
orchestrata_ultimate/verification/constraints/evaluator eval_one L194orchestrata_ultimate/verification/constraints/evaluator evaluate_with_dependencies L177 |
none |
REQ-035-US-002-014 Create constraint validation rules |
orchestrata_ultimate/verification/constraints/definitions is_blocking L9orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator validate_syntax L73 |
none |
REQ-035-US-002-015 Implement constraint persistence |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20orchestrata_ultimate/verification/integrations/redis_manager load_state L67 |
none |
REQ-035-US-002-016 Add constraint versioning |
none | none |
REQ-035-US-002-017 Create constraint import/export |
orchestrata_ultimate/verification/types Constraint L99orchestrata_ultimate/verification/types from_dict L133orchestrata_ultimate/verification/types to_dict L121+3 more |
none |
REQ-035-US-002-018 Unit test: Constraint with all fields validates |
orchestrata_ultimate/verification/types Constraint L99 |
none |
REQ-035-US-002-019 Unit test: Invalid constraint syntax rejected |
orchestrata_ultimate/verification/requirements/prolog_generator PrologGenerator L13orchestrata_ultimate/verification/requirements/prolog_generator validate_syntax L73 |
none |
REQ-035-US-002-020 Unit test: Priority ordering works correctly |
orchestrata_ultimate/verification/constraints/definitions sort_by_priority L26 |
none |
REQ-035-US-002-021 Integration test: Constraint dependencies resolve correctly |
orchestrata_ultimate/verification/constraints/evaluator eval_one L194orchestrata_ultimate/verification/constraints/evaluator evaluate_with_dependencies L177 |
none |
REQ-035-US-002-022 Integration test: Constraint persistence works |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20orchestrata_ultimate/verification/integrations/redis_manager save_state L45 |
none |
REQ-035-US-003-001 Evaluate single constraint satisfaction |
orchestrata_ultimate/orchestrator/verification_gate ArtifactContextBuilder L290orchestrata_ultimate/orchestrator/verification_gate _fold_qa_payload_signals L383orchestrata_ultimate/orchestrator/verification_gate _parses L361+8 more |
tests/orchestrator/test_verification_gate.py::test_artifact_context_builder_computes_actuals |
REQ-035-US-003-002 Evaluate all constraints (AND logic) |
orchestrata_ultimate/verification/constraints/evaluator ConstraintEvaluator L66orchestrata_ultimate/verification/constraints/evaluator evaluate_all L161 |
none |
REQ-035-US-003-003 Evaluate constraint groups (OR logic) |
none | none |
REQ-035-US-003-004 Handle constraint dependencies correctly |
orchestrata_ultimate/verification/constraints/evaluator eval_one L194orchestrata_ultimate/verification/constraints/evaluator evaluate_with_dependencies L177 |
none |
REQ-035-US-003-005 Return specific failure reasons |
orchestrata_ultimate/orchestrator/verification_gate ArtifactContextBuilder L290orchestrata_ultimate/orchestrator/verification_gate VerificationVerdict L132orchestrata_ultimate/verification/constraints/evaluator evaluate L113+4 more |
none |
REQ-035-US-003-006 Cache evaluation results |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20 |
none |
REQ-035-US-003-007 Support partial satisfaction |
orchestrata_ultimate/verification/constraints/evaluator evaluate_all L161orchestrata_ultimate/verification/types CheckpointResult L215 |
none |
REQ-035-US-003-008 Handle constraint timeouts |
none | none |
REQ-035-US-003-009 Log evaluation results |
orchestrata_ultimate/verification/verification/engine VerificationEngine L28orchestrata_ultimate/verification/verification/engine verify L49 |
none |
REQ-035-US-003-010 Support evaluation replay |
orchestrata_ultimate/verification/session/verification_session update_results L120 |
none |
REQ-035-US-003-011 Create constraint evaluator |
orchestrata_ultimate/verification/constraints/evaluator ConstraintEvaluator L66orchestrata_ultimate/verification/constraints/evaluator _bridge L95 |
none |
REQ-035-US-003-012 Implement evaluation result aggregation |
orchestrata_ultimate/verification/constraints/evaluator evaluate_all L161 |
none |
REQ-035-US-003-013 Add result caching (Redis) |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20orchestrata_ultimate/verification/integrations/redis_manager save_state L45 |
none |
REQ-035-US-003-014 Create failure reason generator |
orchestrata_ultimate/orchestrator/verification_gate ArtifactContextBuilder L290orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18orchestrata_ultimate/verification/verification/feedback suggest_fixes L62 |
none |
REQ-035-US-003-015 Implement timeout handling |
none | none |
REQ-035-US-003-016 Add evaluation logging |
orchestrata_ultimate/verification/verification/engine VerificationEngine L28orchestrata_ultimate/verification/verification/engine verify L49 |
none |
REQ-035-US-003-017 Create evaluation replay mechanism |
orchestrata_ultimate/verification/session/verification_session update_results L120 |
none |
REQ-035-US-003-018 Unit test: Satisfied constraint returns SATISFIED |
orchestrata_ultimate/verification/types VerificationStatus L51 |
none |
REQ-035-US-003-019 Unit test: Failed constraint returns NOT_MET with reason |
orchestrata_ultimate/verification/types VerificationResult L148orchestrata_ultimate/verification/types VerificationStatus L51 |
none |
REQ-035-US-003-020 Unit test: Multiple constraints aggregate correctly |
orchestrata_ultimate/verification/constraints/evaluator evaluate_all L161 |
none |
REQ-035-US-003-021 Integration test: Evaluation with timeouts works |
none | none |
REQ-035-US-003-022 Integration test: Caching improves performance |
orchestrata_ultimate/verification/integrations/redis_manager VerificationRedisManager L20 |
none |
REQ-035-US-004-001 Define 5 standard checkpoints (implementation, tests, integration, docs, final) |
orchestrata_ultimate/orchestrator/verification_gate checkpoint_for L508orchestrata_ultimate/verification/checkpoints/definitions get_checkpoint_predicates L60orchestrata_ultimate/verification/types CheckpointName L62 |
none |
REQ-035-US-004-002 Verify at each checkpoint |
orchestrata_ultimate/orchestrator/pipeline_runner _apply_verification_verdict L703orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/orchestrator/verification_gate after_stage L555+3 more |
none |
REQ-035-US-004-003 Support custom checkpoint definitions |
orchestrata_ultimate/orchestrator/verification_gate checkpoint_for L508orchestrata_ultimate/verification/checkpoints/definitions get_checkpoint_predicates L60 |
none |
REQ-035-US-004-004 Track checkpoint progress |
orchestrata_ultimate/orchestrator/verification_gate after_stage L555orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier transition_state L86+1 more |
none |
REQ-035-US-004-005 Allow checkpoint skipping (with approval) |
orchestrata_ultimate/verification/checkpoints/definitions get_next_checkpoint L44 |
none |
REQ-035-US-004-006 Provide checkpoint-specific feedback |
orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier get_checkpoint_criteria L79orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18+1 more |
none |
REQ-035-US-004-007 Support checkpoint rollback |
orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier transition_state L86 |
none |
REQ-035-US-004-008 Maintain checkpoint history |
orchestrata_ultimate/knowledge_graph/temporal get_history L133orchestrata_ultimate/verification/session/verification_session update_results L120 |
none |
REQ-035-US-004-009 Allow concurrent verification at checkpoints |
orchestrata_ultimate/verification/checkpoints/concurrent ConcurrentCheckpointEvaluator L22orchestrata_ultimate/verification/checkpoints/concurrent eval_one L56orchestrata_ultimate/verification/checkpoints/concurrent evaluate_all L41 |
none |
REQ-035-US-004-010 Create checkpoint definitions |
orchestrata_ultimate/verification/checkpoints/definitions get_checkpoint_predicates L60orchestrata_ultimate/verification/checkpoints/definitions get_next_checkpoint L44 |
none |
REQ-035-US-004-011 Implement checkpoint verification logic |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/orchestrator/verification_gate after_stage L555orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24+1 more |
none |
REQ-035-US-004-012 Create checkpoint progress tracker |
orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier transition_state L86 |
none |
REQ-035-US-004-013 Add checkpoint history storage |
none | none |
REQ-035-US-004-014 Implement concurrent verification |
orchestrata_ultimate/verification/checkpoints/concurrent ConcurrentCheckpointEvaluator L22orchestrata_ultimate/verification/protocols/verification_cxp create_verification_request L52 |
none |
REQ-035-US-004-015 Create custom checkpoint API |
orchestrata_ultimate/verification/checkpoints/definitions get_checkpoint_predicates L60 |
none |
REQ-035-US-004-016 Add checkpoint rollback support |
orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier transition_state L86 |
none |
REQ-035-US-004-017 Unit test: Checkpoint criteria evaluate correctly |
orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier get_checkpoint_criteria L79 |
none |
REQ-035-US-004-018 Unit test: Checkpoint progress tracking works |
orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24orchestrata_ultimate/verification/checkpoints/verifier transition_state L86 |
none |
REQ-035-US-004-019 Integration test: Full checkpoint sequence passes |
orchestrata_ultimate/verification/checkpoints/definitions get_next_checkpoint L44orchestrata_ultimate/verification/checkpoints/verifier CheckpointVerifier L24 |
none |
REQ-035-US-004-020 Integration test: Concurrent verification works |
orchestrata_ultimate/verification/checkpoints/concurrent ConcurrentCheckpointEvaluator L22orchestrata_ultimate/verification/checkpoints/concurrent eval_one L56orchestrata_ultimate/verification/checkpoints/concurrent evaluate_all L41+1 more |
none |
REQ-035-US-004-021 Performance test: 5 checkpoints in <5 seconds |
orchestrata_ultimate/verification/checkpoints/concurrent ConcurrentCheckpointEvaluator L22orchestrata_ultimate/verification/checkpoints/concurrent eval_one L56orchestrata_ultimate/verification/checkpoints/concurrent evaluate_all L41 |
none |
REQ-035-US-005-001 Support Claude Opus 4 for verification |
orchestrata_ultimate/verification/agents/supervisor_agent generate_proof_via_llm L77orchestrata_ultimate/verification/models/opus ClaudeOpusModel L13orchestrata_ultimate/verification/models/opus analyze L56+3 more |
none |
REQ-035-US-005-002 Support Deepseek R1 for verification |
orchestrata_ultimate/verification/models/deepseek_r1 DeepseekR1Model L13orchestrata_ultimate/verification/models/deepseek_r1 analyze L56orchestrata_ultimate/verification/models/deepseek_r1 capabilities L35+2 more |
none |
REQ-035-US-005-003 Support Qwen Max for verification |
orchestrata_ultimate/verification/models/qwen_max QwenMaxModel L13orchestrata_ultimate/verification/models/qwen_max analyze L56orchestrata_ultimate/verification/models/qwen_max capabilities L35+2 more |
none |
REQ-035-US-005-004 Support Grok4 MiniMax for verification |
orchestrata_ultimate/verification/models/grok4_minimax GrokMiniMaxModel L13orchestrata_ultimate/verification/models/grok4_minimax analyze L54orchestrata_ultimate/verification/models/grok4_minimax capabilities L33+2 more |
none |
REQ-035-US-005-005 Select appropriate model based on task |
orchestrata_ultimate/verification/agents/verification_agent __init__ L55orchestrata_ultimate/verification/agents/verification_agent analyze_via_llm L84orchestrata_ultimate/verification/models/base_model ModelRegistry L65+3 more |
none |
REQ-035-US-005-006 Handle model unavailability gracefully |
orchestrata_ultimate/verification/agents/supervisor_agent generate_proof_via_llm L77orchestrata_ultimate/verification/models/base_model ModelFallbackChain L109orchestrata_ultimate/verification/models/base_model analyze L54+3 more |
none |
REQ-035-US-005-007 Maintain verification consistency across models |
orchestrata_ultimate/verification/agents/verification_agent VerificationAgent L37orchestrata_ultimate/verification/agents/verification_agent analyze_via_llm L85orchestrata_ultimate/verification/models/base_model LLMModel L28+1 more |
none |
REQ-035-US-005-008 Track model usage for metrics |
orchestrata_ultimate/verification/models/base_model ModelRegistry L65 |
none |
REQ-035-US-005-009 Support model fallback chain |
orchestrata_ultimate/verification/models/base_model ModelFallbackChain L109orchestrata_ultimate/verification/models/base_model generate_proof L57orchestrata_ultimate/verification/models/base_model generate_proof L171+3 more |
none |
REQ-035-US-005-010 Create model abstraction layer |
orchestrata_ultimate/verification/models/base_model LLMModel L28orchestrata_ultimate/verification/models/base_model name L37 |
none |
REQ-035-US-005-011 Implement Opus integration |
orchestrata_ultimate/verification/agents/supervisor_agent generate_proof_via_llm L77orchestrata_ultimate/verification/models/opus ClaudeOpusModel L13orchestrata_ultimate/verification/models/opus analyze L56+3 more |
none |
REQ-035-US-005-012 Implement Deepseek R1 integration |
orchestrata_ultimate/verification/models/deepseek_r1 DeepseekR1Model L13orchestrata_ultimate/verification/models/deepseek_r1 analyze L56orchestrata_ultimate/verification/models/deepseek_r1 capabilities L35+2 more |
none |
REQ-035-US-005-013 Implement Qwen Max integration |
orchestrata_ultimate/verification/models/qwen_max QwenMaxModel L13orchestrata_ultimate/verification/models/qwen_max analyze L56orchestrata_ultimate/verification/models/qwen_max capabilities L35+2 more |
none |
REQ-035-US-005-014 Implement Grok4 MiniMax integration |
orchestrata_ultimate/verification/models/grok4_minimax GrokMiniMaxModel L13orchestrata_ultimate/verification/models/grok4_minimax analyze L54orchestrata_ultimate/verification/models/grok4_minimax capabilities L33+2 more |
none |
REQ-035-US-005-015 Create model selection logic |
orchestrata_ultimate/verification/models/base_model ModelRegistry L65orchestrata_ultimate/verification/models/base_model get L85 |
none |
REQ-035-US-005-016 Add fallback handling |
orchestrata_ultimate/verification/models/base_model ModelFallbackChain L109orchestrata_ultimate/verification/models/base_model generate_proof L57orchestrata_ultimate/verification/models/base_model generate_proof L171+2 more |
none |
REQ-035-US-005-017 Implement usage tracking |
orchestrata_ultimate/verification/models/base_model ModelRegistry L65 |
none |
REQ-035-US-005-018 Create cost optimization |
orchestrata_ultimate/verification/models/base_model ModelFallbackChain L109 |
none |
REQ-035-US-005-019 Unit test: Model abstraction works |
orchestrata_ultimate/verification/models/base_model LLMModel L28orchestrata_ultimate/verification/models/base_model list_models L91 |
none |
REQ-035-US-005-020 Unit test: Fallback triggers correctly |
orchestrata_ultimate/verification/models/base_model ModelFallbackChain L109orchestrata_ultimate/verification/models/base_model verify L40orchestrata_ultimate/verification/models/base_model verify L133 |
none |
REQ-035-US-005-021 Integration test: All models respond correctly |
orchestrata_ultimate/verification/models/deepseek_r1 DeepseekR1Model L13orchestrata_ultimate/verification/models/deepseek_r1 verify L46orchestrata_ultimate/verification/models/grok4_minimax GrokMiniMaxModel L13+5 more |
none |
REQ-035-US-005-022 Integration test: Verification consistent across models |
orchestrata_ultimate/verification/agents/verification_agent VerificationAgent L37orchestrata_ultimate/verification/agents/verification_agent analyze_via_llm L85orchestrata_ultimate/verification/protocols/verification_cxp create_verification_request L52 |
none |
REQ-035-US-005-023 Performance test: Model response times acceptable |
orchestrata_ultimate/verification/models/base_model ModelFallbackChain L109orchestrata_ultimate/verification/models/base_model list_models L91orchestrata_ultimate/verification/models/base_model verify L40+1 more |
none |
REQ-035-US-006-001 Identify common edge cases for task type |
orchestrata_ultimate/verification/edge_cases/detector EdgeCaseDetector L375orchestrata_ultimate/verification/edge_cases/detector _init_common_edge_cases L114orchestrata_ultimate/verification/edge_cases/detector get_edge_cases_for_task L337 |
none |
REQ-035-US-006-002 Verify edge cases are tested |
orchestrata_ultimate/verification/edge_cases/detector EdgeCaseOccurrence L66orchestrata_ultimate/verification/edge_cases/detector _is_edge_case_tested L533 |
none |
REQ-035-US-006-003 Suggest edge cases to test |
orchestrata_ultimate/verification/edge_cases/detector suggest_edge_cases L504 |
none |
REQ-035-US-006-004 Detect untested edge cases |
orchestrata_ultimate/verification/edge_cases/detector EdgeCaseOccurrence L66 |
none |
REQ-035-US-006-005 Categorize edge cases by severity |
orchestrata_ultimate/verification/edge_cases/detector EdgeCaseCategory L33orchestrata_ultimate/verification/edge_cases/detector EdgeCaseSeverity L23orchestrata_ultimate/verification/types EdgeCase L306 |
none |
REQ-035-US-006-006 Track edge case coverage |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/database count L64orchestrata_ultimate/verification/edge_cases/detector EdgeCaseCoverage L83+2 more |
none |
REQ-035-US-006-007 Provide edge case examples |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/database get_by_constraint L46orchestrata_ultimate/verification/edge_cases/detector generate_test_suggestions L588 |
none |
REQ-035-US-006-008 Support custom edge case definitions |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/database add L34orchestrata_ultimate/verification/edge_cases/detector EdgeCase L47+1 more |
none |
REQ-035-US-006-009 Create edge case database |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/detector EdgeCaseDatabase L96 |
none |
REQ-035-US-006-010 Implement edge case detection |
orchestrata_ultimate/verification/edge_cases/detector EdgeCaseDetector L375orchestrata_ultimate/verification/edge_cases/detector detect L391orchestrata_ultimate/verification/edge_cases/detector detect_boundary_cases L425+2 more |
none |
REQ-035-US-006-011 Create edge case suggestion engine |
none | none |
REQ-035-US-006-012 Add edge case coverage tracking |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/database count L64orchestrata_ultimate/verification/edge_cases/detector calculate_coverage L556 |
none |
REQ-035-US-006-013 Implement edge case severity scoring |
orchestrata_ultimate/verification/types EdgeCase L306 |
none |
REQ-035-US-006-014 Create edge case reporting |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/database list_all L51orchestrata_ultimate/verification/edge_cases/detector EdgeCaseReporter L680+4 more |
none |
REQ-035-US-006-015 Unit test: Edge cases identified for task |
none | tests/verification/edge_cases/test_detector.py::test_detect_edge_cases_in_code |
REQ-035-US-006-016 Unit test: Edge case coverage calculated |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13 |
none |
REQ-035-US-006-017 Integration test: Edge case suggestions improve coverage |
none | tests/verification/edge_cases/test_detector.py::test_calculate_coverage_with_tested |
REQ-035-US-006-018 Integration test: Edge case tracking works |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13 |
none |
REQ-035-US-007-001 Accept exhaustive search proofs |
orchestrata_ultimate/verification/proofs/verifier verify_exhaustive_search L43 |
none |
REQ-035-US-007-002 Accept logical deduction proofs |
orchestrata_ultimate/verification/proofs/verifier verify_logical_deduction L76 |
none |
REQ-035-US-007-003 Accept mathematical proofs |
orchestrata_ultimate/verification/proofs/verifier verify_mathematical L100 |
none |
REQ-035-US-007-004 Accept practical explanations |
orchestrata_ultimate/verification/proofs/verifier verify_practical_explanation L124 |
none |
REQ-035-US-007-005 Verify proof completeness |
orchestrata_ultimate/verification/proofs/verifier verify L19 |
none |
REQ-035-US-007-006 Verify proof soundness |
orchestrata_ultimate/verification/proofs/verifier verify L19 |
none |
REQ-035-US-007-007 Verify alternatives exhausted |
orchestrata_ultimate/verification/proofs/verifier verify_exhaustive_search L43 |
none |
REQ-035-US-007-008 Reject invalid proofs |
orchestrata_ultimate/verification/proofs/verifier verify L19orchestrata_ultimate/verification/types ProofResult L203 |
none |
REQ-035-US-007-009 Request additional proof if needed |
orchestrata_ultimate/verification/orchestration/coordinator build_constraint_message L157orchestrata_ultimate/verification/orchestration/coordinator build_escalation_message L185orchestrata_ultimate/verification/orchestration/coordinator build_feedback_message L191+7 more |
none |
REQ-035-US-007-010 Create proof type handlers |
orchestrata_ultimate/verification/proofs/verifier ProofVerifier L15 |
none |
REQ-035-US-007-011 Implement exhaustive search verification |
orchestrata_ultimate/verification/proofs/verifier verify_exhaustive_search L43 |
none |
REQ-035-US-007-012 Implement logical deduction verification |
orchestrata_ultimate/verification/proofs/verifier verify_logical_deduction L76orchestrata_ultimate/verification/protocols/verification_cxp create_verification_request L52 |
none |
REQ-035-US-007-013 Implement mathematical proof verification |
orchestrata_ultimate/verification/proofs/verifier verify_mathematical L100orchestrata_ultimate/verification/protocols/verification_cxp create_verification_request L52 |
none |
REQ-035-US-007-014 Implement practical explanation verification |
orchestrata_ultimate/verification/proofs/verifier verify_practical_explanation L124orchestrata_ultimate/verification/protocols/verification_cxp create_verification_request L52 |
none |
REQ-035-US-007-015 Create proof evaluation criteria |
orchestrata_ultimate/verification/types ProofResult L203 |
none |
REQ-035-US-007-016 Add proof rejection handling |
orchestrata_ultimate/verification/agents/verification_agent VerificationAgent L37orchestrata_ultimate/verification/agents/verification_agent validate_proof L120 |
none |
REQ-035-US-007-017 Implement proof appeal process |
orchestrata_ultimate/verification/integrations/intent_broker processed_count L76orchestrata_ultimate/verification/orchestration/coordinator build_proof_submission L204orchestrata_ultimate/verification/protocols/verification_cxp VerificationCXPAdapter L19+1 more |
none |
REQ-035-US-007-018 Unit test: Valid exhaustive proof accepted |
orchestrata_ultimate/verification/proofs/verifier verify_exhaustive_search L43 |
none |
REQ-035-US-007-019 Unit test: Invalid proof rejected |
orchestrata_ultimate/verification/proofs/verifier verify L19 |
none |
REQ-035-US-007-020 Integration test: Full proof workflow works |
orchestrata_ultimate/verification/agents/supervisor_agent SupervisorAgent L27orchestrata_ultimate/verification/agents/supervisor_agent generate_proof L60orchestrata_ultimate/verification/agents/supervisor_agent submit_proof L110+2 more |
none |
REQ-035-US-007-021 Integration test: Proof appeal process works |
orchestrata_ultimate/verification/protocols/verification_cxp VerificationCXPAdapter L19orchestrata_ultimate/verification/protocols/verification_cxp create_proof_request L141 |
none |
REQ-035-US-008-001 Track verification attempts |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/orchestrator/verification_gate finalize L624orchestrata_ultimate/verification/session/verification_session increment_attempt L98 |
none |
REQ-035-US-008-002 Enforce max attempts (5) |
orchestrata_ultimate/orchestrator/verification_gate VerificationGateConfig L94orchestrata_ultimate/verification/session/verification_session is_max_exceeded L110orchestrata_ultimate/verification/types max_exceeded L288 |
none |
REQ-035-US-008-003 Provide feedback after each attempt |
orchestrata_ultimate/orchestrator/verification_gate VerificationVerdict L132orchestrata_ultimate/orchestrator/verification_gate finalize L624orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18+1 more |
none |
REQ-035-US-008-004 Support automatic retry |
orchestrata_ultimate/orchestrator/verification_gate VerificationGateConfig L94orchestrata_ultimate/orchestrator/verification_gate from_env L111orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43+1 more |
tests/orchestrator/test_verification_gate.py::test_gate_config_from_env_defaults_audit_only |
REQ-035-US-008-005 Support manual intervention |
orchestrata_ultimate/orchestrator/verification_gate VerificationGateConfig L94orchestrata_ultimate/verification/escalation_handler EscalationHandler L18orchestrata_ultimate/verification/escalation_handler build_escalation_report L55+3 more |
none |
REQ-035-US-008-006 Escalate after max attempts |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/orchestrator/verification_gate finalize L624orchestrata_ultimate/verification/escalation_handler EscalationHandler L18+1 more |
none |
REQ-035-US-008-007 Track attempt history |
orchestrata_ultimate/verification/session/verification_session update_results L120 |
none |
REQ-035-US-008-008 Support attempt reset |
orchestrata_ultimate/verification/session/verification_session update_status L142 |
none |
REQ-035-US-008-009 Log all attempts |
orchestrata_ultimate/verification/verification/engine VerificationEngine L28orchestrata_ultimate/verification/verification/engine verify L49 |
none |
REQ-035-US-008-010 Create attempt tracking system |
orchestrata_ultimate/verification/session/verification_session increment_attempt L98 |
none |
REQ-035-US-008-011 Implement max attempt enforcement |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/verification/session/verification_session is_max_exceeded L110 |
none |
REQ-035-US-008-012 Create feedback generation |
orchestrata_ultimate/verification/orchestration/coordinator build_feedback_message L192orchestrata_ultimate/verification/protocols/verification_cxp create_feedback_message L126orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18 |
none |
REQ-035-US-008-013 Add automatic retry support |
orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43orchestrata_ultimate/verification/orchestration/coordinator advance_step L81 |
none |
REQ-035-US-008-014 Implement escalation handler |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/verification/escalation_handler EscalationHandler L18 |
none |
REQ-035-US-008-015 Create attempt history storage |
none | none |
REQ-035-US-008-016 Add attempt reset functionality |
orchestrata_ultimate/verification/session/verification_session update_status L142 |
none |
REQ-035-US-008-017 Implement attempt logging |
orchestrata_ultimate/verification/verification/engine VerificationEngine L28orchestrata_ultimate/verification/verification/engine verify L49 |
none |
REQ-035-US-008-018 Unit test: Max attempts enforced |
orchestrata_ultimate/verification/session/verification_session is_max_exceeded L110 |
none |
REQ-035-US-008-019 Unit test: Feedback generated correctly |
orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18orchestrata_ultimate/verification/verification/feedback generate_feedback L36 |
none |
REQ-035-US-008-020 Integration test: Full retry loop works |
orchestrata_ultimate/orchestrator/pipeline_runner _apply_verification_verdict L703orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43 |
tests/orchestrator/test_verification_gate.py::test_pipeline_runner_default_constructs_gate_nonfatally |
REQ-035-US-008-021 Integration test: Escalation triggers correctly |
orchestrata_ultimate/verification/escalation_handler EscalationHandler L18orchestrata_ultimate/verification/escalation_handler should_escalate L34 |
none |
REQ-035-US-009-001 Receive intents from Intent Broker |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/orchestrator/verification_gate open_session L518orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12+2 more |
none |
REQ-035-US-009-002 Extract constraints from intents |
orchestrata_ultimate/orchestrator/verification_gate WorkTypeConstraintSource L226orchestrata_ultimate/orchestrator/verification_gate constraints_for L247orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12+2 more |
tests/orchestrator/test_verification_gate.py::test_worktype_constraint_source_total_over_worktypes |
REQ-035-US-009-003 Generate verification requirements from intents |
orchestrata_ultimate/orchestrator/verification_gate WorkTypeConstraintSource L226orchestrata_ultimate/orchestrator/verification_gate constraints_for L247orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12+1 more |
none |
REQ-035-US-009-004 Return verification results to Intent Broker |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/orchestrator/verification_gate VerificationVerdict L132orchestrata_ultimate/orchestrator/verification_gate finalize L624+11 more |
tests/orchestrator/test_verification_gate.py::test_verdict_packet_built_via_coordinator_cxp |
REQ-035-US-009-005 Support intent lifecycle integration |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43orchestrata_ultimate/verification/orchestration/coordinator start_workflow L69 |
none |
REQ-035-US-009-006 Track verification history per intent |
orchestrata_ultimate/orchestrator/verification_gate _persist L741orchestrata_ultimate/verification/session/verification_session update_results L120 |
none |
REQ-035-US-009-007 Support intent replay with verification |
orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43orchestrata_ultimate/verification/orchestration/coordinator advance_step L81 |
none |
REQ-035-US-009-008 Coordinate with Prolog validation |
orchestrata_ultimate/verification/agents/verification_agent VerificationAgent L37orchestrata_ultimate/verification/agents/verification_agent check_via_prolog L77 |
none |
REQ-035-US-009-009 Create Intent Broker integration API |
orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12orchestrata_ultimate/verification/session/verification_session create_session_from_intent L67 |
none |
REQ-035-US-009-010 Implement intent constraint extraction |
orchestrata_ultimate/orchestrator/verification_gate WorkTypeConstraintSource L226orchestrata_ultimate/orchestrator/verification_gate constraints_for L247orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12+2 more |
tests/orchestrator/test_verification_gate.py::test_constraint_source_delegates_to_intent_broker_adapter |
REQ-035-US-009-011 Create result forwarding |
orchestrata_ultimate/verification/orchestration/coordinator build_constraint_message L158orchestrata_ultimate/verification/orchestration/coordinator build_verification_result L174orchestrata_ultimate/verification/protocols/verification_cxp VerificationCXPAdapter L19+1 more |
none |
REQ-035-US-009-012 Add lifecycle coordination |
orchestrata_ultimate/orchestrator/verification_gate PipelineVerificationGate L419orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43 |
none |
REQ-035-US-009-013 Implement history tracking |
orchestrata_ultimate/knowledge_graph/temporal get_history L133orchestrata_ultimate/verification/session/verification_session update_results L120 |
none |
REQ-035-US-009-014 Create replay support |
orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43orchestrata_ultimate/verification/orchestration/coordinator advance_step L81 |
none |
REQ-035-US-009-015 Unit test: Intent parsing works |
orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12orchestrata_ultimate/verification/integrations/intent_broker intent_to_constraints L32 |
none |
REQ-035-US-009-016 Unit test: Constraint extraction works |
orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12orchestrata_ultimate/verification/integrations/intent_broker intent_to_constraints L32 |
none |
REQ-035-US-009-017 Integration test: Full integration with Intent Broker |
orchestrata_ultimate/verification/integrations/intent_broker IntentBrokerAdapter L12 |
none |
REQ-035-US-009-018 Integration test: Intent replay works |
orchestrata_ultimate/verification/orchestration/coordinator VerificationCoordinator L43orchestrata_ultimate/verification/orchestration/coordinator advance_step L81 |
none |
REQ-035-US-010-001 Detect hardcoded secrets |
none | none |
REQ-035-US-010-002 Detect SQL injection vulnerabilities |
none | none |
REQ-035-US-010-003 Detect XSS vulnerabilities |
none | none |
REQ-035-US-010-004 Verify authentication implementation |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-010-005 Verify authorization implementation |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-010-006 Check dependency vulnerabilities |
none | none |
REQ-035-US-010-007 Verify encryption usage |
none | none |
REQ-035-US-010-008 Check compliance requirements |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107 |
none |
REQ-035-US-010-009 Support custom security rules |
orchestrata_ultimate/verification/edge_cases/database EdgeCaseDatabase L13orchestrata_ultimate/verification/edge_cases/database add L34 |
none |
REQ-035-US-010-010 Create security constraint library |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-010-011 Implement secret detection |
none | none |
REQ-035-US-010-012 Implement vulnerability scanning |
none | none |
REQ-035-US-010-013 Create authentication verifier |
none | none |
REQ-035-US-010-014 Create authorization verifier |
none | none |
REQ-035-US-010-015 Add dependency auditing |
none | none |
REQ-035-US-010-016 Implement compliance checking |
none | none |
REQ-035-US-010-017 Create custom rule support |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraint_by_id L127 |
none |
REQ-035-US-010-018 Unit test: Secret detection works |
none | none |
REQ-035-US-010-019 Unit test: Vulnerability detection works |
none | none |
REQ-035-US-010-020 Integration test: Security scan completes |
none | none |
REQ-035-US-010-021 Integration test: Custom rules work |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraint_by_id L127 |
none |
REQ-035-US-011-001 Measure response time |
none | none |
REQ-035-US-011-002 Measure throughput |
none | none |
REQ-035-US-011-003 Measure memory usage |
none | none |
REQ-035-US-011-004 Measure cold start time |
none | none |
REQ-035-US-011-005 Support benchmark execution |
none | none |
REQ-035-US-011-006 Support performance profiling |
none | none |
REQ-035-US-011-007 Handle performance variability |
none | none |
REQ-035-US-011-008 Support percentile requirements |
none | none |
REQ-035-US-011-009 Generate performance reports |
orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18orchestrata_ultimate/verification/verification/feedback generate_feedback L36 |
none |
REQ-035-US-011-010 Create performance constraint library |
orchestrata_ultimate/verification/requirements/constraint_definitions get_constraints_by_type L107orchestrata_ultimate/verification/types ConstraintType L28 |
none |
REQ-035-US-011-011 Implement benchmarking |
none | none |
REQ-035-US-011-012 Create profiling support |
none | none |
REQ-035-US-011-013 Add statistical analysis |
none | none |
REQ-035-US-011-014 Implement variability handling |
none | none |
REQ-035-US-011-015 Create report generation |
orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18orchestrata_ultimate/verification/verification/feedback generate_feedback L36 |
none |
REQ-035-US-011-016 Unit test: Response time measurement works |
none | none |
REQ-035-US-011-017 Unit test: Throughput measurement works |
none | none |
REQ-035-US-011-018 Integration test: Performance report generated |
orchestrata_ultimate/verification/verification/feedback FeedbackGenerator L18orchestrata_ultimate/verification/verification/feedback generate_feedback L36 |
none |
REQ-035-US-011-019 Performance test: Measurements accurate |
none | none |