Orchestrata Ultimate / Specs / 097-testing-standards

097-testing-standards: Testing Standards

This specification defines the testing standards, templates, coverage requirements, and quality gates for the orchestrata-ultimate system. All specifications MUST include test scenarios following these standards.

At a glance

Reading this spec

This spec sets the testing standards, templates, coverage thresholds and quality gates for the whole orchestrata-ultimate system. It states that every specification must include test scenarios that follow these standards, so it works as the common contract between spec authors, developers, QA engineers and CI.

Four user stories frame it: a developer writing unit tests from clear templates and naming conventions (test_<module>.py files, test_<method>_<condition>_<expected> methods, fixtures for shared setup, descriptive assertions), a QA engineer running integration tests that check component interactions, a DevOps engineer running end-to-end tests, and a team lead enforcing coverage. The main body defines test types, execution requirements, coverage requirements, quality gates, required test scenarios and test data requirements, and its later sections cover data management, execution, naming, assertions, maintenance and performance testing.

Eleven general requirements (REQ-097-001 to 011) carry the substance. Coverage targets differ by component: Core Kernel 90% unit and 100% integration, CLI Commands 85% and 100%, Tool Implementations 80% and 100%, Integrations 70% and 80%. Suites run through standard pytest commands for unit, integration, end-to-end and coverage runs. Tests clean up after themselves, are deterministic and repeatable, and use factories and fakes instead of production data. Templates exist for unit (Arrange-Act-Assert), integration, end-to-end and property-based tests. Quality gates apply at pre-commit, pull request and release stages. Performance targets are P50 under 100 ms, P95 under 500 ms, at least 1000 requests per second and a baseline memory under 500 MB. A second group (REQ-097-101 to 105) comes from a Redis key-namespacing addendum for parallel safety: a gate that flags Redis writes in test files, allows a flagged site exactly two dispositions, limits a waiver marker to its own physical line, makes the gate blocking, and has the gate fall back to a regex-based scan when the AST parser cannot read a file.

This is a standards spec, so no code module cites it. The requirements traceability matrix lists every ID. The spec's own status line reads Complete and it tracks 4 files. No src/ symbol cites any of the 16 requirements, so measured trace coverage is 0.0%, and in tests/, 0 requirements are cited by a test-side @traces. All 16 requirements have no citing symbol; the standards are enforced through gates and review rather than through code annotations.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

No src/ symbol cites a requirement of this spec.

Checklists

checklists/requirements.md

The requirement list above comes from this file; 5 of 16 checkbox items are ticked.

Other tracked files

• specs/097-testing-standards/compliance-audit.md
• specs/097-testing-standards/redis-key-namespacing-addendum.md

Full spec text

Verbatim from specs/097-testing-standards/spec.md; headings are pushed down two levels.

SPEC-097: Testing Standards

Metadata

• Spec ID: 097-testing-standards
• Title: Testing Standards - Test Templates, Coverage Requirements, Quality Gates
• Author: Mohamed
• Status: Complete
• Created: 2026-04-08
• Last Updated: 2026-04-09
• Priority: P0
• Version: 1.1.0

Testing Requirements

Test Types

Test Execution Requirements

• All tests MUST be deterministic and repeatable
• All tests MUST clean up after execution
• Unit tests MUST complete within 5 minutes
• Integration tests MUST complete within 15 minutes
• E2E tests MUST complete within 30 minutes

Coverage Requirements

• Core Kernel: 90% unit, 100% integration
• CLI Commands: 85% unit, 100% integration
• Tool Implementations: 80% unit, 100% integration
• Integrations: 70% unit, 80% integration
• UI Components: 80% unit

Quality Gates

Required Test Scenarios

Every specification MUST include tests for:

1. Happy Path: Valid input produces expected output
2. Invalid Input: Invalid input produces appropriate error
3. Edge Cases: Boundary conditions handled correctly
4. Error Handling: All error conditions have test coverage
5. Concurrent Access: Thread-safety for shared resources
6. Graceful Degradation: System handles component failures
7. Security: Authorization and authentication enforced

Test Data Requirements

• Use factories for test data generation
• Never use production data in tests
• Use faker library for realistic test data
• Clean up all test data after tests complete

Overview

This specification defines the testing standards, templates, coverage requirements, and quality gates for the orchestrata-ultimate system. All specifications MUST include test scenarios following these standards.

User Scenarios

US-001: Developer Writing Unit Tests

As a developer, I want clear test templates so I can write consistent unit tests that follow project conventions.

Acceptance Criteria:

• Test file follows naming convention test_<module>.py
• Test methods follow test_<method>_<condition>_<expected> pattern
• Fixtures are used for common setup
• Assertions are descriptive

US-002: QA Engineer Running Integration Tests

As a QA engineer, I want integration tests that verify component interactions so I can detect integration issues before deployment.

Acceptance Criteria:

• All API endpoints have integration tests
• Tests use real dependencies (database, cache)
• Tests clean up after themselves
• Tests can run in CI/CD pipeline

US-003: DevOps Running E2E Tests

As a DevOps engineer, I want E2E tests that verify complete workflows so I can validate production readiness.

Acceptance Criteria:

• E2E tests cover critical user paths
• Tests use real system (no mocks)
• Tests can run in isolated environment
• Tests provide clear pass/fail results

US-004: Team Lead Enforcing Coverage

As a team lead, I want coverage requirements and quality gates so I can ensure code quality standards are met.

Acceptance Criteria:

• Coverage reports are generated automatically
• PRs blocked if coverage below threshold
• Quality gates enforced in CI/CD
• Coverage trends tracked over time

Terminology

Test

A function or method that verifies expected behavior under specific conditions.

Fixture

Reusable test setup code that provides consistent initial state for tests.

Coverage

Percentage of code executed during test runs (line, branch, function).

Quality Gate

Automated check that must pass before code can proceed to next stage.

Test Stub

Simplified implementation that replaces external dependencies.

Mock

Object that simulates behavior of real dependencies for testing.

Test Doubles

Generic term for test objects (stubs, mocks, spies, fakes).

Assertion

Statement that verifies expected vs actual result.

Test Suite

Collection of related tests executed together.

Functional Requirements

FR-001: All specifications MUST include test scenarios following defined templates

• Test files follow naming conventions
• Test methods follow defined patterns
• Fixtures used for common setup

FR-002: All code MUST achieve minimum coverage thresholds

• Core Kernel: 90% unit, 100% integration
• CLI Commands: 85% unit, 100% integration
• Tool Implementations: 80% unit, 100% integration
• Integrations: 70% unit, 80% integration
• UI Components: 80% unit

FR-003: Test suites MUST be executable via standardized commands

• Unit tests: pytest tests/unit/ -v
• Integration tests: pytest tests/integration/ -v -m integration
• E2E tests: pytest tests/e2e/ -v -m e2e
• Coverage: pytest tests/ --cov=src --cov-report=html

FR-004: All test suites MUST clean up after execution

• Test data removed after completion
• Temporary files cleaned up
• Mock resources released
• Database state reset

FR-005: Tests MUST be deterministic and repeatable

• No flaky tests due to timing
• No test order dependencies
• No reliance on external state
• No random behavior without seeding

FR-006: Test frameworks and tools MUST be properly configured

• pytest configured for async support
• Coverage reporting enabled
• Proper markers for test categories
• Parallel execution where safe

FR-007: Test templates MUST be provided for all test types

• Unit test template with Arrange-Act-Assert
• Integration test template with real dependencies
• E2E test template for complete workflows
• Property-based test template

FR-008: Quality gates MUST enforce standards at each CI/CD stage

• Pre-commit: Unit tests pass, coverage meets targets
• PR: Integration tests pass, no coverage decrease
• Release: E2E tests pass, benchmarks met

FR-009: Test data MUST use factories and fakes, never production data

• Test data factories for common types
• Faker for realistic test data
• No sensitive data in tests
• Safe test data isolation

FR-010: Error conditions MUST have corresponding test coverage

• All error scenarios tested
• Edge cases validated
• Exception handling verified
• Error messages validated

FR-011: Performance benchmarks MUST be defined and enforced

• Response time targets (P50 < 100ms, P95 < 500ms)
• Throughput minimums (1000 requests/second)
• Memory limits (< 500MB baseline)
• Load testing at 10x expected capacity

Error Conditions

Data Models

TestConfiguration

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["test_type", "coverage_target"],
  "properties": {
    "test_type": {
      "type": "string",
      "enum": ["unit", "integration", "e2e", "property"]
    },
    "coverage_target": {
      "type": "number",
      "minimum": 0,
      "maximum": 100
    },
    "timeout_seconds": {
      "type": "integer",
      "minimum": 1
    },
    "retry_count": {
      "type": "integer",
      "minimum": 0
    }
  }
}

TestResult

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["test_id", "status", "timestamp"],
  "properties": {
    "test_id": {
      "type": "string"
    },
    "status": {
      "type": "string",
      "enum": ["passed", "failed", "skipped", "error"]
    },
    "timestamp": {
      "type": "string",
      "format": "date-time"
    },
    "duration_ms": {
      "type": "number"
    },
    "error_message": {
      "type": "string"
    },
    "stack_trace": {
      "type": "string"
    }
  }
}

CoverageReport

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["timestamp", "line_coverage"],
  "properties": {
    "timestamp": {
      "type": "string",
      "format": "date-time"
    },
    "line_coverage": {
      "type": "number",
      "minimum": 0,
      "maximum": 100
    },
    "branch_coverage": {
      "type": "number",
      "minimum": 0,
      "maximum": 100
    },
    "files_covered": {
      "type": "integer"
    },
    "files_total": {
      "type": "integer"
    }
  }
}

Visual Architecture

Class Diagram

Diagram source shown as text: Mermaid could not parse it (Error: Parse error on line 2:).

classDiagram-direction LR
    <<dataclass>>
    class TraceInfo <<dataclass>> {
        -spec: str = ''
        -story: str = ''
        -reqs: List[str] = field(default_factory=list)
        -module: str = ''
        -class_name: str = ''
        -method: str = ''
        -file_path: str = ''
        -line_number: int = 0
        ---
        +to_dict(): Dict[str, Any]
    }
    <<dataclass>>
    class Requirement <<dataclass>> {
        -id: str
        -description: str
        -story_id: str = ''
        -spec_id: str = ''
        -priority: str = 'NORMAL'
        -category: str = ''
        ---
    }
    <<dataclass>>
    class UserStory <<dataclass>> {
        -id: str
        -title: str
        -as_a: str = ''
        -i_want: str = ''
        -so_that: str = ''
        -acceptance_criteria: List[str] = field(default_factory=list)
        -requirements: List[Requirement] = field(default_factory=list)
        -spec_id: str = ''
        ---
        +to_dict(): Dict[str, Any]
    }
    <<dataclass>>
    class ValidationReport <<dataclass>> {
        -verified: List[str] = field(default_factory=list)
        -missing: List[str] = field(default_factory=list)
        -orphan: List[str] = field(default_factory=list)
        -duplicates: Dict[str, List[str]] = field(default_factory=dict)
        ---
        {@property} +total_verified(): int
        {@property} +total_missing(): int
        {@property} +total_orphan(): int
        {@property} +coverage_percent(): float
        +to_dict(): Dict[str, Any]
    }
    <<dataclass>>
    class RTMReport <<dataclass>> {
        -entries: List[Dict[str, Any]] = field(default_factory=list)
        -validation: Optional[ValidationReport] = None
        -generated_at: str = ''
        -project: str = ''
        ---
        +to_dict(): Dict[str, Any]
    }
    UserStory *-- Requirement "1" "requirements" "1"
    RTMReport *-- ValidationReport "1" "validation" "1"

Test Standard Modification

• Changes to test standards require PR approval
• Breaking changes require version bump
• Changes must be documented in CHANGELOG
• Test templates are immutable once released

Quality Gate Enforcement

• Pre-commit: All unit tests pass
• PR: Coverage above threshold
• Release: All E2E tests pass

API Contracts

Test Runner Interface

async def run_tests(
    test_path: str,
    test_type: TestType,
    coverage_target: float,
    timeout_seconds: int = 300
) -> TestResult:
    """Execute test suite with configuration.
    
    Args:
        test_path: Path to test files
        test_type: Unit, integration, or e2e
        coverage_target: Minimum coverage percentage
        timeout_seconds: Maximum execution time
    
    Returns:
        TestResult with status, duration, coverage metrics
    """

Coverage Reporter Interface

def generate_coverage_report(
    source_path: str,
    report_format: str = "html"
) -> CoverageReport:
    """Generate coverage report for source code.
    
    Args:
        source_path: Source directory to analyze
        report_format: Output format (html, xml, json)
    
    Returns:
        CoverageReport with metrics and file counts
    """

Quality Gate Checker Interface

def check_quality_gates(
    test_results: TestResult[],
    coverage_report: CoverageReport,
    gate_type: GateType
) -> QualityGateResult:
    """Verify quality gates are met.
    
    Args:
        test_results: Latest test execution results
        coverage_report: Current coverage metrics
        gate_type: pre-commit, pr, or release
    
    Returns:
        QualityGateResult with pass/fail and details
    """

Test Data Factory Interface

class TestDataFactory:
    """Factory for generating test data."""
    
    def create_user(self, **overrides) -> User:
        """Create test user with realistic data."""
    
    def create_resource(self, **overrides) -> Resource:
        """Create test resource with valid state."""
    
    def cleanup_all(self):
        """Clean up all generated test data."""

Non-Functional Requirements

Performance

• Unit tests MUST complete within 5 minutes
• Integration tests MUST complete within 15 minutes
• E2E tests MUST complete within 30 minutes
• Test execution SHOULD support parallelization
• Coverage report generation MUST complete within 2 minutes

Scalability

• Test framework MUST support 10,000+ test cases
• Test execution MUST support distributed runners
• Coverage data MUST support incremental reporting
• Test results MUST be queryable and filterable

Reliability

• Test execution MUST have 99.9% success rate
• Flaky tests MUST be identified and quarantined automatically
• Test results MUST be reproducible across environments
• Coverage metrics MUST be consistent across runs

Usability

• Test documentation MUST be auto-generated from code
• Test failures MUST provide actionable error messages
• Test output MUST be human-readable and machine-parsable
• Test discovery MUST be intuitive and discoverable

Technical Design

Test Execution Architecture

┌─────────────────────────────────────────┐
│           Test Command Line             │
│         (pytest with plugins)           │
└────────────────┬────────────────────────┘
                 │
┌────────────────▼────────────────────────┐
│         Test Collection Phase           │
│  - Discover all test modules            │
│  - Load fixtures and plugins            │
│  - Parse markers and configurations     │
└────────────────┬────────────────────────┘
                 │
┌────────────────▼────────────────────────┐
│         Test Execution Phase            │
│  - Run tests in parallel workers        │
│  - Collect real-time coverage data      │
│  - Stream results to reporters          │
└────────────────┬────────────────────────┘
                 │
┌────────────────▼────────────────────────┐
│         Result Aggregation Phase        │
│  - Merge coverage data                  │
│  - Generate final reports               │
│  - Check quality gates                  │
│  - Update tracking metrics              │
└─────────────────────────────────────────┘

Coverage Tracking Design

• Line coverage: Track every executed source line
• Branch coverage: Track all conditional branches
• Function coverage: Track all function definitions
• Incremental coverage: Calculate coverage of changes only

Quality Gate Implementation

• Pre-commit hook: Run unit tests, check coverage, lint, type check
• PR validation: Run integration tests, verify no coverage regression
• Release validation: Run full E2E suite, performance benchmarks

Examples

Example 1: Complete Unit Test

import pytest
from orchestrata_ultimate.core import ConfigManager

class TestConfigManager:
    """Unit tests for ConfigManager."""
    
    @pytest.fixture
    def temp_dir(self, tmp_path):
        """Create temporary config directory."""
        return tmp_path / "config"
    
    @pytest.fixture
    def subject(self, temp_dir):
        """Create config manager under test."""
        return ConfigManager(config_dir=temp_dir)
    
    def test_load_config_given_valid_file_returns_config(
        self, subject, temp_dir
    ):
        """Test loading valid configuration file."""
        # Arrange
        config_file = temp_dir / "settings.yaml"
        config_file.write_text("key: value\ntimeout: 30")
        
        # Act
        config = subject.load_config("settings.yaml")
        
        # Assert
        assert config["key"] == "value"
        assert config["timeout"] == 30
    
    def test_load_config_given_missing_file_raises_error(self, subject):
        """Test handling of missing configuration file."""
        # Act/Assert
        with pytest.raises(FileNotFoundError):
            subject.load_config("nonexistent.yaml")

Example 2: Integration Test Suite

import pytest
import httpx
from orchestrata_ultimate.api import app

@pytest.mark.integration
class TestUserAPI:
    """Integration tests for user API endpoints."""
    
    @pytest.fixture
    async def client(self):
        """Create test client."""
        async with httpx.AsyncClient(
            app=app,
            base_url="http://test"
        ) as client:
            yield client
    
    async def test_create_user_given_valid_data_returns_201(
        self, client
    ):
        """Test successful user creation."""
        # Act
        response = await client.post(
            "/api/v1/users",
            json={"username": "testuser", "email": "test@example.com"}
        )
        
        # Assert
        assert response.status_code == 201
        assert response.json()["username"] == "testuser"

Example 3: E2E Workflow Test

import pytest
import subprocess
import json

@pytest.mark.e2e
class TestFullWorkflow:
    """End-to-end workflow tests."""
    
    def test_complete_development_cycle(self):
        """Test full development workflow from intent to deployment."""
        # Step 1: Create project
        result = subprocess.run(
            ["ultimate", "init", "test-project"],
            capture_output=True,
            text=True
        )
        assert result.returncode == 0
        
        try:
            # Step 2: Submit intent
            result = subprocess.run(
                ["ultimate", "intent", "Add user authentication"],
                capture_output=True,
                text=True
            )
            assert result.returncode == 0
            assert "Task created" in result.stdout
            
            # Step 3: Monitor progress
            for _ in range(30):  # Wait up to 5 minutes
                result = subprocess.run(
                    ["ultimate", "status", "--json"],
                    capture_output=True,
                    text=True
                )
                status = json.loads(result.stdout)
                if status["state"] == "completed":
                    break
            
            assert status["state"] == "completed"
            
        finally:
            # Cleanup
            subprocess.run(
                ["rm", "-rf", "test-project"],
                check=False
            )

Security

Test Environment Security

• Test environments MUST be isolated from production
• Test credentials MUST be separate from production credentials
• Test data MUST NOT contain sensitive information
• Test environments MUST be destroyed after test execution

Code Security

• Test code MUST be reviewed with same rigor as production code
• Test code MUST NOT introduce security vulnerabilities
• Tests MUST verify security controls are working
• Security tests MUST be part of all test suites

Test Data Security

• Usefaker library MUST be used for generating test data
• No real passwords, keys, or tokens in test code
• Test data MUST be randomized to prevent data leakage
• Access to test data SHOULD be logged for audit

Dependency Security

• All test dependencies MUST be scanned for vulnerabilities
• Test dependencies MUST be kept up to date
• Test framework versions MUST be pinned and reviewed
• Third-party test libraries MUST be vetted

Integration Points

CI/CD Pipeline Integration

• GitHub Actions workflow for automated test execution
• Jenkins pipeline support for custom deployments
• GitLab CI configuration templates available
• Pre-commit hooks for local validation
• Merge check integration for PR validation

IDE Integration

• VSCode extension for test discovery and execution
• PyCharm/IntelliJ plugin for test runner integration
• Vim/Neovim test runner commands
• Test coverage display in IDE

Monitoring Integration

• Test results sent to monitoring system
• Coverage trends tracked over time
• Test execution metrics collected
• Alert on test failure or coverage regression

Documentation Integration

• Test documentation auto-generated from code
• Coverage reports embedded in documentation
• Test examples embedded in API docs
• Test results visible in project dashboard

Success Criteria

Test Framework Adoption

• 100% of new code has corresponding tests
• All existing code has minimum 70% coverage
• All test suites run successfully in CI/CD
• Zero flaky tests in production environment

Quality Standards Met

• All tests complete within defined timeouts
• All quality gates enforced at appropriate stages
• Test failure provides actionable feedback
• Coverage reports generated and reviewed for all changes

Developer Experience

• All developers can run tests locally
• Test templates used consistently across codebase
• Test documentation is comprehensive and up-to-date
• New team members can write tests using provided templates

Automated Enforcement

• Pre-commit hooks enforce unit test requirements
• PR checks enforce integration test requirements
• Release pipeline enforces E2E test requirements
• Coverage regressions block merges

Observability

Logging Requirements

1. Component Logging
   • Log all major operations with appropriate log levels
   • Include correlation IDs for request tracing
   • Log performance metrics for critical operations
   • Log errors with full context for debugging

Metrics Requirements

1. Performance Metrics
   • Operation duration histograms
   • Request/error rate counters
   • Resource utilization gauges
   • Business metrics relevant to domain

Tracing Requirements

1. Distributed Tracing
   • Trace propagation across component boundaries
   • Span attributes for domain-specific context
   • Trace sampling based on operational needs

Alerting Requirements

1. Critical Alerts
   • Error rate spikes
   • Performance degradation
   • Resource exhaustion
   • Business metric anomalies

Dashboard Requirements

1. Operational Dashboard
   • Real-time system health
   • Performance trends
   • Error analysis
   • Resource monitoring

Revision History

1.1 Unit Tests

• Test individual functions/methods in isolation
• Mock external dependencies
• Target: 80% code coverage minimum

1.2 Integration Tests

• Test interactions between components
• Use real dependencies where possible
• Target: All API contracts tested

1.3 End-to-End Tests

• Test complete user workflows
• Use real system (no mocks)
• Target: Critical paths covered

1.4 Property-Based Tests

• Test invariants across random inputs
• Use libraries like hypothesis
• Target: All data transformations

2. Test Templates

2.1 Unit Test Template

import pytest
from unittest.mock import Mock, patch

# Module under test
from orchestrata_ultimate.module import ClassUnderTest

class TestClassUnderTest:
    """Unit tests for ClassUnderTest."""
    
    @pytest.fixture
    def subject(self):
        """Create subject under test."""
        return ClassUnderTest()
    
    @pytest.fixture
    def mock_dependency(self):
        """Mock external dependency."""
        return Mock()
    
    # Happy path tests
    def test_method_name_given_condition_expected_result(self, subject, mock_dependency):
        """Test method returns expected result for valid input."""
        # Arrange
        subject.dependency = mock_dependency
        mock_dependency.method.return_value = "expected"
        
        # Act
        result = subject.method("valid_input")
        
        # Assert
        assert result == "expected"
        mock_dependency.method.assert_called_once_with("valid_input")
    
    # Edge case tests
    def test_method_name_given_blank_string_returns_blank_string(self, subject):
        """Test method handles blank string input."""
        # Act
        result = subject.method("")
        
        # Assert
        assert result == ""
    
    def test_method_name_given_none_raises_error(self, subject):
        """Test method raises TypeError for None input."""
        # Act/Assert
        with pytest.raises(TypeError):
            subject.method(None)
    
    # Error handling tests
    def test_method_name_given_invalid_input_raises_custom_error(self, subject):
        """Test method raises ValidationError for invalid input."""
        # Act/Assert
        with pytest.raises(ValidationError) as exc_info:
            subject.method("invalid")
        
        assert "validation failed" in str(exc_info.value).lower()

2.2 Integration Test Template

import pytest
import httpx
from unittest.mock import Mock, patch

# For integration tests, use real components where possible
class TestAPIEndpoint:
    """Integration tests for API endpoint."""
    
    @pytest.fixture
    def base_url(self):
        """Base URL for API."""
        return "http://localhost:8080/api/v1"
    
    @pytest.fixture
    def auth_headers(self):
        """Authentication headers."""
        return {"Authorization": "Bearer test-token"}
    
    @pytest.mark.integration
    async def test_endpoint_returns_200_with_valid_request(self, base_url, auth_headers):
        """Test endpoint returns success for valid request."""
        # Act
        async with httpx.AsyncClient() as client:
            response = await client.get(
                f"{base_url}/resource",
                headers=auth_headers
            )
        
        # Assert
        assert response.status_code == 200
        assert "data" in response.json()
    
    @pytest.mark.integration
    async def test_endpoint_returns_401_without_auth(self, base_url):
        """Test endpoint returns 401 without authentication."""
        # Act
        async with httpx.AsyncClient() as client:
            response = await client.get(f"{base_url}/resource")
        
        # Assert
        assert response.status_code == 401
    
    @pytest.mark.integration
    async def test_endpoint_returns_429_when_rate_limited(self, base_url, auth_headers):
        """Test endpoint returns 429 when rate limited."""
        # Act - make many requests
        async with httpx.AsyncClient() as client:
            for _ in range(70):
                response = await client.get(
                    f"{base_url}/resource",
                    headers=auth_headers
                )
                if response.status_code == 429:
                    break
        
        # Assert
        assert response.status_code == 429
        assert "Retry-After" in response.headers

2.3 End-to-End Test Template

import pytest
import subprocess
import time
import os

class TestUserWorkflow:
    """End-to-end tests for complete user workflows."""
    
    @pytest.fixture(autouse=True)
    def setup_and_teardown(self):
        """Setup and teardown for each test."""
        # Setup: Start runtime
        subprocess.run(["ultimate", "init", "test-project"], check=True)
        subprocess.run(["ultimate", "start", "--daemon"], check=True)
        
        yield
        
        # Teardown: Stop runtime
        subprocess.run(["ultimate", "stop"], check=False)
        subprocess.run(["rm", "-rf", "test-project"], check=False)
    
    @pytest.mark.e2e
    def test_full_development_workflow(self):
        """Test complete development workflow."""
        # Step 1: Submit intent
        result = subprocess.run(
            ["ultimate", "intent", "Add user authentication"],
            capture_output=True,
            text=True
        )
        assert result.returncode == 0
        assert "Task created" in result.stdout
        
        # Step 2: Check status
        result = subprocess.run(
            ["ultimate", "status", "--json"],
            capture_output=True,
            text=True
        )
        assert result.returncode == 0
        status = json.loads(result.stdout)
        assert status["state"] == "running"
        
        # Step 3: Create checkpoint
        result = subprocess.run(
            ["ultimate", "checkpoint", "Before auth"],
            capture_output=True,
            text=True
        )
        assert result.returncode == 0

3. Coverage Requirements

3.1 Coverage Targets

3.2 Required Test Scenarios

For each specification, the following MUST be tested:

1. Happy Path: Valid input produces expected output
2. Invalid Input: Invalid input produces appropriate error
3. Edge Cases: Boundary conditions handled correctly
4. Error Handling: All error conditions have test coverage
5. Concurrent Access: Thread-safety for shared resources
6. Graceful Degradation: System handles component failures
7. Security: Authorization and authentication enforced

4. Quality Gates

4.1 Pre-Commit Gate

• All unit tests pass
• Code coverage meets target
• No linting errors
• No type errors

4.2 PR Gate

• All integration tests pass
• E2E tests pass for changed functionality
• Coverage not decreased
• Security scan passes

4.3 Release Gate

• Full test suite passes
• Performance benchmarks within limits
• No critical vulnerabilities
• Documentation updated

5. Test Data Management

5.1 Fixtures

• Use factories for test data
• Never use production data
• Use faker for realistic data
• Clean up after tests

5.2 Mocking Rules

• Mock external APIs
• Mock file system (use tmp_path)
• Mock time (use freezegun)
• Don't mock the subject under test

6. Test Execution

6.1 Commands

# Run all tests
pytest tests/ -v

# Run unit tests only
pytest tests/unit/ -v

# Run with coverage
pytest tests/ --cov=src --cov-report=html

# Run specific test
pytest tests/unit/test_module.py::TestClass::test_method -v

# Run integration tests
pytest tests/integration/ -v -m integration

# Run E2E tests
pytest tests/e2e/ -v -m e2e

# Run with performance profiling
pytest tests/ --profile

6.2 CI/CD Integration

# .github/workflows/test.yml
name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install -e .[test]
      
      - name: Run unit tests
        run: pytest tests/unit/ --cov
      
      - name: Run integration tests
        run: pytest tests/integration/ -v -m integration
      
      - name: Upload coverage
        uses: codecov/codecov-action@v4

7. Test Naming Conventions

7.1 Test File Names

• test_<module>.py for unit tests
• test_integration_<feature>.py for integration tests
• test_e2e_<workflow>.py for E2E tests

7.2 Test Method Names

Format: test_<method>_<condition>_<expected>

Examples:

• test_handle_edit_given_valid_input_returns_success
• test_handle_edit_given_empty_string_raises_error
• test_handle_edit_given_permission_denied_returns_403

8. Assertion Guidelines

8.1 Use Descriptive Assertions

# Bad
assert result

# Good
assert result is not None, "Result should not be None"
assert result.status == "success", f"Expected success, got {result.status}"

8.2 Test One Thing Per Test

# Bad
def test_method(self):
    result = subject.method(input)
    assert result.value == expected
    assert result.timestamp is not None

# Good
def test_method_returns_expected_value(self):
    result = subject.method(input)
    assert result.value == expected

def test_method_returns_timestamp(self):
    result = subject.method(input)
    assert result.timestamp is not None

9. Test Maintenance

9.1 Test Review Checklist

• [ ] Tests are deterministic (no flaky tests)
• [ ] Tests run in reasonable time (<5 min for unit)
• [ ] Tests are independent (no order dependency)
• [ ] Test data is realistic
• [ ] Error messages are actionable
• [ ] Tests document behavior

9.2 Test Documentation

Each test file MUST have:

• Module docstring explaining what's tested
• Class docstring for test class
• Descriptive docstrings for each test

10. Performance Testing

10.1 Benchmarks

• Response time: P50 < 100ms, P95 < 500ms
• Throughput: 1000 requests/second
• Memory: < 500MB baseline

10.2 Load Testing

• Test with 10x expected load
• Test degradation behavior
• Test recovery after overload

Related

• RTM page for 097-testing-standards
• All specs
• RTM overview
• Gates