Orchestrata Ultimate / Specs / 063-repl-tool

063-repl-tool: REPLTool Specification

The REPLTool provides an interactive Read-Eval-Print Loop (REPL) environment for testing code snippets and evaluating expressions across multiple programming languages. It enables developers to execute code in an isolated, sandboxed environment with configurable resource limits.

At a glance

Reading this spec

This spec defines REPLTool, an interactive read-eval-print loop that lets developers and QA engineers test code snippets, evaluate expressions and try shell commands without risking the host system. It is one of the tool specs in the Claude Code parity plan (spec 062). Each session gets an isolated subprocess or container-based execution environment, with sandbox isolation, resource limits and persistent session state.

Four user stories frame the scope: evaluating Python expressions, running multi-line JavaScript, running Bash commands under resource limits, and keeping session state across calls. The requirements split into session management (REQ-063-001 to 007: unique session identifiers, on-demand termination, an idle timeout with a 300 second default, and a global cap on concurrent sessions), code execution (Python, JavaScript on a persistent Node.js process, and Bash, each captured as stdout and stderr with the value of a trailing expression returned), resource limits (a wall-clock timeout, CPU and memory limits, an output size cap, a per-session expression limit and per-execution usage reporting), sandbox and security (a separate OS subprocess per session, filesystem allowlisting, best-effort detection of malicious code patterns and three sandbox levels), and the API (execute, get-session and paginated history payloads, input validation, and REST endpoints). Some items the spec lists, such as WebSocket streaming, a standalone CLI and transport-level security, are marked as not part of the traced requirements.

The design supports Python 3.11 and later, Node.js 18 and later, and Bash, and limits are enforced both with resource.setrlimit and with timeouts.

The implementation lives in the REPL tool module. The requirements traceability matrix lists each ID. The spec's own status line reads Production Ready and it tracks 3 files. Measured coverage is 23 of 32 requirements cited by at least one @traces in src/ (71.9%), spread over 1 module; the module citing the most of them is orchestrata_ultimate/tools/repl_tool (23 requirements). In tests/, 0 requirements are cited by a test-side @traces. 9 requirements have no citing symbol; the behaviour may still exist in code that was never annotated.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

Checklists

checklists/requirements.md

The requirement list above comes from this file; 25 of 32 checkbox items are ticked.

Other tracked files

• specs/063-repl-tool/compliance-audit.md

Full spec text

Verbatim from specs/063-repl-tool/spec.md; headings are pushed down two levels.

SPEC-063: REPLTool Specification

Metadata

• Spec ID: SPEC-063
• Title: REPLTool - Interactive Code Execution Environment
• Author: Mohamed
• Status: Production Ready
• Created: 2026-04-08
• Last Updated: 2026-04-09
• Priority: P1
• Version: 1.0.0

Overview

What

The REPLTool provides an interactive Read-Eval-Print Loop (REPL) environment for testing code snippets and evaluating expressions across multiple programming languages. It enables developers to execute code in an isolated, sandboxed environment with configurable resource limits.

Why

Developers and QA engineers need a secure, isolated environment to test code snippets, evaluate mathematical expressions, and validate shell commands without risking the host system. The REPLTool addresses this need by providing a managed execution environment with sandbox isolation, resource limits, and session state persistence.

How

The REPLTool creates isolated subprocess or container-based execution environments for each session. It supports Python (3.11+), JavaScript (Node.js 18+), and Bash execution with real-time output capture, configurable resource limits (CPU time, memory, output size), and comprehensive error reporting.

User Scenarios

US-001: Python Expression Evaluation

As a Developer
I want to evaluate Python expressions and capture results
So that I can test code snippets and verify logic in an isolated environment

Acceptance Criteria:

• Session initializes with Python language support
• Expressions execute and return correct output
• stdout and stderr are captured separately
• Expression results are returned for evaluation expressions

US-002: Multi-line JavaScript Code Execution

As a Frontend Developer
I want to define and execute multi-line JavaScript functions
So that I can test complex logic and function definitions

Acceptance Criteria:

• Multi-line code blocks execute correctly
• Function definitions persist within the session
• Function invocations return correct results
• Variables and state persist across expressions

US-003: Bash Command Execution with Resource Limits

As a DevOps Engineer
I want to execute bash commands with enforced resource limits
So that I can safely test shell commands without system impact

Acceptance Criteria:

• Commands execute within configured resource limits
• Timeout enforcement terminates long-running commands
• Memory limits prevent excessive resource consumption
• Appropriate error messages display on limit violations

US-004: Session State Persistence

As a Data Scientist
I want to maintain state across multiple expressions
So that I can build complex computations incrementally

Acceptance Criteria:

• Variable assignments persist within the session
• Import statements remain active across expressions
• Functions defined in earlier expressions are callable later
• Session state resets on session termination

Terminology

Data Models

Session Model

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "REPLSession",
  "type": "object",
  "properties": {
    "session_id": {
      "type": "string",
      "description": "Unique session identifier",
      "pattern": "^[a-zA-Z0-9]{8,32}$"
    },
    "language": {
      "type": "string",
      "enum": ["python", "javascript", "bash", "shell"],
      "description": "Programming language for the REPL"
    },
    "created_at": {
      "type": "string",
      "format": "date-time",
      "description": "Session creation timestamp"
    },
    "last_activity": {
      "type": "string",
      "format": "date-time",
      "description": "Last activity timestamp"
    },
    "resource_limits": {
      "$ref": "#/definitions/ResourceLimits"
    },
    "status": {
      "type": "string",
      "enum": ["active", "idle", "terminated", "error"],
      "description": "Current session status"
    }
  },
  "required": ["session_id", "language", "created_at", "status"],
  "definitions": {
    "ResourceLimits": {
      "type": "object",
      "properties": {
        "max_cpu_time": {
          "type": "integer",
          "minimum": 1,
          "maximum": 300,
          "default": 30,
          "description": "Maximum CPU time in seconds"
        },
        "max_memory_mb": {
          "type": "integer",
          "minimum": 16,
          "maximum": 4096,
          "default": 256,
          "description": "Maximum memory allocation in MB"
        },
        "max_output_size": {
          "type": "integer",
          "minimum": 1024,
          "maximum": 10485760,
          "default": 1048576,
          "description": "Maximum output size in bytes"
        },
        "max_expressions": {
          "type": "integer",
          "minimum": 1,
          "maximum": 10000,
          "default": 1000,
          "description": "Maximum number of expressions per session"
        }
      }
    }
  }
}

Execution Request Model

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ExecutionRequest",
  "type": "object",
  "properties": {
    "session_id": {
      "type": "string",
      "description": "Session identifier"
    },
    "code": {
      "type": "string",
      "description": "Code to execute",
      "minLength": 1,
      "maxLength": 1048576
    },
    "timeout_ms": {
      "type": "integer",
      "minimum": 100,
      "maximum": 300000,
      "description": "Execution timeout in milliseconds"
    }
  },
  "required": ["session_id", "code"]
}

Execution Result Model

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ExecutionResult",
  "type": "object",
  "properties": {
    "session_id": {
      "type": "string",
      "description": "Session identifier"
    },
    "success": {
      "type": "boolean",
      "description": "Execution success status"
    },
    "output": {
      "type": "string",
      "description": "Standard output"
    },
    "error": {
      "type": "string",
      "description": "Standard error output"
    },
    "result": {
      "type": ["string", "number", "boolean", "object", "array", "null"],
      "description": "Expression result (if applicable)"
    },
    "execution_time_ms": {
      "type": "integer",
      "description": "Execution time in milliseconds"
    },
    "timestamp": {
      "type": "string",
      "format": "date-time",
      "description": "Execution timestamp"
    }
  },
  "required": ["session_id", "success", "execution_time_ms", "timestamp"]
}

Session Configuration Model

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "SessionConfig",
  "type": "object",
  "properties": {
    "language": {
      "type": "string",
      "enum": ["python", "javascript", "bash", "shell"],
      "description": "Programming language"
    },
    "sandbox_mode": {
      "type": "string",
      "enum": ["strict", "standard", "relaxed"],
      "default": "standard",
      "description": "Sandbox enforcement level"
    },
    "environment": {
      "type": "object",
      "description": "Environment variables",
      "additionalProperties": {
        "type": "string"
      }
    },
    "working_directory": {
      "type": "string",
      "description": "Working directory for execution"
    },
    "resource_limits": {
      "$ref": "#/definitions/ResourceLimits"
    }
  },
  "required": ["language"],
  "definitions": {
    "ResourceLimits": {
      "type": "object",
      "properties": {
        "max_cpu_time": {
          "type": "integer",
          "minimum": 1,
          "maximum": 300
        },
        "max_memory_mb": {
          "type": "integer",
          "minimum": 16,
          "maximum": 4096
        },
        "max_output_size": {
          "type": "integer",
          "minimum": 1024,
          "maximum": 10485760
        },
        "max_expressions": {
          "type": "integer",
          "minimum": 1,
          "maximum": 10000
        }
      }
    }
  }
}

Functional Requirements

Core Features

Session Management

• FR-001: The system MUST support creating new REPL sessions with unique identifiers
• FR-002: The system MUST support terminating sessions and releasing resources
• FR-003: The system MUST support session state persistence across expressions
• FR-004: The system MUST support session idle timeout (default: 300 seconds)
• FR-005: The system MUST support session activity tracking

Code Execution

• FR-006: The system MUST support Python code execution (version 3.11+)
• FR-007: The system MUST support JavaScript code execution (Node.js 18+)
• FR-008: The system MUST support Bash command execution
• FR-009: The system MUST capture stdout and stderr separately
• FR-010: The system MUST return expression results for evaluation expressions

Resource Management

• FR-011: The system MUST enforce CPU time limits per execution
• FR-012: The system MUST enforce memory allocation limits
• FR-013: The system MUST enforce output size limits
• FR-014: The system MUST terminate executions that exceed resource limits
• FR-015: The system MUST report resource usage after each execution

Sandbox Security

• FR-016: The system MUST isolate each session in a separate sandbox
• FR-017: The system MUST restrict file system access in sandbox mode
• FR-018: The system MUST restrict network access in sandbox mode
• FR-019: The system MUST prevent execution of malicious code
• FR-020: The system MUST support multiple sandbox levels (strict/standard/relaxed)

5.2 API Features

5.2.1 REST API Endpoints

• FR-021: The system MUST provide POST /sessions for creating sessions
• FR-022: The system MUST provide GET /sessions/{id} for session info
• FR-023: The system MUST provide DELETE /sessions/{id} for session termination
• FR-024: The system MUST provide POST /sessions/{id}/execute for code execution
• FR-025: The system MUST provide GET /sessions/{id}/history for execution history

5.2.2 WebSocket Support

• FR-026: The system MUST support WebSocket connections for real-time REPL
• FR-027: The system MUST support streaming output via WebSocket
• FR-028: The system MUST support bidirectional communication

5.3 User Interface Features

5.3.1 CLI Interface

• FR-029: The system MUST provide a command-line interface for REPL interaction
• FR-030: The CLI MUST support session creation with language selection
• FR-031: The CLI MUST support interactive input mode
• FR-032: The CLI MUST support executing code from files

Error Conditions

API Contracts

7.1 Create Session

Endpoint: POST /api/v1/sessions
Content-Type: application/json

Request:

{
  "language": "python",
  "sandbox_mode": "standard",
  "resource_limits": {
    "max_cpu_time": 30,
    "max_memory_mb": 256,
    "max_output_size": 1048576
  }
}

Response (201 Created):

{
  "session_id": "abc123xyz",
  "language": "python",
  "created_at": "2026-04-08T10:30:00Z",
  "status": "active",
  "resource_limits": {
    "max_cpu_time": 30,
    "max_memory_mb": 256,
    "max_output_size": 1048576
  }
}

7.2 Execute Code

Endpoint: POST /api/v1/sessions/{session_id}/execute
Content-Type: application/json

Request:

{
  "code": "print('Hello, World!')",
  "timeout_ms": 5000
}

Response (200 OK):

{
  "session_id": "abc123xyz",
  "success": true,
  "output": "Hello, World!\n",
  "error": "",
  "result": null,
  "execution_time_ms": 12,
  "timestamp": "2026-04-08T10:30:05Z"
}

7.3 Get Session Info

Endpoint: GET /api/v1/sessions/{session_id}
Response (200 OK):

{
  "session_id": "abc123xyz",
  "language": "python",
  "created_at": "2026-04-08T10:30:00Z",
  "last_activity": "2026-04-08T10:30:05Z",
  "status": "active",
  "expression_count": 5,
  "resource_limits": {
    "max_cpu_time": 30,
    "max_memory_mb": 256,
    "max_output_size": 1048576
  }
}

7.4 Terminate Session

Endpoint: DELETE /api/v1/sessions/{session_id}
Response (204 No Content): Null body

7.5 Get Execution History

Endpoint: GET /api/v1/sessions/{session_id}/history?limit=10&offset=0
Response (200 OK):

{
  "session_id": "abc123xyz",
  "total": 5,
  "limit": 10,
  "offset": 0,
  "executions": [
    {
      "code": "print('Hello, World!')",
      "success": true,
      "execution_time_ms": 12,
      "timestamp": "2026-04-08T10:30:05Z"
    }
  ]
}

Non-Functional Requirements

8.1 Security Requirements

8.1.1 Authentication and Authorization

• SEC-001: All API endpoints MUST require authentication unless explicitly public
• SEC-002: Session operations MUST be restricted to authorized users only
• SEC-003: The system MUST validate user permissions for resource access
• SEC-004: Session identifiers MUST be cryptographically secure and unguessable

8.1.2 Sandboxing and Isolation

• SEC-005: Each session MUST execute in a completely isolated environment
• SEC-006: Sessions MUST NOT be able to access other session data or state
• SEC-007: File system access MUST be restricted based on sandbox mode
• SEC-008: Network access MUST be limited according to configured sandbox level
• SEC-009: Process isolation MUST prevent cross-session interference

8.1.3 Resource Protection

• SEC-010: Resource limits MUST be strictly enforced to prevent DoS
• SEC-011: CPU time limits MUST prevent infinite loop attacks
• SEC-012: Memory limits MUST prevent memory exhaustion attacks
• SEC-013: Output size limits MUST prevent buffer overflow attacks
• SEC-014: Execution timeouts MUST terminate runaway processes immediately

8.1.4 Input Validation and Sanitization

• SEC-015: All code input MUST be validated for malicious patterns
• SEC-016: Dangerous system calls MUST be blocked via seccomp filtering
• SEC-017: Shell injection attempts MUST be detected and prevented
• SEC-018: File system traversal patterns MUST be blocked
• SEC-019: Import of dangerous modules MUST be restricted (os, sys, subprocess, etc.)

8.1.5 Data Protection

• SEC-020: Execution results MUST be encrypted in transit (HTTPS/TLS 1.3+)
• SEC-021: Session state MUST be protected from unauthorized access
• SEC-022: Sensitive data MUST NOT be logged in plaintext
• SEC-023: Audit logs MUST be tamper-evident and securely stored
• SEC-024: Session cleanup MUST securely delete all sensitive data

8.1.6 Audit and Compliance

• SEC-025: All security events MUST be logged with timestamps
• SEC-026: Failed authentication attempts MUST be logged and monitored
• SEC-027: Sandbox violations MUST trigger immediate alerts
• SEC-028: Resource limit violations MUST be tracked for patterns
• SEC-029: Audit logs MUST be retained for minimum 90 days
• SEC-030: Security incidents MUST provide complete forensic data

8.1.7 Threat Mitigation

• SEC-031: The system MUST prevent code injection attacks
• SEC-032: The system MUST prevent remote code execution vulnerabilities
• SEC-033: The system MUST prevent privilege escalation attempts
• SEC-034: The system MUST detect and block malicious code patterns
• SEC-035: Rate limiting MUST prevent brute force and abuse attacks

8.1.8 Secure Configuration

• SEC-036: Default sandbox mode MUST be "standard" (secure by default)
• SEC-037: Default resource limits MUST prevent abuse while allowing normal use
• SEC-038: Debug mode MUST NOT expose sensitive information in production
• SEC-039: Configuration changes MUST require proper authorization
• SEC-040: Security configurations MUST be version controlled and auditable

8.1.9 Incident Response

• SEC-041: Security violations MUST be logged with full context
• SEC-042: The system MUST support real-time security monitoring
• SEC-043: Automatic session termination MUST occur on critical violations
• SEC-044: The system MUST provide security incident reporting
• SEC-045: Recovery procedures MUST be documented and tested

8.2 Performance Requirements

8.3 Reliability Requirements

• System MUST achieve 99.9% uptime
• Sessions MUST be recoverable after system restart
• Failed executions MUST NOT affect other concurrent sessions
• Session cleanup MUST occur within 60 seconds of termination

8.4 Scalability Requirements

• System MUST support horizontal scaling via load balancing
• Session state MUST be distributable across multiple instances
• Resource limits MUST be enforced consistently in distributed mode

Security Requirements

This section consolidates all security-related requirements for the REPLTool system. All security requirements MUST be enforced at all times.

Authentication and Authorization

• SEC-001: All API endpoints MUST require authentication unless explicitly public
• SEC-002: Session operations MUST be restricted to authorized users only
• SEC-003: The system MUST validate user permissions for resource access
• SEC-004: Session identifiers MUST be cryptographically secure and unguessable

Sandboxing and Isolation

• SEC-005: Each session MUST execute in a completely isolated environment
• SEC-006: Sessions MUST NOT be able to access other session data or state
• SEC-007: File system access MUST be restricted based on sandbox mode
• SEC-008: Network access MUST be limited according to configured sandbox level
• SEC-009: Process isolation MUST prevent cross-session interference

Resource Protection

• SEC-010: Resource limits MUST be strictly enforced to prevent DoS
• SEC-011: CPU time limits MUST prevent infinite loop attacks
• SEC-012: Memory limits MUST prevent memory exhaustion attacks
• SEC-013: Output size limits MUST prevent buffer overflow attacks
• SEC-014: Execution timeouts MUST terminate runaway processes immediately

Input Validation and Sanitization

• SEC-015: All code input MUST be validated for malicious patterns
• SEC-016: Dangerous system calls MUST be blocked via seccomp filtering
• SEC-017: Shell injection attempts MUST be detected and prevented
• SEC-018: File system traversal patterns MUST be blocked
• SEC-019: Import of dangerous modules MUST be restricted (os, sys, subprocess, etc.)

Data Protection

• SEC-020: Execution results MUST be encrypted in transit (HTTPS/TLS 1.3+)
• SEC-021: Session state MUST be protected from unauthorized access
• SEC-022: Sensitive data MUST NOT be logged in plaintext
• SEC-023: Audit logs MUST be tamper-evident and securely stored
• SEC-024: Session cleanup MUST securely delete all sensitive data

Audit and Compliance

• SEC-025: All security events MUST be logged with timestamps
• SEC-026: Failed authentication attempts MUST be logged and monitored
• SEC-027: Sandbox violations MUST trigger immediate alerts
• SEC-028: Resource limit violations MUST be tracked for patterns
• SEC-029: Audit logs MUST be retained for minimum 90 days
• SEC-030: Security incidents MUST provide complete forensic data

Threat Mitigation

• SEC-031: The system MUST prevent code injection attacks
• SEC-032: The system MUST prevent remote code execution vulnerabilities
• SEC-033: The system MUST prevent privilege escalation attempts
• SEC-034: The system MUST detect and block malicious code patterns
• SEC-035: Rate limiting MUST prevent brute force and abuse attacks

Secure Configuration

• SEC-036: Default sandbox mode MUST be "standard" (secure by default)
• SEC-037: Default resource limits MUST prevent abuse while allowing normal use
• SEC-038: Debug mode MUST NOT expose sensitive information in production
• SEC-039: Configuration changes MUST require proper authorization
• SEC-040: Security configurations MUST be version controlled and auditable

Incident Response

• SEC-041: Security violations MUST be logged with full context
• SEC-042: The system MUST support real-time security monitoring
• SEC-043: Automatic session termination MUST occur on critical violations
• SEC-044: The system MUST provide security incident reporting
• SEC-045: Recovery procedures MUST be documented and tested

Technical Design

9.1 Architecture Overview

The REPLTool follows a microservice architecture with the following components:

1. API Layer: FastAPI-based REST API handling session management and execution requests
2. Session Manager: Orchestrates session lifecycle, state persistence, and resource allocation
3. Execution Engine: Manages language-specific interpreters (Python, JavaScript, Bash)
4. Sandbox Manager: Implements isolation via gVisor containers or subprocess isolation
5. Resource Monitor: Tracks CPU, memory, and output usage per session

9.2 Execution Flow

1. Client creates session via POST /sessions
2. Session Manager initializes sandbox environment
3. Client sends code via POST /sessions/{id}/execute
4. Execution Engine validates input and applies resource limits
5. Code runs in isolated sandbox with timeout enforcement
6. Output captured and returned to client
7. Session state updated with execution metadata

9.3 Sandbox Implementation

Examples

10.1 Python Session Example

# Create Python session
curl -X POST http://localhost:8000/api/v1/sessions \
  -H "Content-Type: application/json" \
  -d '{"language": "python", "sandbox_mode": "standard"}'

# Response
{"session_id": "abc123xyz", "status": "active", ...}

# Execute code
curl -X POST http://localhost:8000/api/v1/sessions/abc123xyz/execute \
  -H "Content-Type: application/json" \
  -d '{"code": "print(2 + 2)"}'

# Response
{"success": true, "output": "4\n", "result": null, ...}

10.2 JavaScript Session Example

# Create JavaScript session
curl -X POST http://localhost:8000/api/v1/sessions \
  -H "Content-Type: application/json" \
  -d '{"language": "javascript"}'

# Execute multi-line code
curl -X POST http://localhost:8000/api/v1/sessions/xyz789abc/execute \
  -H "Content-Type: application/json" \
  -d '{"code": "function factorial(n) { return n <= 1 ? 1 : n * factorial(n-1); }\nfactorial(5)"}'

# Response
{"success": true, "output": "", "result": 120, ...}

10.3 CLI Usage Example

# Start interactive Python REPL
repltool interactive --language python

# Execute code from file
repltool execute --file script.py --language python

# Create session with custom limits
repltool create --language bash --max-cpu-time 10 --max-memory 128

Security Requirements

11.1 Sandboxing Architecture

The REPLTool implements a multi-layered sandbox architecture:

1. Process Isolation: Each session runs in an isolated subprocess
2. Container Isolation: Optional Docker-based container isolation for strict mode
3. Filesystem Restrictions: Read-only filesystem with tmpfs for temporary files
4. Network Restrictions: No network access in strict mode; limited in standard mode
5. System Call Filtering: seccomp-bpf for system call restrictions

11.2 Resource Limits

11.3 Security Policies

11.3.1 Strict Mode

• No file system access
• No network access
• No subprocess spawning
• No import of external modules (Python)
• Maximum resource restrictions

11.3.2 Standard Mode

• Temporary directory access only
• No network access
• Limited subprocess (shell built-ins only)
• Whitelisted module imports only

11.3.3 Relaxed Mode

• Temporary directory access
• Local network only
• Subprocess allowed (restricted)
• Most modules allowed

11.4 Malicious Code Detection

The system detects and blocks:

• Infinite loops (via timeout)
• Memory exhaustion attacks
• Recursive function abuse
• Import of dangerous modules (os, sys, subprocess in Python)
• Shell injection attempts
• File system traversal attempts

11.5 Audit Logging

All executions are logged with:

• Session ID
• Timestamp
• Code hash (for security analysis)
• Execution result
• Resource usage
• Error conditions

Governance

12.1 Policy Enforcement

• Code execution requires tool:execute permission
• Resource limits enforced by default
• Execution logged for audit trail

12.2 Audit Logging

async def log_execution(code: str, language: str, result: dict):
    await audit_log({
        "event_type": "repl_execution",
        "language": language,
        "success": result.success,
        "duration_ms": result.duration_ms
    })

Dependencies

13.1 Runtime Dependencies

13.2 Python Dependencies

13.3 External Services

Testing Requirements

Test Environment Requirements

Test Data Requirements

• Test Sessions: Pre-created sessions for Python, JavaScript, and Bash with various states
• Code Samples: Valid, invalid, and malicious code samples for each language
• Resource Scenarios: Code samples designed to trigger limit violations
• Error Scenarios: Request payloads that trigger each error condition

Test Coverage Requirements

Test Execution Requirements

• All tests MUST pass before release
• Performance tests MUST meet latency thresholds
• Security tests MUST pass without vulnerabilities
• Integration tests MUST run against actual (not mocked) dependencies

Testing Requirements

14.1 Unit Tests

14.2 Integration Tests

14.3 Security Tests

14.4 Performance Tests

14.5 Test Execution

# Run all unit tests
pytest tests/unit/ -v --cov=repltool --cov-report=html

# Run integration tests
pytest tests/integration/ -v

# Run security tests
pytest tests/security/ -v

# Run performance tests
pytest tests/performance/ -v --benchmark-only

# Run all tests
pytest tests/ -v --cov=repltool

Integration Points

15.1 Integration with Orchestrata CLI

The REPLTool integrates with the Orchestrata CLI as tool ID repl_execute:

{
  "tool": "repl_execute",
  "parameters": {
    "language": "python",
    "code": "print('Hello')",
    "sandbox_mode": "standard",
    "resource_limits": {
      "max_cpu_time": 30,
      "max_memory_mb": 256
    }
  }
}

15.2 Integration with Redis (Optional)

For distributed deployments, sessions can be cached in Redis:

# Session state serialization
await redis.set(f"session:{session_id}", json.dumps(session_state), ex=3600)

15.3 Integration with Prometheus Metrics

Exported metrics include:

• repl_sessions_created_total - Counter of created sessions
• repl_executions_total - Counter of executions
• repl_execution_duration_seconds - Histogram of execution times
• repl_active_sessions - Gauge of active sessions

Success Criteria

16.1 Functional Acceptance

• [ ] Sessions can be created for Python, JavaScript, and Bash
• [ ] Code execution returns correct output and results
• [ ] Resource limits are enforced correctly
• [ ] Sandbox isolation prevents unauthorized access
• [ ] Session state persists across expressions
• [ ] API endpoints respond correctly
• [ ] WebSocket connections work for real-time REPL

16.2 Security Acceptance

• [ ] Sandbox escapes are prevented
• [ ] Resource exhaustion is prevented
• [ ] Malicious code is detected and blocked
• [ ] Input validation prevents injection attacks

16.3 Performance Acceptance

• [ ] Session creation completes in < 100ms
• [ ] Code execution completes in < 500ms
• [ ] System handles 50 concurrent sessions
• [ ] Memory usage stays under limits

16.4 Reliability Acceptance

• [ ] System recovers from crashes gracefully
• [ ] Sessions are cleaned up on termination
• [ ] Errors are logged appropriately
• [ ] No data corruption occurs

Observability

Logging Requirements

1. Component Logging
   • Log all major operations with appropriate log levels
   • Include correlation IDs for request tracing
   • Log performance metrics for critical operations
   • Log errors with full context for debugging

Metrics Requirements

1. Performance Metrics
   • Operation duration histograms
   • Request/error rate counters
   • Resource utilization gauges
   • Business metrics relevant to domain

Tracing Requirements

1. Distributed Tracing
   • Trace propagation across component boundaries
   • Span attributes for domain-specific context
   • Trace sampling based on operational needs

Alerting Requirements

1. Critical Alerts
   • Error rate spikes
   • Performance degradation
   • Resource exhaustion
   • Business metric anomalies

Dashboard Requirements

1. Operational Dashboard
   • Real-time system health
   • Performance trends
   • Error analysis
   • Resource monitoring

Revision History

End of Specification

Related

• RTM page for 063-repl-tool
• All specs
• RTM overview
• Gates