Orchestrata Ultimate / Specs / 068-schedule-cron-tool

068-schedule-cron-tool: ScheduleCronTool - Cron Job Scheduling with Timezone Support

A production-ready tool for scheduling recurring jobs using cron expressions with comprehensive timezone support, missed job recovery, and concurrent execution protection.

At a glance

Reading this spec

This spec defines ScheduleCronTool, a tool for scheduling recurring jobs with cron expressions. It adds timezone support for distributed teams, recovery of missed jobs after system failures, and protection against concurrent execution of sensitive jobs. Its three scenarios are daily report generation, backup coordination across a distributed team and critical health-check monitoring.

The functional requirements (FR-001 onward) cover cron expression parsing, timezone support, job persistence, missed job recovery, concurrent execution protection and job execution. The overview describes the design as cron parsing with timezones, persistent job storage, a background asynchronous scheduler, recovery mechanisms and distributed locking.

The 30 requirement IDs (REQ-068-NNN, plus three REQ-068-EXT extensions) record where the implementation differs from the overview, which is the most useful thing a reader can learn here. Standard 5-field cron expressions are supported with lists, ranges and steps; timezones are IANA names through the standard-library zoneinfo; timestamps are stored in UTC. Job definitions persist in Redis through ScheduleCronStore, and the requirement notes this deviates from the SQLite store in the overview (a brief-directed fork). Jobs run asynchronously through asyncio.create_subprocess_shell, can be listed with filters and triggered manually with run_job_now, and old execution records can be cleared. The requirements mark as not implemented: UTC offset notation such as +05:30, job import and export, bulk operations, automatic missed-execution recovery on startup (it needs a long-running scheduler process), true FIFO queue semantics (allow and queue both run unlocked in this MVP) and an exponential backoff option.

The implementation is in the schedule cron tool module. The requirements traceability matrix lists each ID. The spec's own status line reads Complete and it tracks 3 files. Measured coverage is 23 of 30 requirements cited by at least one @traces in src/ (76.7%), spread over 1 module; the module citing the most of them is orchestrata_ultimate/tools/schedule_cron_tool (23 requirements). In tests/, 0 requirements are cited by a test-side @traces. The 7 requirements with no citing symbol are four REQ-068 items and the three EXT items; most are the not-implemented features above.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

Checklists

checklists/requirements.md

The requirement list above comes from this file; 23 of 30 checkbox items are ticked.

Other tracked files

• specs/068-schedule-cron-tool/compliance-audit.md

Full spec text

Verbatim from specs/068-schedule-cron-tool/spec.md; headings are pushed down two levels.

SPEC-068: ScheduleCronTool - Cron Job Scheduling with Timezone Support

Metadata

• Spec ID: 068-schedule-cron-tool
• Title: ScheduleCronTool - Cron Job Scheduling with Timezone Support
• Author: Mohamed
• Status: Complete
• Created: 2026-04-08
• Last Updated: 2026-04-09
• Priority: P1
• Version: 1.0.0

Overview

What

A production-ready tool for scheduling recurring jobs using cron expressions with comprehensive timezone support, missed job recovery, and concurrent execution protection.

Why

• Automate recurring tasks with cron-like scheduling
• Handle timezone-aware job execution for distributed teams
• Ensure reliable job recovery after system failures
• Prevent concurrent execution conflicts for sensitive jobs

How

• Cron expression parsing with timezone support
• Persistent job storage with SQLite
• Background scheduler with async execution
• Recovery mechanisms for missed jobs
• Distributed locking for concurrent execution protection

Tool Interface

from datetime import datetime, timezone
from typing import Optional
from enum import Enum

class MissedJobPolicy(Enum):
    """Policy for handling missed jobs."""
    SKIP = "skip"
    RUN_ONCE = "run_once"
    RUN_ALL = "run_all"
    NOTIFY_ONLY = "notify_only"

class ConcurrencyPolicy(Enum):
    """Policy for handling concurrent job execution."""
    ALLOW = "allow"
    SKIP = "skip"
    QUEUE = "queue"
    FORBID = "forbid"

class JobStatus(Enum):
    """Job execution status."""
    PENDING = "pending"
    RUNNING = "running"
    COMPLETED = "completed"
    FAILED = "failed"
    CANCELLED = "cancelled"
    RECOVERED = "recovered"

@dataclass
class CronJob:
    """Cron job definition."""
    id: str
    name: str
    cron_expression: str
    timezone: str
    command: str
    args: dict
    enabled: bool = True
    miss_policy: MissedJobPolicy = MissedJobPolicy.RUN_ONCE
    concurrency_policy: ConcurrencyPolicy = ConcurrencyPolicy.ALLOW
    max_retries: int = 3
    retry_delay_seconds: int = 60
    timeout_seconds: int = 300
    description: str = ""
    tags: list[str] = field(default_factory=list)
    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
    updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
    last_run_at: Optional[datetime] = None
    next_run_at: Optional[datetime] = None

@dataclass
class JobExecution:
    """Job execution record."""
    id: str
    job_id: str
    status: JobStatus
    started_at: datetime
    completed_at: Optional[datetime] = None
    error_message: Optional[str] = None
    output: Optional[str] = None
    retry_count: int = 0
    is_recovery: bool = False
    execution_node: Optional[str] = None

@dataclass
class JobScheduleResult:
    """Result of scheduling a job."""
    success: bool
    job_id: Optional[str] = None
    next_run_at: Optional[datetime] = None
    error_message: Optional[str] = None

@dataclass
class JobExecutionResult:
    """Result of job execution."""
    success: bool
    execution_id: Optional[str] = None
    output: Optional[str] = None
    error_message: Optional[str] = None
    duration_seconds: Optional[float] = None

class ScheduleCronTool(BaseTool):
    """Schedule and manage recurring jobs using cron expressions."""
    
    name = "ScheduleCronTool"
    description = "Schedule recurring jobs using cron expressions with timezone support"
    
    async def schedule_job(
        self,
        name: str,
        cron_expression: str,
        command: str,
        timezone: str = "UTC",
        args: dict = None,
        miss_policy: str = "run_once",
        concurrency_policy: str = "allow",
        max_retries: int = 3,
        retry_delay_seconds: int = 60,
        timeout_seconds: int = 300,
        description: str = "",
        tags: list[str] = None,
        enabled: bool = True
    ) -> JobScheduleResult:
        """Schedule a new cron job."""
    
    async def unschedule_job(
        self,
        job_id: str
    ) -> ToolResult:
        """Remove a scheduled job."""
    
    async def update_job(
        self,
        job_id: str,
        **updates
    ) -> ToolResult:
        """Update an existing job."""
    
    async def list_jobs(
        self,
        enabled_only: bool = False,
        tags: list[str] = None,
        limit: int = 100
    ) -> list[dict]:
        """List scheduled jobs."""
    
    async def get_job_status(
        self,
        job_id: str
    ) -> dict:
        """Get job status and execution history."""
    
    async def run_job_now(
        self,
        job_id: str,
        force: bool = False
    ) -> JobExecutionResult:
        """Trigger immediate job execution."""
    
    async def cancel_job(
        self,
        job_id: str
    ) -> ToolResult:
        """Cancel a running job."""
    
    async def get_next_runs(
        self,
        job_id: str,
        count: int = 5
    ) -> list[datetime]:
        """Get upcoming scheduled run times."""
    
    async def validate_cron_expression(
        self,
        expression: str,
        timezone: str = "UTC"
    ) -> dict:
        """Validate a cron expression."""
    
    async def pause_job(
        self,
        job_id: str
    ) -> ToolResult:
        """Pause a job (disable without removing)."""
    
    async def resume_job(
        self,
        job_id: str
    ) -> ToolResult:
        """Resume a paused job."""
    
    async def get_execution_history(
        self,
        job_id: str = None,
        status: str = None,
        limit: int = 50
    ) -> list[dict]:
        """Get job execution history."""
    
    async def clear_history(
        self,
        job_id: str = None,
        older_than_days: int = 30
    ) -> ToolResult:
        """Clear execution history records."""

User Scenarios

Scenario 1: Daily Report Generation

User: DevOps engineer automating daily system reports Context: Needs to run a report generation script at 8:00 AM in their local timezone (America/New_York) Flow:

1. Schedule job with cron expression 0 8 * * *
2. Set timezone to America/New_York
3. Configure miss_policy to run_once to handle missed runs
4. Set concurrency_policy to forbid to prevent duplicate reports
5. Job executes daily at 8 AM EST/EDT automatically

Scenario 2: Distributed Team Backup Coordination

User: Platform team coordinating backups across global regions Context: Multiple team members in different timezones need to schedule backup jobs Flow:

1. Team lead creates jobs for each region (US, EU, APAC)
2. Each job uses corresponding timezone (America/Los_Angeles, Europe/London, Asia/Tokyo)
3. System automatically handles DST transitions
4. Recovery policy ensures no missed backups after system downtime
5. Execution history provides audit trail for compliance

Scenario 3: Critical Health Check Monitoring

User: SRE team running critical health checks Context: Health checks must run every 5 minutes without fail Flow:

1. Schedule job with */5 * * * * (every 5 minutes)
2. Set timeout of 60 seconds
3. Configure max_retries=3 with 30-second delay
4. Set concurrency_policy to forbid to prevent overlapping checks
5. Enable recovery mode to catch up after deployment windows

Terminology

Functional Requirements

FR-001: Cron Expression Parsing

• Support standard 5-field cron expressions (minute, hour, day, month, weekday)
• Support extended 6-field format with seconds
• Support special characters: *, ,, -, /, L, W, #
• Support named months and weekdays (JAN-DEC, SUN-SAT)
• Validate expression syntax before scheduling

FR-002: Timezone Support

• Accept IANA timezone database names (e.g., "America/New_York")
• Support UTC offset notation (e.g., "+05:30")
• Handle DST transitions automatically
• Store all timestamps in UTC internally
• Convert to job timezone for execution scheduling

FR-003: Job Persistence

• Store jobs in SQLite database
• Persist job definitions across restarts
• Maintain execution history
• Support job import/export

FR-004: Missed Job Recovery

• Detect missed executions on startup
• Apply configurable recovery policy
• Track recovery runs separately
• Limit recovery batch size to prevent overload

FR-005: Concurrent Execution Protection

• Acquire distributed lock before execution
• Support multiple concurrency policies
• Detect and handle stuck jobs
• Provide lock status in job metadata

FR-006: Job Execution

• Execute jobs asynchronously
• Support timeout enforcement
• Capture stdout/stderr output
• Handle execution errors gracefully

FR-007: Retry Mechanism

• Automatic retry on failure
• Configurable retry count and delay
• Exponential backoff option
• Track retry attempts in execution record

FR-008: Job Management

• List, update, pause, resume jobs
• Manual trigger support
• Bulk operations
• Job tagging and filtering

Error Conditions

API Contracts

Schedule Job

Request:

{
  "name": "daily-report",
  "cron_expression": "0 8 * * *",
  "timezone": "America/New_York",
  "command": "/scripts/generate_report.sh",
  "args": {
    "format": "pdf",
    "recipients": ["team@example.com"]
  },
  "miss_policy": "run_once",
  "concurrency_policy": "forbid",
  "max_retries": 3,
  "retry_delay_seconds": 60,
  "timeout_seconds": 300,
  "description": "Generate daily system report",
  "tags": ["reporting", "daily"],
  "enabled": true
}

Response (Success):

{
  "success": true,
  "job_id": "job_abc123",
  "next_run_at": "2026-04-09T08:00:00-04:00",
  "timezone": "America/New_York",
  "created_at": "2026-04-08T10:30:00Z"
}

Response (Error):

{
  "success": false,
  "error_code": "E001",
  "error_message": "Invalid cron expression: '0 8 * *'",
  "validation_errors": [
    "Expression must have 5 fields, got 4"
  ]
}

List Jobs

Request:

{
  "enabled_only": false,
  "tags": ["reporting"],
  "limit": 50
}

Response:

{
  "jobs": [
    {
      "id": "job_abc123",
      "name": "daily-report",
      "cron_expression": "0 8 * * *",
      "timezone": "America/New_York",
      "enabled": true,
      "status": "active",
      "last_run_at": "2026-04-08T08:00:00-04:00",
      "next_run_at": "2026-04-09T08:00:00-04:00",
      "tags": ["reporting", "daily"]
    }
  ],
  "total": 1,
  "limit": 50
}

Get Execution History

Request:

{
  "job_id": "job_abc123",
  "status": "failed",
  "limit": 20
}

Response:

{
  "executions": [
    {
      "id": "exec_xyz789",
      "job_id": "job_abc123",
      "status": "failed",
      "started_at": "2026-04-08T08:00:00-04:00",
      "completed_at": "2026-04-08T08:05:00-04:00",
      "error_message": "Command exited with code 1",
      "output": "Report generation failed: database connection error",
      "retry_count": 2,
      "is_recovery": false
    }
  ],
  "total": 1,
  "limit": 20
}

Non-Functional Requirements

Performance

• Job scheduling latency must be under 100ms for 90th percentile
• Concurrent job execution support for up to 100 simultaneous jobs
• Database query performance under 50ms for job retrieval operations
• Memory footprint under 256MB for idle scheduler with 1000 registered jobs

Reliability

• System must maintain 99.9% availability for job scheduling
• Jobs must execute within 1 second of scheduled time under normal load
• Recovery from database failure within 30 seconds
• Zero data loss for job definitions during system crash (persisted immediately)

Scalability

• Support for 10,000+ registered jobs per scheduler instance
• Horizontal scaling via job sharding across multiple nodes
• Efficient handling of burst scheduling (1000+ jobs starting at same time)

Compatibility

• Python 3.11+ runtime required
• SQLite 3.31+ for persistence
• Redis 6.0+ for distributed locking (optional)
• IANA timezone database (Olson DB) for timezone handling

Technical Design

Architecture Overview

The ScheduleCronTool follows a modular architecture with clear separation of concerns:

┌─────────────────────────────────────────────────────────┐
│                    ScheduleCronTool                      │
├─────────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │   Scheduler  │  │   Executor   │  │   Storage    │   │
│  │    Engine    │  │    Service   │  │    Layer     │   │
│  └──────────────┘  └──────────────┘  └──────────────┘   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │    Lock      │  │  Recovery    │  │    Config    │   │
│  │   Manager    │  │   Service    │  │   Manager    │   │
│  └──────────────┘  └──────────────┘  └──────────────┘   │
└─────────────────────────────────────────────────────────┘

Core Components

Scheduler Engine

• Uses croniter library for cron expression parsing
• Calculates next run times in job-specified timezone
• Maintains priority queue of upcoming job executions
• Triggers executor service at calculated times

Executor Service

• Asynchronous job execution using asyncio
• Spawns subprocess for command execution
• Enforces timeout controls
• Captures stdout/stderr output
• Implements retry logic with configurable delays

Storage Layer

• SQLite database for job persistence
• Job definitions stored in cron_jobs table
• Execution history stored in job_executions table
• Async operations via aiosqlite

Lock Manager

• Redis-based distributed locking (when available)
• Fallback to file-based locking for single-node deployments
• Automatic lock release on job completion
• Configurable lock timeout per job

Recovery Service

• Scans for missed jobs on startup
• Applies configured recovery policy
• Limits recovery batch size to prevent overload
• Marks recovered executions with is_recovery flag

Examples

Example 1: Daily Email Report Job

Schedule a job that sends a daily report at 9:00 AM in Europe/London timezone:

result = await tool.schedule_job(
    name="daily-email-report",
    cron_expression="0 9 * * *",
    timezone="Europe/London",
    command="/scripts/send_daily_report.py",
    args={
        "recipients": ["team@example.com"],
        "format": "html",
        "include_charts": True
    },
    miss_policy="run_once",
    concurrency_policy="forbid",
    timeout_seconds=600,
    description="Send daily sales report to team",
    tags=["reporting", "email", "daily"]
)

Example 2: Hourly Health Check with Recovery

Configure a critical health check that runs every hour with recovery enabled:

result = await tool.schedule_job(
    name="service-health-check",
    cron_expression="0 * * * *",
    timezone="UTC",
    command="/scripts/health_check.sh",
    args={"service": "api-gateway", "threshold": 5},
    miss_policy="run_all",
    concurrency_policy="forbid",
    max_retries=3,
    retry_delay_seconds=30,
    timeout_seconds=30,
    enabled=True,
    tags=["monitoring", "critical"]
)

Example 3: Weekly Database Backup

Schedule a weekly backup job that runs on Sunday at 2:00 AM:

result = await tool.schedule_job(
    name="weekly-database-backup",
    cron_expression="0 2 * * 0",
    timezone="America/Los_Angeles",
    command="/usr/local/bin/backup_db.sh",
    args={
        "database": "production",
        "compression": "gzip",
        "retention_days": 30
    },
    miss_policy="skip",
    concurrency_policy="forbid",
    max_retries=2,
    timeout_seconds=3600,
    description="Weekly production database backup",
    tags=["backup", "database", "weekly", "critical"]
)

Example 4: Monitoring and Alerts

Get execution history and set up alerting for failed jobs:

# Get recent failed executions
executions = await tool.get_execution_history(
    job_id="job_abc123",
    status="failed",
    limit=10
)

# Check next scheduled runs
next_runs = await tool.get_next_runs(
    job_id="job_abc123",
    count=5
)
for run_time in next_runs:
    print(f"Next run: {run_time}")

Integration Points

External System Integrations

Redis (Distributed Locking)

• Purpose: Prevent concurrent job execution across multiple nodes
• Connection: Via redis:// URL configuration
• Fallback: File-based locking when Redis unavailable

SQLite Database

• Purpose: Persistent storage for job definitions and execution history
• Location: Configurable path (default: "./data/scheduler.db")
• Schema: Automatic migration on startup

Command Execution

• Purpose: Run user-defined job commands
• Interface: Subprocess spawning with timeout control
• Security: Optional command allowlisting

API Integration Patterns

Direct Tool Usage

tool = ScheduleCronTool(config)
await tool.schedule_job(name="my-job", cron_expression="0 * * * *", command="echo hello")

REST API Exposure

• ScheduleCronTool can be wrapped in FastAPI/Flask for HTTP access
• All methods support async execution
• Returns structured JSON responses

Event-Driven Integration

• Emit events on job completion/failure for external processing
• Support webhook notifications for critical job statuses
• Integrate with monitoring systems (Prometheus, DataDog)

Data Models

CronJob Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "CronJob",
  "type": "object",
  "required": ["id", "name", "cron_expression", "timezone", "command"],
  "properties": {
    "id": {
      "type": "string",
      "pattern": "^job_[a-zA-Z0-9_]{8,}$"
    },
    "name": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128,
      "pattern": "^[a-zA-Z][a-zA-Z0-9_-]*$"
    },
    "cron_expression": {
      "type": "string",
      "description": "Cron expression: minute hour day month weekday. Supports numbers, *, ranges (1-5), steps (*/5), lists (1,3,5), and extensions (L, W, # for advanced scheduling)"
    },
    "timezone": {
      "type": "string",
      "pattern": "^[A-Za-z]+/[A-Za-z_]+|[+-]\\d{2}:\\d{2}$"
    },
    "command": {
      "type": "string",
      "minLength": 1,
      "maxLength": 4096
    },
    "args": {
      "type": "object"
    },
    "enabled": {
      "type": "boolean",
      "default": true
    },
    "miss_policy": {
      "type": "string",
      "enum": ["skip", "run_once", "run_all", "notify_only"],
      "default": "run_once"
    },
    "concurrency_policy": {
      "type": "string",
      "enum": ["allow", "skip", "queue", "forbid"],
      "default": "allow"
    },
    "max_retries": {
      "type": "integer",
      "minimum": 0,
      "maximum": 10,
      "default": 3
    },
    "retry_delay_seconds": {
      "type": "integer",
      "minimum": 0,
      "maximum": 3600,
      "default": 60
    },
    "timeout_seconds": {
      "type": "integer",
      "minimum": 1,
      "maximum": 86400,
      "default": 300
    },
    "description": {
      "type": "string",
      "maxLength": 1024
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string",
        "pattern": "^[a-zA-Z][a-zA-Z0-9_-]*$"
      },
      "maxItems": 20
    },
    "created_at": {
      "type": "string",
      "format": "date-time"
    },
    "updated_at": {
      "type": "string",
      "format": "date-time"
    },
    "last_run_at": {
      "type": ["string", "null"],
      "format": "date-time"
    },
    "next_run_at": {
      "type": ["string", "null"],
      "format": "date-time"
    }
  }
}

JobExecution Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "JobExecution",
  "type": "object",
  "required": ["id", "job_id", "status", "started_at"],
  "properties": {
    "id": {
      "type": "string",
      "pattern": "^exec_[a-zA-Z0-9_]{8,}$"
    },
    "job_id": {
      "type": "string",
      "pattern": "^job_[a-zA-Z0-9_]{8,}$"
    },
    "status": {
      "type": "string",
      "enum": ["pending", "running", "completed", "failed", "cancelled", "recovered"]
    },
    "started_at": {
      "type": "string",
      "format": "date-time"
    },
    "completed_at": {
      "type": ["string", "null"],
      "format": "date-time"
    },
    "error_message": {
      "type": ["string", "null"]
    },
    "output": {
      "type": ["string", "null"]
    },
    "retry_count": {
      "type": "integer",
      "minimum": 0,
      "maximum": 10
    },
    "is_recovery": {
      "type": "boolean",
      "default": false
    },
    "execution_node": {
      "type": ["string", "null"]
    }
  }
}

ScheduleJobRequest Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ScheduleJobRequest",
  "type": "object",
  "required": ["name", "cron_expression", "command"],
  "properties": {
    "name": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128
    },
    "cron_expression": {
      "type": "string"
    },
    "timezone": {
      "type": "string",
      "default": "UTC"
    },
    "command": {
      "type": "string"
    },
    "args": {
      "type": "object"
    },
    "miss_policy": {
      "type": "string",
      "enum": ["skip", "run_once", "run_all", "notify_only"],
      "default": "run_once"
    },
    "concurrency_policy": {
      "type": "string",
      "enum": ["allow", "skip", "queue", "forbid"],
      "default": "allow"
    },
    "max_retries": {
      "type": "integer",
      "minimum": 0,
      "maximum": 10,
      "default": 3
    },
    "retry_delay_seconds": {
      "type": "integer",
      "minimum": 0,
      "maximum": 3600,
      "default": 60
    },
    "timeout_seconds": {
      "type": "integer",
      "minimum": 1,
      "maximum": 86400,
      "default": 300
    },
    "description": {
      "type": "string",
      "maxLength": 1024
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "maxItems": 20
    },
    "enabled": {
      "type": "boolean",
      "default": true
    }
  }
}

Security Requirements

SR-001: Timezone Handling

• Validate all timezone strings against IANA database
• Reject invalid timezone identifiers
• Handle UTC offset notation securely
• Store timestamps in UTC internally
• Display times in job-specified timezone only

SR-002: Missed Job Recovery

• Limit recovery batch to 100 jobs max
• Require explicit enablement of recovery
• Log all recovery operations
• Track recovered executions separately
• Send notification when recovery threshold exceeded

SR-003: Concurrent Execution Protection

• Use distributed locking mechanism (Redis/zookeeper)
• Lock timeout configurable per job
• Automatic lock release on job completion
• Manual lock override for administrators
• Deadlock detection and prevention

SR-004: Command Execution Security

• Validate command paths before execution
• Sanitize command arguments
• Run commands with minimal required privileges
• Log all command executions
• Support command allowlisting mode

SR-005: Audit Logging

• Log all job state changes
• Record execution start/end times
• Capture command output for debugging
• Retain logs per retention policy
• Exportable audit trail

SR-006: Access Control

• Role-based job management permissions
• Job-level access control
• API key authentication for scheduling
• Rate limiting on job scheduling

Testing Requirements

Unit Tests

• Cron expression parsing validation
• Timezone conversion accuracy
• Job state machine transitions
• Retry logic verification
• Lock acquisition/release

Integration Tests

• Database persistence operations
• Scheduler execution timing
• Recovery mechanism after restart
• Concurrent job execution handling
• Timeout and cancellation

E2E Tests

• Complete job lifecycle (create, run, complete)
• Timezone-aware scheduling across DST
• Recovery from simulated failures
• High-volume job scheduling
• Performance under load

Test Coverage Requirements

• Minimum 80% code coverage
• All error conditions must have test cases
• Edge cases for cron expressions
• Timezone boundary conditions

Dependencies

Runtime Dependencies

Development Dependencies

Configuration

scheduler:
  database:
    path: "./data/scheduler.db"
    pool_size: 5
  
  execution:
    default_timeout: 300
    max_concurrent: 10
    retry_delay: 60
  
  recovery:
    enabled: true
    max_batch: 100
    check_interval: 300
  
  locking:
    provider: "redis"
    redis_url: "redis://10.53.2.11:6379"  # example: test fleet; in practice from the active ORCHESTRATA_ENV_PROFILE (RUL-003, no hardcoded default)
    lock_timeout: 600
    lock_retry_delay: 1
  
  logging:
    level: "INFO"
    format: "json"
    output: "stdout"
    retention_days: 90
  
  security:
    command_whitelist_enabled: false
    allowed_commands:
      - "/scripts/*.sh"
      - "/usr/bin/python3"
    run_as_user: "scheduler"

Governance

% Job scheduling governance rules
may_schedule_job(Agent, Job) :-
    not restricted_agent(Agent),
    has_permission(Agent, schedule_jobs).

may_execute_job(Agent, Job) :-
    may_schedule_job(Agent, Job),
    job_enabled(Job),
    not job_paused(Job).

may_modify_job(Agent, Job) :-
    owner(Agent, Job),
    has_permission(Agent, modify_jobs).

may_delete_job(Agent, Job) :-
    owner(Agent, Job),
    has_permission(Agent, delete_jobs).

% Restricted execution contexts
restricted_agent(ai_agent).
restricted_agent(automated_script) :- not whitelisted_script(script_id).

% Job ownership
owner(Agent, Job) :- created_by(Job, Agent).
owner(admin, Job) :- system_job(Job).

Success Criteria

AC-001: Job Scheduling

• [ ] Can schedule job with valid cron expression
• [ ] Cron expression validation rejects invalid syntax
• [ ] Next run time calculated correctly in specified timezone
• [ ] Job persists across application restarts

AC-002: Timezone Handling

• [ ] Jobs execute at correct time in any timezone
• [ ] DST transitions handled correctly
• [ ] Invalid timezone rejected with clear error
• [ ] All timestamps stored in UTC internally

AC-003: Recovery

• [ ] Missed jobs detected on startup
• [ ] Recovery policy applied correctly
• [ ] Recovered executions marked appropriately
• [ ] Recovery batch size enforced

AC-004: Concurrency

• [ ] Concurrent execution prevented when policy is "forbid"
• [ ] Lock acquisition logged
• [ ] Lock release on job completion or failure
• [ ] Deadlock detection functional

AC-005: Execution

• [ ] Jobs execute at scheduled times
• [ ] Timeout enforced
• [ ] Retry mechanism works
• [ ] Output captured and stored

AC-006: Management

• [ ] List jobs with filters
• [ ] Pause/resume jobs
• [ ] Update job configuration
• [ ] Delete jobs cleanly

Observability

Logging Requirements

1. Component Logging
   • Log all major operations with appropriate log levels
   • Include correlation IDs for request tracing
   • Log performance metrics for critical operations
   • Log errors with full context for debugging

Metrics Requirements

1. Performance Metrics
   • Operation duration histograms
   • Request/error rate counters
   • Resource utilization gauges
   • Business metrics relevant to domain

Tracing Requirements

1. Distributed Tracing
   • Trace propagation across component boundaries
   • Span attributes for domain-specific context
   • Trace sampling based on operational needs

Alerting Requirements

1. Critical Alerts
   • Error rate spikes
   • Performance degradation
   • Resource exhaustion
   • Business metric anomalies

Dashboard Requirements

1. Operational Dashboard
   • Real-time system health
   • Performance trends
   • Error analysis
   • Resource monitoring

Revision History

End of SPEC-068

Related

• RTM page for 068-schedule-cron-tool
• All specs
• RTM overview
• Gates