Orchestrata Ultimate / Specs / 060-enhanced-pam-governance

060-enhanced-pam-governance: Enhanced PAM++ Governance - Runtime Integration

This specification extends SPEC-007 (Prolog Governance/PAM++) with runtime integration, enforcement mechanisms, and operational features required for production deployment.

At a glance

Reading this spec

This spec extends SPEC-007 (Prolog governance, PAM++) with what production needs on top of the policy language: runtime enforcement, policy updates without a restart, monitoring, and tight integration with the orchestrata-ultimate kernel. The approach is middleware-based enforcement at the API and tool level, a Redis-backed decision cache for low latency, WebSocket push for policy updates, and Prometheus metrics for governance health.

Its four user stories are runtime policy enforcement, hot policy reload, governance monitoring and graceful degradation. The functional requirements (FR-001 onward) cover middleware enforcement, decision caching, hot reload and metrics collection, and data models describe the PolicyDecision and the GovernanceConfig. All 75 requirement IDs are numbered REQ-060-NNN.

Key constraints: a hot reload completes without dropping in-flight requests and in under 10 seconds; governance decisions are cached in Redis; decision counts are tracked by outcome; and the system handles 10,000 or more concurrent governance requests. When a new policy has a syntax error, the system rejects it and keeps the old policy active, and an invalid policy scope is rejected at validation with error details. The PAM++ interface handles timeouts and errors and supports both async and sync use. The policy store is a PostgreSQL database with JSONB policy storage, CRUD operations and version history, and policy changes go through an approval workflow. Components log major operations with correlation IDs for request tracing.

No code module cites this spec; the governance code that exists in the repository lives under the governance reference pages but is not annotated with this spec's IDs. The requirements traceability matrix lists every ID. The spec's own status line reads Complete and it tracks 2 files. No src/ symbol cites any of the 75 requirements, so measured trace coverage is 0.0%, and in tests/, 0 requirements are cited by a test-side @traces. All 75 requirements have no citing symbol; untraced means no @traces names the ID, and the behaviour may still exist in code that was never annotated.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

No src/ symbol cites a requirement of this spec.

Other tracked files

• specs/060-enhanced-pam-governance/compliance-audit.md

Full spec text

Verbatim from specs/060-enhanced-pam-governance/spec.md; headings are pushed down two levels.

SPEC-060: Enhanced PAM++ Governance - Runtime Integration

Metadata

• Spec ID: 060-enhanced-pam-governance
• Title: Enhanced PAM++ Governance - Runtime Integration
• Author: Mohamed
• Status: Complete
• Created: 2026-04-08
• Last Updated: 2026-04-08
• Priority: P0
• Version: 1.0.0
• Extends: SPEC-007-prolog-governance

Overview

This specification extends SPEC-007 (Prolog Governance/PAM++) with runtime integration, enforcement mechanisms, and operational features required for production deployment.

What

Enhanced PAM++ governance with runtime enforcement, hot-reload, monitoring, and integration with the orchestrata-ultimate kernel.

Why

• Enable real-time policy enforcement in production
• Provide runtime policy updates without restart
• Monitor governance health and performance
• Integrate seamlessly with all system components

How

• Middleware-based enforcement at API and tool level
• Redis-backed decision cache for low latency
• WebSocket-based policy push updates
• Prometheus metrics for governance monitoring

User Scenarios

US-001: Runtime Policy Enforcement

As a system administrator, I want policies enforced at runtime so that governance violations are blocked immediately.

Acceptance Criteria:

• All API requests pass through governance middleware
• Governance decisions cached for <5ms latency
• Violations return 451 status code with policy details

US-002: Hot Policy Reload

As a security officer, I want to update policies without restart so I can respond to security incidents quickly.

Acceptance Criteria:

• Policy changes take effect within 10 seconds
• No dropped requests during reload
• Previous policies preserved for rollback

REQ-060-001: The system SHALL complete a hot policy reload without dropping in-flight requests.

US-003: Governance Monitoring

As an ops engineer, I want governance metrics so I can monitor system health and detect anomalies.

Acceptance Criteria:

• Decision counts by outcome (allow/deny/escalate)
• Latency percentiles (p50, p95, p99)
• Violation rate alerts configured

US-004: Graceful Degradation

As a system architect, I want governance to fail gracefully so system continues operating if PAM++ is unavailable.

Acceptance Criteria:

• Default-deny if PAM++ unavailable
• Cache serves decisions during outage
• Alerts triggered on governance failure

REQ-060-002: The system SHALL default to deny when PAM++ is unavailable.

Terminology

PAM++

Policy and Administration Module++ - Prolog-based governance engine that evaluates policies using logical predicates.

Governance Decision

The result of policy evaluation including allow/deny/escalate outcomes with reasoning.

Policy Cache

Redis-backed cache storing recent governance decisions for low-latency access.

Hot Reload

The ability to update governance policies without restarting the system.

Degradation Mode

Fallback behavior when PAM++ is unavailable (allow_all, deny_all, cache_only).

Data Models

PolicyDecision

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["decision_id", "subject", "action", "resource", "result", "timestamp"],
  "properties": {
    "decision_id": {
      "type": "string",
      "pattern": "^dec_[a-zA-Z0-9]+$"
    },
    "subject": {
      "type": "string"
    },
    "action": {
      "type": "string"
    },
    "resource": {
      "type": "string"
    },
    "result": {
      "type": "string",
      "enum": ["allow", "deny", "escalate", "defer"]
    },
    "reason": {
      "type": "string"
    },
    "policy_id": {
      "type": "string"
    },
    "latency_ms": {
      "type": "number"
    },
    "timestamp": {
      "type": "string",
      "format": "date-time"
    }
  }
}

REQ-060-003: A governance decision record SHALL carry decision_id (matching ^dec_[a-zA-Z0-9]+$), subject, action, resource, result and timestamp, and its result SHALL be one of allow, deny, escalate or defer.

GovernanceConfig

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["enabled", "default_policy", "cache_ttl"],
  "properties": {
    "enabled": {
      "type": "boolean"
    },
    "default_policy": {
      "type": "string",
      "enum": ["allow", "deny"]
    },
    "cache_ttl": {
      "type": "integer",
      "minimum": 1
    },
    "cache_max_size": {
      "type": "integer"
    },
    "timeout_ms": {
      "type": "integer"
    },
    "retry_count": {
      "type": "integer"
    },
    "degradation_mode": {
      "type": "string",
      "enum": ["allow_all", "deny_all", "cache_only"]
    }
  }
}

REQ-060-004: The governance configuration SHALL carry enabled, default_policy (allow or deny) and cache_ttl (minimum 1), and SHALL accept cache_max_size, timeout_ms, retry_count and a degradation_mode of allow_all, deny_all or cache_only.

Functional Requirements

FR-001: Middleware Enforcement

• System SHALL enforce governance at API middleware level
• System SHALL block requests that fail governance checks
• System SHALL return 451 status for governance blocks

REQ-060-005: The system SHALL enforce governance at the API middleware level.

REQ-060-006: The system SHALL block requests that fail governance checks.

REQ-060-007: The system SHALL return 451 status for governance blocks.

FR-002: Decision Caching

• System SHALL cache governance decisions in Redis
• System SHALL respect cache TTL
• System SHALL invalidate cache on policy changes

REQ-060-008: The system SHALL cache governance decisions in Redis.

REQ-060-009: The system SHALL respect cache TTL.

REQ-060-010: The system SHALL invalidate the cache on policy changes.

FR-003: Hot Reload

• System SHALL reload policies without restart
• System SHALL push policy updates via WebSocket
• System SHALL support policy rollback

REQ-060-011: The system SHALL reload policies without restart.

REQ-060-012: The system SHALL push policy updates via WebSocket.

REQ-060-013: The system SHALL support policy rollback.

FR-004: Metrics Collection

• System SHALL expose governance metrics via Prometheus
• System SHALL track decision counts by outcome
• System SHALL track latency percentiles

REQ-060-014: The system SHALL expose governance metrics via Prometheus.

REQ-060-015: The system SHALL track decision counts by outcome.

REQ-060-016: The system SHALL track latency percentiles.

FR-005: Graceful Degradation

• System SHALL continue if PAM++ unavailable
• System SHALL use cache during outages
• System SHALL alert on governance failures

REQ-060-017: The system SHALL continue operating if PAM++ is unavailable.

REQ-060-018: The system SHALL use the cache during outages.

REQ-060-019: The system SHALL alert on governance failures.

Non-Functional Requirements

Performance

• Decision latency: <5ms (cached), <50ms (uncached)
• Cache hit rate: >95%
• Policy reload: <10 seconds
• Throughput: 1000 decisions/second

REQ-060-101: Decision latency SHALL be under 5 ms for cached decisions and under 50 ms for uncached decisions.

REQ-060-102: The decision cache hit rate SHALL exceed 95%.

REQ-060-103: A policy reload SHALL complete in under 10 seconds.

REQ-060-104: The system SHALL sustain a throughput of 1000 decisions/second.

Reliability

• System SHALL maintain operation during PAM++ outages via degradation modes
• System SHALL preserve policy versions for rollback capability
• System SHALL log all governance decisions for audit

REQ-060-105: The system SHALL maintain operation during PAM++ outages via degradation modes.

REQ-060-106: The system SHALL preserve policy versions for rollback capability.

REQ-060-107: The system SHALL log all governance decisions for audit.

Scalability

• System SHALL support horizontal scaling of governance middleware
• System SHALL distribute cache across Redis cluster
• System SHALL handle 10,000+ concurrent governance requests

REQ-060-108: The system SHALL support horizontal scaling of governance middleware.

REQ-060-109: The system SHALL distribute the cache across a Redis cluster.

REQ-060-110: The system SHALL handle 10,000+ concurrent governance requests.

Availability

• System SHALL achieve 99.9% uptime for governance services
• System SHALL provide sub-second failover to degradation mode
• System SHALL support geographic distribution of governance nodes

REQ-060-111: The system SHALL achieve 99.9% uptime for governance services.

REQ-060-112: The system SHALL provide sub-second failover to degradation mode.

REQ-060-113: The system SHALL support geographic distribution of governance nodes.

Error Conditions

The handling column above is normative:

REQ-060-020: On a PAM++ timeout (ERR-001) the system SHALL serve the cached decision, log a warning and increment the timeout metric.

REQ-060-021: When PAM++ is unavailable (ERR-002) the system SHALL activate degradation mode and alert on-call.

REQ-060-022: On a decision cache miss (ERR-003) the system SHALL query PAM++, cache the result and continue the request.

REQ-060-023: On a policy syntax error (ERR-004) the system SHALL reject the new policy and keep the old policy active.

REQ-060-024: The system SHALL detect a circular dependency in a policy (ERR-005) at load time, reject the policy and notify the administrator.

REQ-060-025: On a Redis connection failure (ERR-006) the system SHALL fall back to an in-memory cache and alert that it is degraded.

REQ-060-026: When the decision cache is full (ERR-007) the system SHALL evict the oldest entries using LRU and log a warning.

REQ-060-027: On a WebSocket disconnect (ERR-008) the system SHALL auto-reconnect with exponential backoff.

REQ-060-028: On a policy reload failure (ERR-009) the system SHALL roll back to the previous version and alert the administrator.

REQ-060-029: On a metric export failure (ERR-010) the system SHALL log locally, continue operation and alert monitoring.

REQ-060-030: The system SHALL reject an invalid policy scope (ERR-011) at validation and provide error details.

REQ-060-031: On unauthorized policy access (ERR-012) the system SHALL deny access and log a security event.

API Contracts

Governance Middleware

class GovernanceMiddleware:
    async def __call__(self, request, handler):
        decision = await self.governance.check(
            subject=request.user,
            action=request.action,
            resource=request.path
        )
        
        if decision.result == "deny":
            raise GovernanceError(decision.reason, decision.policy_id)
        
        return await handler(request)

REQ-060-032: The governance middleware SHALL evaluate every request against subject, action and resource, SHALL raise a GovernanceError carrying the decision reason and policy id when the decision result is deny, and SHALL otherwise pass the request to the next handler.

Policy Management API

GET    /api/v1/governance/policies         # List policies
POST   /api/v1/governance/policies         # Add policy
PUT    /api/v1/governance/policies/{id}    # Update policy
DELETE /api/v1/governance/policies/{id}    # Delete policy
POST   /api/v1/governance/reload           # Hot reload
GET    /api/v1/governance/decisions        # Query decisions
GET    /api/v1/governance/metrics         # Prometheus metrics

REQ-060-033: The system SHALL expose the policy management API: GET/POST /api/v1/governance/policies, PUT/DELETE /api/v1/governance/policies/{id}, POST /api/v1/governance/reload, GET /api/v1/governance/decisions and GET /api/v1/governance/metrics.

Technical Design

Architecture Overview

The governance system uses middleware-based enforcement with Redis caching and WebSocket-based policy updates:

┌──────────────┐    ┌────────────────┐    ┌─────────┐
│   API Layer  │───▶│  Governance    │───▶│  PAM++  │
│  Middleware  │    │   Middleware   │    │ Engine  │
└──────────────┘    └────────────────┘    └─────────┩
       │                    │
       ▼                    ▼
┌──────────────┐    ┌────────────────┐
│    Redis     │    │   Prometheus   │
│ Decision     │    │    Metrics     │
│    Cache     │    │                │
└──────────────┘    └────────────────┘

Component Design

GovernanceMiddleware

• Implements FastAPI middleware interface
• Performs policy lookups before request processing
• Catches and handles GovernanceError exceptions
• Attaches decision metadata to response headers

REQ-060-034: The governance middleware SHALL implement the FastAPI middleware interface and SHALL perform its policy lookup before request processing.

REQ-060-035: The governance middleware SHALL catch and handle GovernanceError exceptions and SHALL attach the decision metadata to the response headers.

PAM++ Interface

• Uses pyswip for Prolog integration
• Converts policy queries to Prolog predicates
• Handles timeout and error conditions
• Supports async and sync operations

REQ-060-036: The PAM++ interface SHALL convert policy queries into Prolog predicates.

REQ-060-037: The PAM++ interface SHALL handle timeout and error conditions and SHALL support both async and sync operations.

Decision Cache

• Redis-based with configurable TTL
• Keys: governance:decision:<hash(subject,action,resource)>
• Values: JSON-serialized PolicyDecision
• Invalidation on policy reload

REQ-060-038: The decision cache SHALL be Redis-backed with a configurable TTL, SHALL key entries as governance:decision:<hash(subject,action,resource)> and SHALL store each value as a JSON-serialized decision record.

Policy Manager

• Manages policy lifecycle (create, update, delete)
• Validates policy syntax via Prolog compilation
• Maintains policy version history
• Triggers cache invalidation on updates

REQ-060-039: The policy manager SHALL manage the policy lifecycle (create, update, delete) and SHALL validate policy syntax by Prolog compilation before a policy is accepted.

REQ-060-040: The policy manager SHALL maintain policy version history and SHALL trigger cache invalidation on policy updates.

Data Flow

1. Request Flow:

   Client Request → FastAPI Router → Governance Middleware
                                        ↓
                              Check Decision Cache
                                        ↓
                              PAM++ Evaluation (if miss)
                                        ↓
                              Cache Decision → Continue Handler
   

2. Policy Update Flow:

   Admin Request → Policy Manager → Validate Syntax
                                       ↓
                              Load into PAM++ (atomic swap)
                                       ↓
                              Invalidate Cache → WebSocket Push
   

3. Metrics Flow:

   Decision Made → Update Prometheus Metrics
                                ↓
                     Expose /metrics endpoint
                                ↓
                     Grafana Dashboard
   

WebSocket Protocol

{
  "type": "policy_update",
  "policy_id": "pol_123",
  "version": "2.0.0",
  "timestamp": "2026-04-09T12:00:00Z"
}

REQ-060-041: A policy-update notification pushed over WebSocket SHALL carry type of policy_update, the policy_id, the policy version and a timestamp.

Policy Format

Policies are Prolog facts and rules:

% Allow admin users full access
allow(Subject, Action, Resource) :-
    has_role(Subject, admin),
    owns(Subject, Resource).

% Deny delete on sensitive resources
deny(Subject, delete, Resource) :-
    has_tag(Resource, sensitive),
    not(has_role(Subject, security_officer)).

REQ-060-042: Governance policies SHALL be expressed as Prolog facts and rules.

Security Requirements

• All governance APIs require admin role
• Policy changes require approval workflow
• Audit log all policy modifications
• Encrypt sensitive policy data

REQ-060-114: All governance APIs SHALL require the admin role.

REQ-060-115: Policy changes SHALL require an approval workflow.

REQ-060-116: All policy modifications SHALL be written to the audit log.

REQ-060-117: Sensitive policy data SHALL be encrypted.

Visual Architecture

Class Diagram

Diagram source shown as text: Mermaid could not parse it (Error: Parse error on line 11:).

classDiagram-direction LR
    class GovernanceEngine {
        +__init__()
        #_load_default_policies(): dict[str, Any]
        +async check(action: str, context: dict[str, Any]): bool
        #async _check_edit_file(context: dict[str, Any]): bool
        #async _check_create_project(context: dict[str, Any]): bool
        #async _check_execute_command(context: dict[str, Any]): bool
    }

Unit Tests

class TestGovernanceMiddleware:
    async def test_allows_valid_request(self):
        middleware = GovernanceMiddleware()
        result = await middleware.check(user, "read", "project:1")
        assert result.result == "allow"
    
    async def test_blocks_unauthorized(self):
        middleware = GovernanceMiddleware()
        result = await middleware.check(user, "delete", "project:1")
        assert result.result == "deny"
    
    async def test_uses_cache(self):
        middleware = GovernanceMiddleware()
        await middleware.check(user, "read", "project:1")
        # Second call should use cache
        result = await middleware.check(user, "read", "project:1")
        assert result.cached == True

Integration Tests

• PAM++ connectivity
• Redis cache operations
• WebSocket policy push
• Fallback behavior

REQ-060-118: Integration tests SHALL cover PAM++ connectivity, Redis cache operations, WebSocket policy push and fallback behavior.

Test Coverage

• Unit: >90%
• Integration: 100%
• All predicates tested

REQ-060-119: Unit test coverage SHALL exceed 90%, integration coverage SHALL be 100%, and every governance predicate SHALL be tested.

Integration Points

Orchestrata Kernel

• Integration Type: Direct import
• Interface: Runtime hooks at request processing
• Data Exchange: Decision context, metrics, cache invalidation
• Error Handling: Propagate governance errors to kernel logger

REQ-060-043: The governance system SHALL integrate with the orchestrata kernel by direct import, exposing runtime hooks at request processing, exchanging decision context, metrics and cache invalidation, and SHALL propagate governance errors to the kernel logger.

Auth Service

• Integration Type: REST API
• Endpoint: /api/v1/auth/user-info
• Data Needed: User roles, permissions, context
• Frequency: Per request (cached for 60s)

REQ-060-044: The governance system SHALL obtain user roles, permissions and context from the auth service REST endpoint /api/v1/auth/user-info per request, cached for 60 seconds.

Policy Store

• Integration Type: Database
• Schema: PostgreSQL with JSONB policy storage
• Operations: CRUD on policies, version history
• Consistency: ACID transactions for policy updates

REQ-060-045: The policy store SHALL be a PostgreSQL database with JSONB policy storage supporting CRUD on policies and version history, and policy updates SHALL run in ACID transactions.

Monitoring Stack

• Integration Type: Prometheus metrics
• Metrics Endpoint: /metrics
• Alerts: Governance failures, degradation mode, policy errors
• Dashboards: Grafana dashboards for decision metrics

REQ-060-046: The governance system SHALL publish Prometheus metrics at /metrics, SHALL define alerts for governance failures, degradation mode and policy errors, and SHALL provide Grafana dashboards for decision metrics.

Notification Service

• Integration Type: Event stream
• Event Types: Policy updates, degradation mode, violations
• Delivery: WebSocket for real-time, webhook for critical alerts
• Channels: Slack, PagerDuty, email

REQ-060-047: The governance system SHALL emit policy-update, degradation-mode and violation events on an event stream, delivering real-time notifications over WebSocket and critical alerts by webhook to Slack, PagerDuty and email.

Cache Layer

• Integration Type: Redis
• Cluster: Redis Cluster with 3 masters, 3 replicas
• Persistence: AOF enabled with every-1-second sync
• Operations: GET/SET with TTL, pub/sub for invalidation

REQ-060-048: The cache layer SHALL run as a Redis Cluster of 3 masters and 3 replicas with AOF persistence at one-second sync, using GET/SET with TTL and pub/sub for invalidation.

Prolog Engine

• Integration Type: pyswip
• Runtime: SWI-Prolog native
• Performance: 1000 decisions/second
• Memory: Dedicated 2GB heap per instance

REQ-060-049: The policy engine SHALL run on the native SWI-Prolog runtime at 1000 decisions/second with a dedicated 2 GB heap per instance.

Governance

Policy Management

• Version control for policies
• Rollback capability
• Change approval workflow
• Audit trail for changes

Constitutional Enforcement

• All 11 articles enforced
• Violation detection
• Escalation paths
• Safe mode triggers

REQ-060-050: The governance system SHALL enforce all 11 constitutional articles.

REQ-060-051: The governance system SHALL provide violation detection, escalation paths and safe-mode triggers.

Performance Targets

• Decision latency: <5ms (cached), <50ms (uncached)
• Cache hit rate: >95%
• Policy reload: <10 seconds
• Throughput: 1000 decisions/second

Examples

Example 1: Request with Allow Decision

Request:

GET /api/v1/projects/123/files
Headers:
  Authorization: Bearer user_456_token

Decision Flow:

1. Middleware extracts user user_456 from token
2. Generates cache key: governance:decision:abc123...
3. Cache hit! Returns cached allow decision
4. Request proceeds to handler
5. Handler returns 200 OK with file data

Decision Record:

{
  "decision_id": "dec_x1y2z3",
  "subject": "user_456",
  "action": "read",
  "resource": "project:123:file",
  "result": "allow",
  "reason": "User has read permission via role 'developer'",
  "policy_id": "pol_basic_access",
  "latency_ms": 3,
  "cached": true,
  "timestamp": "2026-04-09T12:00:00Z"
}

Example 2: Request with Deny Decision

Request:

DELETE /api/v1/projects/123
Headers:
  Authorization: Bearer user_789_token

Decision Flow:

1. Middleware extracts user user_789 from token
2. Cache miss, queries PAM++
3. PAM++ evaluates: user_789 has role 'viewer', not 'admin'
4. Policy denies delete on projects for non-admins
5. Returns 451 Governance Required with error details

Response:

451 Governance Required
Content-Type: application/json

{
  "error": "GOVERNANCE_DENIED",
  "message": "Access denied by governance policy",
  "decision_id": "dec_abc987",
  "policy_id": "pol_project_access",
  "reason": "User lacks required role 'admin' for delete action on project resources",
  "subject": "user_789",
  "action": "delete",
  "resource": "project:123"
}

Example 3: Policy Hot Reload

Step 1: Submit New Policy

curl -X POST /api/v1/governance/policies \
  -H "Authorization: Bearer admin_token" \
  -d '{
    "content": "allow(user, Action, Resource) :- has_role(user, 'auditor'), Action == 'read'.",
    "description": "Allow auditors to read all resources"
  }'

Step 2: System Response

{
  "policy_id": "pol_auditor_read",
  "version": "1.0.0",
  "status": "loaded",
  "reload_time_ms": 450
}

Step 3: WebSocket Notification

{
  "type": "policy_update",
  "policy_id": "pol_auditor_read",
  "version": "1.0.0",
  "invalidation_count": 0,
  "timestamp": "2026-04-09T12:00:05Z"
}

Step 4: Cache Hit Verification

curl -X GET /api/v1/governance/decisions \
  -H "Authorization: Bearer auditor_token"
  
# Next request uses updated policy
# Decision shows cached: false on first hit

Example 4: Degradation Mode Activation

Scenario: PAM++ service becomes unavailable

System Behavior:

1. Health check fails for PAM++
2. Switches to deny_all degradation mode
3. All new requests default to deny
4. Existing cache entries served for 5 minutes
5. Alert sent to on-call team

Metrics Observed:

governance_pampp_status 0
governance_decisions_total{mode="degradation"} 123
governance_decisions_total{result="deny"} 456

Admin Dashboard:

• Status: DEGRADED
• Degradation Mode: deny_all
• Cache Entries: 15,432 (expiring in 4m 23s)
• Alerts: 1 critical (PAM++ Down)

Example 5: Policy with Circular Dependency Detection

Invalid Policy:

allow(User, Action, Resource) :- can_access(User, Resource).
can_access(User, Resource) :- allow(User, Action, Resource).

Detection Response:

{
  "error": "CIRCULAR_DEPENDENCY",
  "message": "Policy contains circular dependency between 'allow' and 'can_access' predicates",
  "location": "pol_cyclic:3:1",
  "cycle": ["allow/3", "can_access/2", "allow/3"]
}

Example 6: Prometheus Metrics Query

Query Decision Count by Outcome:

sum(rate(governance_decisions_total[5m])) by (result)

Result:

{result="allow"} 245
{result="deny"} 23
{result="escalate"} 2

Query Latency Percentiles:

histogram_quantile(0.95, rate(governance_decision_latency_seconds_bucket[5m]))

Result:

0.043  # 95th percentile is 43ms

Query Cache Hit Rate:

governance_cache_hits / (governance_cache_hits + governance_cache_misses)

Result:

0.97  # 97% cache hit rate

Success Criteria

Functional Criteria

• [ ] All API requests pass through governance middleware
• [ ] Governance decisions cached for <5ms latency
• [ ] Policy changes take effect within 10 seconds
• [ ] Graceful degradation when PAM++ unavailable
• [ ] All 11 constitutional articles enforced

Operational Criteria

• [ ] Decision latency: <5ms cached, <50ms uncached
• [ ] Cache hit rate: >95%
• [ ] Policy reload: <10 seconds
• [ ] Throughput: 1000 decisions/second
• [ ] System uptime: 99.9%

Security Criteria

• [ ] All governance APIs require admin role
• [ ] Policy changes require approval workflow
• [ ] All policy modifications logged to audit
• [ ] Sensitive policy data encrypted

Monitoring Criteria

• [ ] Prometheus metrics exposed at /metrics
• [ ] Decision counts tracked by outcome
• [ ] Latency percentiles (p50, p95, p99) available
• [ ] Alert rules configured for governance failures

Observability

Logging Requirements

1. Component Logging
   • Log all major operations with appropriate log levels
   • Include correlation IDs for request tracing
   • Log performance metrics for critical operations
   • Log errors with full context for debugging

REQ-060-120: Governance components SHALL log all major operations at appropriate log levels, SHALL include correlation IDs for request tracing, SHALL log performance metrics for critical operations, and SHALL log errors with full context.

Metrics Requirements

1. Performance Metrics
   • Operation duration histograms
   • Request/error rate counters
   • Resource utilization gauges
   • Business metrics relevant to domain

REQ-060-121: The governance system SHALL emit operation duration histograms, request and error rate counters, resource utilization gauges, and the business metrics relevant to the governance domain.

Tracing Requirements

1. Distributed Tracing
   • Trace propagation across component boundaries
   • Span attributes for domain-specific context
   • Trace sampling based on operational needs

REQ-060-122: The governance system SHALL propagate traces across component boundaries, SHALL attach span attributes carrying domain-specific context, and SHALL sample traces according to operational needs.

Alerting Requirements

1. Critical Alerts
   • Error rate spikes
   • Performance degradation
   • Resource exhaustion
   • Business metric anomalies

REQ-060-123: The governance system SHALL raise critical alerts on error rate spikes, performance degradation, resource exhaustion and business metric anomalies.

Dashboard Requirements

1. Operational Dashboard
   • Real-time system health
   • Performance trends
   • Error analysis
   • Resource monitoring

REQ-060-124: The governance system SHALL provide an operational dashboard showing real-time system health, performance trends, error analysis and resource monitoring.

Revision History

Related

• RTM page for 060-enhanced-pam-governance
• All specs
• RTM overview
• Gates