Orchestrata Ultimate / Specs / 072-tool-search-tool

072-tool-search-tool: ToolSearchTool Specification

The ToolSearchTool provides semantic search capabilities across all available tools in the Orchestrata system. It enables users to find relevant tools based on natural language queries, keyword matching, and semantic similarity. The tool maintains a searchable index of tool metadata including names, descriptions, parameters, and usage patterns.

At a glance

Reading this spec

This spec defines ToolSearchTool, semantic search across every tool in the Orchestrata system. Users need to find the right tool in a large ecosystem without knowing its exact name, so the tool accepts a natural-language query, keyword matching or fuzzy input and returns ranked results. It keeps a searchable index of tool metadata: names, descriptions, parameters and usage patterns. The method combines embedding-based semantic similarity, keyword matching with TF-IDF weighting, fuzzy matching for typo tolerance and a cached in-memory index.

Five user stories set the scope: finding tools by capability (a query such as "send HTTP requests" returns relevant HTTP tools, with semantic similarity ahead of exact keyword matches and a relevance score on each result), discovering related tools, filtering by tool properties, fuzzy matching for typos, and search history with suggestions. The 40 requirement IDs (REQ-072-NNN) cover core search (a relevance score from 0 to 1, semantic matching with keyword fallback), index management (an in-memory index, updates when tools register or change, rebuild on startup, persistence to disk and incremental updates), caching with a configurable TTL, and performance (95th percentile under 100 ms, concurrent searches and an index of at least 10,000 tools). They also cover faceted search, filtering by category, tags, return type and author, pagination, and a set of named errors such as INVALID_QUERY, INDEX_NOT_READY, INDEX_CORRUPTED, and EMBEDDING_SERVICE_ERROR, which falls back to keyword search.

The implementation is in the search module, with one citation in the tool registry. The requirements traceability matrix lists each ID. The spec's own status line reads Production Ready and it tracks 4 files. Measured coverage is 40 of 40 requirements cited by at least one @traces in src/ (100.0%), spread over 2 modules; the module citing the most of them is orchestrata_ultimate/tools/search (40 requirements). In tests/, 0 requirements are cited by a test-side @traces, so test-side citations are thinner than code-side ones.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

Code that traces this spec

Checklists

checklists/requirements.md

The requirement list above comes from this file; 0 of 40 checkbox items are ticked.

Other tracked files

• specs/072-tool-search-tool/compliance-audit.md
• specs/072-tool-search-tool/requirements.md

Full spec text

Verbatim from specs/072-tool-search-tool/spec.md; headings are pushed down two levels.

SPEC-072: ToolSearchTool Specification

Metadata

• Spec ID: 072-tool-search-tool
• Title: ToolSearchTool - Semantic Search Across Tools
• Author: Mohamed
• Status: Production Ready
• Created: 2024-01-15
• Last Updated: 2024-06-20
• Priority: P1
• Version: 1.0.0
• Extends: None

Overview

The ToolSearchTool provides semantic search capabilities across all available tools in the Orchestrata system. It enables users to find relevant tools based on natural language queries, keyword matching, and semantic similarity. The tool maintains a searchable index of tool metadata including names, descriptions, parameters, and usage patterns.

Why: Users need to discover and locate appropriate tools within a large ecosystem without knowing exact tool names or commands.

How: The tool uses embedding-based semantic similarity, keyword matching with TF-IDF weighting, fuzzy matching for typo tolerance, and a cached in-memory index for fast retrieval.

User Scenarios

US-001: Finding Tools by Capability

• As a system user
• I want to search for tools using natural language queries like "send HTTP requests" or "make API calls"
• So that I can discover relevant tools without knowing their exact names

Acceptance Criteria:

• [ ] Query "send HTTP requests" returns HttpClient, FetchTool, ApiRequestTool
• [ ] Semantic similarity takes precedence over exact keyword matches
• [ ] Results include relevance scores

US-002: Discovering Related Tools

• As a data processing engineer
• I want to find all tools related to "JSON parsing"
• So that I can identify all available options for working with JSON data

Acceptance Criteria:

• [ ] Search returns JsonParser, JsonValidator, JsonFormatter
• [ ] Related tools with JSON functionality in descriptions are included
• [ ] Faceted results show category distribution

US-003: Filtering by Tool Properties

• As a developer
• I want to filter search results by category, return type, and required parameters
• So that I can narrow down to tools that match my specific requirements

Acceptance Criteria:

• [ ] Combined filters (category + return_type) work correctly
• [ ] Filtered results maintain relevance ranking
• [ ] Blank filter combinations return appropriate error message

US-004: Fuzzy Matching for Typos

• As a user with keyboard input errors
• I want to find tools even with typos in my query
• So that I can still discover tools despite spelling mistakes

Acceptance Criteria:

• [ ] Query "reacieve" returns ReceiveMessageTool
• [ ] Fuzzy threshold is configurable
• [ ] Results indicate fuzzy match was applied

US-005: Search History and Suggestions

• As a frequent user
• I want to receive search suggestions based on my history and popular searches
• So that I can quickly access previously found tools

Acceptance Criteria:

• [ ] Suggestions appear after 3+ historical queries
• [ ] Popular searches within organization are included
• [ ] Suggestions update in real-time

Terminology

Data Models

4.1 Tool Index Entry

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ToolIndexEntry",
  "type": "object",
  "properties": {
    "tool_id": {
      "type": "string",
      "pattern": "^[a-zA-Z0-9_-]{1,64}$",
      "description": "Unique identifier for the tool"
    },
    "name": {
      "type": "string",
      "minLength": 1,
      "maxLength": 128,
      "description": "Human-readable tool name"
    },
    "description": {
      "type": "string",
      "maxLength": 4096,
      "description": "Detailed description of tool functionality"
    },
    "category": {
      "type": "string",
      "enum": [
        "network",
        "data_processing",
        "file_system",
        "security",
        "database",
        "communication",
        "analytics",
        "integration",
        "utility",
        "custom"
      ]
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string",
        "maxLength": 32
      },
      "maxItems": 20
    },
    "parameters": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/ParameterDefinition"
      }
    },
    "return_type": {
      "type": "string",
      "enum": ["string", "number", "boolean", "object", "array", "binary", "stream", "void"]
    },
    "version": {
      "type": "string",
      "pattern": "^\\d+\\.\\d+\\.\\d+$"
    },
    "author": {
      "type": "string",
      "maxLength": 128
    },
    "created_at": {
      "type": "string",
      "format": "date-time"
    },
    "updated_at": {
      "type": "string",
      "format": "date-time"
    },
    "usage_count": {
      "type": "integer",
      "minimum": 0
    },
    "success_rate": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    }
  },
  "required": ["tool_id", "name", "description", "category"],
  "definitions": {
    "ParameterDefinition": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "type": {
          "type": "string"
        },
        "required": {
          "type": "boolean"
        },
        "description": {
          "type": "string"
        },
        "default": {}
      },
      "required": ["name", "type"]
    }
  }
}

Search Request

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ToolSearchRequest",
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 500,
      "description": "Search query string"
    },
    "filters": {
      "$ref": "#/definitions/SearchFilters"
    },
    "options": {
      "$ref": "#/definitions/SearchOptions"
    }
  },
  "required": ["query"],
  "definitions": {
    "SearchFilters": {
      "type": "object",
      "properties": {
        "categories": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "tags": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "return_types": {
          "type": "array",
          "items": {
            "type": "string",
            "enum": ["string", "number", "boolean", "object", "array", "binary", "stream", "void"]
          }
        },
        "min_success_rate": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "authors": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "date_range": {
          "$ref": "#/definitions/DateRange"
        }
      }
    },
    "SearchOptions": {
      "type": "object",
      "properties": {
        "limit": {
          "type": "integer",
          "minimum": 1,
          "maximum": 100,
          "default": 10
        },
        "offset": {
          "type": "integer",
          "minimum": 0,
          "default": 0
        },
        "fuzzy_threshold": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "default": 0.7
        },
        "include_disabled": {
          "type": "boolean",
          "default": false
        },
        "rank_by": {
          "type": "string",
          "enum": ["relevance", "popularity", "recency", "success_rate"],
          "default": "relevance"
        }
      }
    },
    "DateRange": {
      "type": "object",
      "properties": {
        "start": {
          "type": "string",
          "format": "date-time"
        },
        "end": {
          "type": "string",
          "format": "date-time"
        }
      }
    }
  }
}

Search Response

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "ToolSearchResponse",
  "type": "object",
  "properties": {
    "results": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/SearchResult"
      }
    },
    "total_count": {
      "type": "integer"
    },
    "query": {
      "type": "string"
    },
    "pagination": {
      "$ref": "#/definitions/PaginationInfo"
    },
    "suggestions": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "facets": {
      "$ref": "#/definitions/SearchFacets"
    }
  },
  "definitions": {
    "SearchResult": {
      "type": "object",
      "properties": {
        "tool": {
          "$ref": "ToolIndexEntry"
        },
        "score": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "matched_fields": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "highlight": {
          "type": "string"
        }
      }
    },
    "PaginationInfo": {
      "type": "object",
      "properties": {
        "limit": {
          "type": "integer"
        },
        "offset": {
          "type": "integer"
        },
        "has_more": {
          "type": "boolean"
        }
      }
    },
    "SearchFacets": {
      "type": "object",
      "properties": {
        "categories": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/FacetCount"
          }
        },
        "tags": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/FacetCount"
          }
        },
        "return_types": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/FacetCount"
          }
        }
      }
    },
    "FacetCount": {
      "type": "object",
      "properties": {
        "value": {
          "type": "string"
        },
        "count": {
          "type": "integer"
        }
      }
    }
  }
}

Functional Requirements

Core Search Capabilities

Index Management

Caching

Performance

Error Conditions

API Contracts

Search Endpoint

POST /api/v1/tools/search

Request Headers

Content-Type: application/json
Authorization: Bearer <token>
X-Request-ID: <uuid>

Request Body

{
  "query": "HTTP client for REST APIs",
  "filters": {
    "categories": ["network"],
    "return_types": ["object"]
  },
  "options": {
    "limit": 10,
    "offset": 0,
    "rank_by": "relevance"
  }
}

Success Response (200)

{
  "results": [
    {
      "tool": {
        "tool_id": "http_client_v2",
        "name": "HTTP Client",
        "description": "Advanced HTTP client with retry logic and timeout support",
        "category": "network",
        "tags": ["http", "rest", "api", "client"],
        "parameters": [...],
        "return_type": "object",
        "version": "2.1.0",
        "author": "platform-team",
        "created_at": "2024-01-15T10:00:00Z",
        "updated_at": "2024-06-20T14:30:00Z",
        "usage_count": 15420,
        "success_rate": 0.987
      },
      "score": 0.95,
      "matched_fields": ["name", "description", "tags"],
      "highlight": "Advanced <em>HTTP</em> <em>client</em> with retry logic"
    }
  ],
  "total_count": 5,
  "query": "HTTP client for REST APIs",
  "pagination": {
    "limit": 10,
    "offset": 0,
    "has_more": false
  },
  "suggestions": ["http request", "rest client"],
  "facets": {
    "categories": [{"value": "network", "count": 5}],
    "tags": [{"value": "http", "count": 4}, {"value": "rest", "count": 3}],
    "return_types": [{"value": "object", "count": 3}]
  }
}

Index Management Endpoints

POST /api/v1/tools/index/rebuild - Rebuild the entire search index
POST /api/v1/tools/index/update - Update index with new/modified tool
DELETE /api/v1/tools/index/tool/{tool_id} - Remove tool from index
GET /api/v1/tools/index/stats - Get index statistics

Cache Management Endpoints

DELETE /api/v1/tools/search/cache - Clear all cached results
DELETE /api/v1/tools/search/cache/{tool_id} - Invalidate cache for specific tool
GET /api/v1/tools/search/cache/stats - Get cache statistics

Non-Functional Requirements

Performance Requirements

Scalability Requirements

• Horizontal Scaling: Support multiple search instances with shared cache
• Index Sharding: Partition index by category for faster filtering
• Embedding Storage: Efficient storage for 768-dimension embeddings
• Cache Distribution: Redis cluster support for distributed caching
• Rate Limiting: 100 requests/minute per user, configurable per user group

Availability Requirements

• Uptime: 99.9% availability (SLA)
• Graceful Degradation: Continue with keyword search if embeddings unavailable
• Recovery Time: Index rebuild within 30 seconds of corruption detection
• Cache Persistence: Survive restart with persisted cache data

Reliability Requirements

• Data Consistency: Index synchronized with tool registry
• Cache Coherence: Invalidations propagate within 100ms
• Search Accuracy: >90% user satisfaction in relevance testing
• Zero Data Loss: Persistent index prevents data loss on restart

Security Requirements

Authentication and Authorization

Input Validation

Data Protection

Rate Limiting and DDoS Protection

Audit and Monitoring

Secure Configuration

Security Testing

Compliance Requirements

Technical Design

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Client Application                        │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                    Load Balancer                            │
└──────────────────────────┬──────────────────────────────────┘
                           │
            ┌──────────────┴──────────────┐
            ▼                             ▼
┌────────────────────┐          ┌────────────────────┐
│   Search API       │          │   Search API       │
│   Instance 1       │          │   Instance 2       │
└──────────┬─────────┘          └──────────┬─────────┘
           │                               │
           ▼                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  In-Memory Index                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │  Category   │  │ Embeddings  │  │  Tool ID    │         │
│  │  Partition  │  │   Cache     │  │    Index    │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                   Redis Cache Cluster                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │  Search     │  │  Embedding  │  │    Query    │         │
│  │  Results    │  │   Cache     │  │  History    │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
└─────────────────────────────────────────────────────────────┘

Search Pipeline

1. Query Processing: Input validation, query parsing, partial query handling
2. Index Lookup: Category filtering, candidate retrieval, embedding lookup
3. Ranking Computation: Semantic similarity, TF-IDF scoring, popularity weighting
4. Result Assembly: Sorting, pagination, highlighting, facet calculation
5. Caching: Cache key generation, result storage with TTL, LRU eviction

Index Management

• Build Process: Tool retrieval, embedding generation, TF-IDF vocabulary, category partitioning, disk persistence
• Update Process: Change notification, embedding update, index update, cache invalidation

Caching Strategy

• Cache Keys: Hash of query + normalized filters + options
• TTL: 5 minutes default, configurable
• Invalidation: Tool updates, index rebuild, manual invalidation
• Statistics: Hit/miss ratio, latency comparison, memory usage

Examples

Basic Search

Request:

POST /api/v1/tools/search
{
  "query": "HTTP client"
}

Response:

{
  "results": [
    {
      "tool": {
        "tool_id": "http_client_v2",
        "name": "HTTP Client",
        "description": "Advanced HTTP client with retry logic",
        "category": "network",
        "tags": ["http", "client", "rest"],
        "version": "2.1.0"
      },
      "score": 0.95,
      "matched_fields": ["name", "description", "tags"]
    }
  ],
  "total_count": 1,
  "query": "HTTP client"
}

Filtered Search

Request:

POST /api/v1/tools/search
{
  "query": "parse data",
  "filters": {
    "categories": ["data_processing"],
    "return_types": ["object"]
  },
  "options": {
    "limit": 5
  }
}

Paginated Search

Request:

POST /api/v1/tools/search
{
  "query": "database",
  "options": {
    "limit": 10,
    "offset": 20,
    "rank_by": "popularity"
  }
}

Security

Search Ranking Algorithm

The search ranking algorithm combines multiple signals:

1. Semantic Similarity (40%): Embedding-based similarity between query and tool description
2. Keyword Match Score (25%): TF-IDF weighted keyword matching
3. Popularity Factor (15%): Normalized usage count
4. Success Rate (10%): Tool success rate as quality indicator
5. Recency Bonus (10%): Recently updated tools get a boost

final_score = (0.40 * semantic_score) + 
              (0.25 * keyword_score) + 
              (0.15 * popularity_score) + 
              (0.10 * success_rate_score) + 
              (0.10 * recency_score)

Indexing Strategy

• Initial Index: Built from tool registry on startup
• Incremental Updates: New tools added to index within 100ms
• Index Partitioning: Tools partitioned by category for faster filtering
• Embedding Storage: Pre-computed embeddings stored with index entries
• Deduplication: Duplicate entries prevented via tool_id uniqueness constraint

Cache Invalidation

Access Control

• Search queries require valid authentication token
• Rate limiting: 100 requests per minute per user
• Results filtered by user's tool access permissions
• Audit logging for all search queries
• Input validation prevents injection attacks
• Query logs sanitized to remove sensitive data

Data Privacy

• Search queries not persisted by default
• User search history stored separately with opt-in consent
• Analytics data anonymized
• Embeddings generated without user-identifying information
• Cache entries expire automatically

Governance

Ownership

Change Management

• Minor Changes: Team lead approval, version bump (X.Y)
• Major Changes: Architecture review board, version bump (X.0)
• Emergency Fixes: Can deploy immediately with post-hoc review
• Rollback: Automated rollback on error rate threshold breach

Deprecation Policy

• Announcement: 90 days before deprecation
• Compatibility: Maintained for 1 year minimum
• Migration Guide: Provided for major versions
• Sunset Date: Clearly communicated in API responses

Compliance

• Audit: Monthly review of search logs and access patterns
• Privacy: GDPR compliant query handling
• Security: Quarterly security assessment
• Performance: Weekly performance report

Dependencies

Production Dependencies

Development Dependencies

External Services

Testing Requirements

Unit Tests

Integration Tests

Performance Tests

Test Coverage Targets

• Line Coverage: ≥ 90%
• Branch Coverage: ≥ 85%
• Function Coverage: ≥ 95%
• Critical Path Coverage: 100%

Integration Points

Tool Registry Integration

• Connection: REST API to tool registry service
• Protocol: HTTP/HTTPS with authentication
• Data Flow: Pull-based synchronization
• Update Mechanism: Webhook notifications on tool changes
• Failure Handling: Retry with exponential backoff
• Data Format: JSON with schema validation

Embedding Service Integration

• Connection: gRPC to embedding service
• Protocol: Protocol Buffers over HTTP/2
• Data Flow: Synchronous generation per tool
• Update Mechanism: Batch embedding generation
• Failure Handling: Graceful degradation to keyword-only
• Model Version: Configurable embedding model

Redis Cache Integration

• Connection: Redis Cluster with client-side sharding
• Protocol: RESP (Redis Serialization Protocol)
• Data Flow: Direct client connection with connection pooling
• Update Mechanism: Cache-aside pattern
• Failure Handling: Continue without cache, log degradation
• Data Format: Serialized search results with TTL

Monitoring Integration

• Metrics: Prometheus metrics endpoint
• Logging: Structured logs to central logging service
• Tracing: OpenTelemetry distributed tracing
• Alerting: PagerDuty integration for critical issues
• Dashboard: Grafana dashboards for performance metrics

User Feedback Integration

• Feedback Collection: API endpoint for search result feedback
• Analytics: Anonymous usage analytics
• A/B Testing: Framework for testing ranking improvements
• Quality Metrics: Automated relevance scoring

Success Criteria

• [ ] Search returns relevant results within 100ms for 95% of queries
• [ ] Fuzzy matching corrects up to 2 character substitutions
• [ ] All filters (category, tags, return_type) work correctly in combination
• [ ] Pagination returns correct result subsets
• [ ] Cache reduces average latency by at least 50%
• [ ] Cache invalidation occurs within 100ms of tool update
• [ ] Index rebuild completes for 10,000 tools in under 30 seconds
• [ ] All error conditions return appropriate HTTP status codes
• [ ] Search results include relevance scores
• [ ] All unit tests pass with >90% coverage
• [ ] Semantic search provides better relevance than keyword-only (user study)
• [ ] System handles 1000 searches/second at P95 latency target
• [ ] Cache hit ratio exceeds 80% in production
• [ ] Zero data loss during normal operations
• [ ] Graceful degradation when embedding service unavailable
• [ ] All 12 error conditions tested and documented
• [ ] Search suggestions improve user efficiency by 25%
• [ ] Integration tests pass with real external dependencies
• [ ] Security audit passes with no critical findings
• [ ] Performance tests meet all P95 and P99 targets

Observability

Logging Requirements

1. Component Logging
   • Log all major operations with appropriate log levels
   • Include correlation IDs for request tracing
   • Log performance metrics for critical operations
   • Log errors with full context for debugging

Metrics Requirements

1. Performance Metrics
   • Operation duration histograms
   • Request/error rate counters
   • Resource utilization gauges
   • Business metrics relevant to domain

Tracing Requirements

1. Distributed Tracing
   • Trace propagation across component boundaries
   • Span attributes for domain-specific context
   • Trace sampling based on operational needs

Alerting Requirements

1. Critical Alerts
   • Error rate spikes
   • Performance degradation
   • Resource exhaustion
   • Business metric anomalies

Dashboard Requirements

1. Operational Dashboard
   • Real-time system health
   • Performance trends
   • Error analysis
   • Resource monitoring

Revision History

Related

• RTM page for 072-tool-search-tool
• All specs
• RTM overview
• Gates