Spec Word Cloud

scripts/spec_word_cloud.py is a small, dependency-free (stdlib-only) tool that scans every spec directory under specs/*/ (concatenating each spec's spec.md and requirements.md) and computes a TF-IDF term-prominence ranking across the whole corpus. It is meant to surface the "important objects and nodes" of the project at a glance -- the identifiers, subsystems, and recurring concepts that specs and requirements keep coming back to -- both as a quick visual (the word cloud below) and as a structured JSON report (plans/spec_term_prominence.json) that other tooling (or a human) can consume.

The tokenizer treats two kinds of terms specially. CamelCase and snake_case identifiers (e.g. PipelineRunner, CXPPacket, intent_broker) are counted twice: once intact, under their original casing and with a 2x weight boost, because they read as concrete "objects" or graph nodes in the codebase; and once split into their lowercase constituent words (pipeline, runner, cxp, packet, intent, broker), so ordinary word-frequency statistics are not skewed purely toward compound identifiers. A standard TF-IDF weighting is then applied: term frequency per spec-document, times a corpus-wide inverse document frequency (log(N_docs / (1 + df))), summed across documents for the global ranking below. This means a term that is rare but concentrated in a single spec can outrank a term that appears everywhere with the same raw count -- exactly the signal that makes a term "prominent" rather than merely "common."

The ranking pipeline:

Optionally (--apply-to-graph), the tool writes a prominence property onto matching entities in the semantic graph (Memgraph/Neo4j-compatible bolt backend), so GraphRAG retrieval and other graph-aware tooling can weight nodes by how central they are to the specs and requirements that describe them. This path is fail-soft: if the graph is unreachable it prints a message and exits cleanly rather than erroring.

Top 40 global terms

Word cloud