orchestrata_ultimate.core.atomic_json

Reference › orchestrata_ultimate › core › atomic_json

Atomic file writes for JSON and text, used by the generator and gate scripts and by traceability_utils. A plain open(path, "w") truncates the target at once, so two processes writing the same path leave a splice of both documents. The module records a measured case: a 75 MB AST snapshot decoded as a complete object followed by trailing bytes of a previous generation, which crashed a gate run before any gate reported.

dump_json_atomic and dump_text_atomic both go through _write_atomic: write to a temporary file in the destination directory (same filesystem, so the final step is a rename), fsync it, then os.replace it over the target. Concurrent writers therefore produce a whole document with last-writer-wins, and a crashed writer leaves the previous file untouched. Parent directories are created, a new file gets mode 0644, and an existing target keeps its mode. Options include indent, default, sort_keys and trailing_newline.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

API

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/core/atomic_json.py

"""Atomic JSON writes — a truncate-in-place write is a corruption waiting to land.

``open(path, "w")`` truncates the target the instant it is called and then
streams bytes into it. Two processes doing that to the same path do not
serialise: each truncates, each writes from offset 0, and the file that
survives is a SPLICE of both — the shorter writer's prefix followed by the
longer writer's tail. The splice is still a file, still readable, and
``json.loads`` reports it as ``Extra data: line N column M`` far from
anything a human wrote.

Measured, 2026-08-12: ``plans/ast-ultimate.json`` (75,563,054 bytes) decoded
to a complete object at byte 73,507,683 with 2,055,371 bytes of a previous
generation trailing it, and hard-crashed ``scripts/gates.py --all`` in
``_runtime_call_count`` before a single gate reported. Three call sites wrote
that path with a bare ``open(..., "w")``: ``serialize_codebase_ast._cmd_serialize``
and ``build_runtime_callgraph.{build,reaugment}``.

:func:`dump_json_atomic` writes to a temporary file in the SAME directory —
same filesystem, so :func:`os.replace` is a rename, not a copy — fsyncs it,
then atomically replaces the target. Concurrent writers then produce
last-writer-wins with a whole document either way, and a crashed writer
leaves the previous file untouched instead of a truncated stub.
"""

from __future__ import annotations

import json
import os
import tempfile
from collections.abc import Callable
from pathlib import Path
from typing import Any, TextIO

__all__ = ["dump_json_atomic", "dump_text_atomic"]

#: Mode applied to a newly created file. An existing target's mode is
#: preserved instead — ``tempfile.mkstemp`` creates 0600, which would
#: silently tighten permissions on every rewrite of a shared artifact.
_DEFAULT_MODE = 0o644


def dump_json_atomic(
    path: str | os.PathLike[str],
    obj: Any,
    *,
    indent: int | None = 2,
    default: Callable[[Any], Any] | None = None,
    sort_keys: bool = False,
    trailing_newline: bool = False,
) -> None:
    """Serialise ``obj`` to ``path`` as JSON, atomically.

    The target is either fully replaced by the new document or left exactly
    as it was; no reader ever observes a partial or spliced file.

    Args:
        path: Destination file. Parent directories are created.
        obj: Any ``json.dump``-serialisable object.
        indent: Passed through to ``json.dump``.
        default: Fallback serialiser, passed through to ``json.dump``.
        sort_keys: Passed through to ``json.dump``.
        trailing_newline: Append ``"\\n"`` after the document. Several
            existing artifacts are written that way; keeping the flag makes
            this a byte-for-byte drop-in for them.

    Raises:
        OSError: The temporary file could not be created, written, or
            renamed. The destination is untouched in every such case.
        TypeError: ``obj`` is not serialisable. Destination untouched.
    """

    def _emit(handle: TextIO) -> None:
        json.dump(obj, handle, indent=indent, default=default, sort_keys=sort_keys)
        if trailing_newline:
            handle.write("\n")

    _write_atomic(path, _emit)


def dump_text_atomic(
    path: str | os.PathLike[str],
    text: str,
    *,
    trailing_newline: bool = False,
) -> None:
    """Write ``text`` to ``path`` atomically — the text sibling of the above.

    Same guarantee, same reason. A generated ``.md`` is exactly as unsafe to
    truncate in place as a generated ``.json``: ``docs/CODEBASE_MAP.md`` and
    ``docs/CODEBASE_MAP_FULL.md`` are regenerated on a SessionStart hook while
    the tree they describe is being staged, and ``Path.write_text`` truncates
    the destination IN PLACE, keeping the inode. A reader holding an mmap of
    that inode — ``git update-index`` hashing the file to stage it — then
    faults on pages past the new end and dies with ``SIGBUS``/``BUS_ADRERR``.

    Measured 2026-08-19: two ``git update-index --add`` cores, 28 minutes
    apart, one inside ``zng_deflate``. Hardware was excluded on four
    independent counters (NVMe media errors 0, btrfs corruption/csum/IO
    counters all 0, scrub clean over 344 GiB, SMART PASSED), which leaves a
    file that shrank under a live mapping.

    Replacing the target instead of truncating it also fixes the reader:
    ``os.replace`` gives the destination a NEW inode, so an mmap already open
    on the old one keeps reading the complete previous file.

    Args:
        path: Destination file. Parent directories are created.
        text: Full document to write.
        trailing_newline: Append ``"\\n"`` unless ``text`` already ends with
            one, so this is a byte-for-byte drop-in for existing writers.

    Raises:
        OSError: The temporary file could not be created, written, or
            renamed. The destination is untouched in every such case.
    """

    def _emit(handle: TextIO) -> None:
        handle.write(text)
        if trailing_newline and not text.endswith("\n"):
            handle.write("\n")

    _write_atomic(path, _emit)


def _write_atomic(path: str | os.PathLike[str], emit: Callable[[TextIO], None]) -> None:
    """Temp file in the destination directory, fsync, then ``os.replace``.

    Extracted so the delicate half — preserve an existing target's mode,
    create the temp on the SAME filesystem so the replace is a rename rather
    than a copy, fsync before publishing, and never leave scratch behind —
    exists once. It was already duplicated privately in four modules
    (``session/context.py``, ``claude/tools/session/context.py``,
    ``secrets_cli.py``, ``serialize_codebase_ast.py``); a second copy here
    would have been the fifth, and the fsync/chmod ordering is precisely the
    sort of detail that drifts between copies unnoticed.
    """
    target = Path(path)
    target.parent.mkdir(parents=True, exist_ok=True)

    try:
        mode = target.stat().st_mode & 0o777
    except FileNotFoundError:
        mode = _DEFAULT_MODE

    fd, tmp_name = tempfile.mkstemp(
        dir=str(target.parent), prefix=f".{target.name}.", suffix=".tmp"
    )
    tmp_path = Path(tmp_name)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            emit(handle)
            handle.flush()
            os.fsync(handle.fileno())
        os.chmod(tmp_path, mode)
        os.replace(tmp_path, target)
    except BaseException:
        # Never leave the scratch file behind — including on KeyboardInterrupt
        # / SystemExit, which is exactly when a 75MB stray hurts most.
        tmp_path.unlink(missing_ok=True)
        raise