Architecture: the kernel and its integrations

Storm Pulse is a kernel with integrations plugged into it. The kernel does the universal work: connect to the dashboard, prove identity, dispatch commands, ship metrics and logs, run background jobs. Everything that knows how to drive a specific system, a Garage S3 node, a Caddy edge proxy, lives in an integration that plugs into the kernel through one contract.

The kernel names no integration. It does not import Garage or Caddy by name, it iterates whatever integrations registered themselves at startup. Garage and Caddy are not special cases baked into the core, they are the first two integrations, written against the same contract yours would use. If you understand the contract, you can read either of them as a worked example, and you can write a third without touching the kernel.

This page is the mental model: what the kernel is, what an integration is, why it is shaped this way, and how to write your own.

---

The model in one picture

                      ┌─────────────────────────────────────┐
                      │              the kernel             │
   dashboard  ◀──────▶│  connect · auth · dispatch · jobs   │
   (mTLS/WS)          │  metrics · logs · refresh           │
                      └───────────────┬─────────────────────┘
                                      │ iterates registered integrations
                ┌─────────────────────┼─────────────────────┐
                ▼                     ▼                      ▼
          ┌───────────┐        ┌───────────┐         ┌─────────────┐
          │  garage   │        │   caddy   │         │   yours     │
          │ (S3 node) │        │  (proxy)  │         │  (anything) │
          └───────────┘        └───────────┘         └─────────────┘

The kernel boots, asks every registered integration "are you configured and healthy?", and for the ones that are, merges their commands into the live registry and starts collecting their state. Nothing in the kernel mentions an integration by name. Adding the box on the right is one registration line plus your module.

---

Why it is built this way

A management agent runs as the most privileged thing touching a server. Two pressures follow from that, and the kernel/integration split is how Storm Pulse answers both.

Integration code stays out of the core. The kernel is the part that is the same no matter what you are driving: connect, auth, dispatch, jobs, metrics, logs. The code that knows a specific system is hundreds of lines, and none of it lives in the kernel, it lives in that system's integration package. Adding Garage added a package, not kernel code. Keeping that boundary is the point: the core should change when the protocol or the security model changes, not every time the agent learns to drive something new.

Capabilities are added at the edge, not the center. You should be able to make the agent manage a new system without editing connect, auth, dispatch, or the loops. An integration declares what it can do and registers itself; the kernel loads it. This is what makes Storm Pulse a platform rather than a fixed tool, and it is why an outside contributor can add real capability without us having to trust them with the core.

Who can add a command. A command enters the whitelist three ways, all controlled by whoever runs the box: the agent's built-in commands, an integration's specs (in this repository or your fork), and subprocess commands an operator writes directly in stormpulse.toml (see Customize Commands). What is not supported yet is a fourth way: a third-party package that registers commands into a running agent, loaded as external code. That code would execute at the agent's privilege, so command contribution stays in-tree, in your fork, or in your own config until an out-of-tree loader earns its own trust boundary (signing, confinement, a dependency allowlist). You extend by forking, by contributing through review, or by config, not by dropping someone else's code into a live agent.

---

The integration contract

An integration is a small descriptor registered at import time. It lives in stormpulse/integrations/registry.py:

@dataclass(frozen=True, slots=True)
class Integration:
    id: str                              # "garage", "caddy", yours
    parse_config: ParseConfig            # raw TOML table -> your typed config
    enabled: EnabledPredicate            # is this turned on?
    preconditions: Preconditions | None = None      # is it healthy enough to run?
    specs: BuildSpecs | None = None                 # the commands it contributes
    discover: CollectState | None = None            # one-shot discovery
    collect_state: CollectState | None = None       # periodic + on-demand state
    detect: Detector | None = None                  # fast new-resource signal + its cadence
    read_affected: ReadAffected | None = None       # post-mutation targeted re-read
    log_enrichers: Mapping[str, BuildLogEnricher] | None = None  # keyed by parser

Only three fields are required: an id, a parse_config, and an enabled predicate. Everything else is opt-in. A read-only monitor that just reports state declares collect_state and nothing else. An integration with no state declares no collect_state. There are no empty stubs to fill in.

You register it once, at the bottom of your module:

register_integration(MY_INTEGRATION)

and add one import line to stormpulse/agent/integrations_manifest.py so it loads. That is the entire wiring. The kernel's startup (stormpulse/agent/bootstrap.py) loops over every registered integration:

for integ in registered_integrations():
    raw = config.integrations.get(integ.id)
    if raw is None:
        continue                 # not configured on this host, skip
    rt = _resolve_integration(integ, raw, commands)   # parse, gate, merge
    ...

There is no if integ.id == "garage" anywhere. Add a fourth integration and bootstrap does not change.

---

Commands are single-source

A command is one CommandSpec (in stormpulse/config.py). The spec carries the command's schema and, for a long-running command, its handler. There is no second map to keep in sync, because there is no second map.

@dataclass(frozen=True, slots=True)
class CommandSpec:
    group: str
    command: list[str]
    timeout: int
    mode: CommandMode = "subprocess"     # "subprocess" | "job" | "refresh"
    requires_confirmation: bool = False
    sensitive_output: bool = False
    read_only: bool = False
    handler: CommandHandler | None = None
    params: dict[str, ParamDef] = field(default_factory=dict)

mode is how the dispatcher routes it. There are exactly three:

subprocess runs a real program. command is the argv, executed with shell=False, and command[0] must be an absolute path. This is the common case: docker exec ... garage bucket list. No handler.

job is a long-running operation handed to the background JobManager: an admin-API call, an S3 walk, an orchestrated provision with rollback. It carries a handler, a lazy thunk that builds the job only when it is dispatched, never at registration. command is a sentinel [name].

refresh you never write by hand. If your integration declares collect_state, the kernel synthesizes a {id}_refresh command for you: "collect this integration's state now and push it." Garage gets garage_refresh because it collects state; your integration gets yours_refresh the same way. It is owned by the kernel (stormpulse/agent/refresh.py), identical for every integration.

Illegal combinations cannot be built. The dataclass rejects them at construction:

• a job with no handler (the classic half-registered command) raises immediately,
• a subprocess whose command[0] is not absolute raises immediately,
• a non-job carrying a handler raises immediately.

This used to be enforced by a hand-maintained list of expected command names in a test. Now it is structural: a malformed command is a ValueError the moment you write it, on your machine, before anything ships. That is the whole reason commands are single-source, a command and its handler that can drift apart will drift apart.

---

The security model, in one place

The command registry is a security layer. The full picture is in Security Architecture; the part that touches integration authors:

• Whitelist only. The agent runs commands that exist in the registry, nothing else. There is no "run arbitrary string" path.
• Absolute paths, shell=False. Subprocess commands name their binary by absolute path and are executed as an argv list. No shell, no PATH games, no injection through arguments.
• Every parameter is validated. A ParamDef declares either a regex pattern or a max_bytes cap. A parameter with neither cannot be constructed. Dashboard-supplied values are checked against the pattern before they ever reach the command.
• Fail loud at load. The construction guards above mean a misconfigured command stops the build, it does not surface as a mystery failure at runtime.
• Command-contributing code is yours, not a stranger's. Commands come from the agent's built-ins, an integration in this repository or your fork, or your own stormpulse.toml. The agent does not load external code that registers commands at runtime, that code would run at the agent's privilege. The registration shape is ready for an out-of-tree loader; the trust boundary for one is reserved for its own decision.

---

The failure model

The rule is simple: a broken integration never takes the agent down.

Invalid core configuration (no dashboard URL, no certificate, no agent id) is fatal, the agent cannot do its job, so it refuses to start. But an integration that fails to parse its config, fails a precondition, or whose commands will not build does not crash anything. It soft-disables: that one integration goes dark, reports a reason the dashboard shows, and the kernel plus every other integration stays up.

This is the same for a first-party integration and a third-party one. Garage gets no privilege here that yours would not. If your integration's spec trips a construction guard at startup, your integration disables with a clear reason and the agent keeps running everything else.

Storm Pulse config lives on the box and is edited in place, so the loud signal is the restart. When the agent comes back up it prints which integrations went dark and why, to the terminal and the dashboard, where a hands-on operator sees it immediately.

---

Write your own integration

First, do you even need one? If you just want a single subprocess command on your own box, you do not, add it to stormpulse.toml (Customize Commands). Reach for an integration when you need what config cannot express: long-running jobs with progress and rollback, state collection and its free {id}_refresh, preconditions, soft-disable, or a capability surface you want to reuse and ship. That is the point of an integration, structured capability, not just one command.

Here is a minimal integration end to end. It manages a fictional service called widget running in a Docker container: it reports the service's state and exposes one command. Because it declares collect_state, it gets widget_refresh for free.

A Storm Pulse integration is a Python package under stormpulse/. Four small files.

1. Config (stormpulse/widget/config.py), parse your slice of the TOML into a typed object:

from dataclasses import dataclass

@dataclass(frozen=True, slots=True)
class WidgetConfig:
    enabled: bool
    container_name: str
    docker_binary: str

def parse_widget_config(raw: dict) -> WidgetConfig:
    return WidgetConfig(
        enabled=bool(raw.get("enabled", False)),
        container_name=str(raw["container_name"]),
        docker_binary=str(raw.get("docker_binary", "/usr/bin/docker")),
    )

2. State (stormpulse/widget/state.py), what the agent collects and reports. The state object owns its to_dict(), and optionally a summary() for a nicer refresh message:

from dataclasses import asdict, dataclass

@dataclass(frozen=True, slots=True)
class WidgetState:
    running: bool
    queue_depth: int

    def to_dict(self) -> dict:
        return asdict(self)

    def summary(self) -> str:          # optional; refresh falls back without it
        return f"{self.queue_depth} queued"

def collect_widget_state(config: WidgetConfig) -> WidgetState | None:
    # Read your service however you like; return None to skip this tick.
    ...

3. Commands (stormpulse/widget/commands.py), the specs. One subprocess command here. Build them from config so paths are not hard-coded:

from stormpulse.config import CommandSpec, ParamDef

def build_widget_specs(config: WidgetConfig) -> dict[str, CommandSpec]:
    docker, container = config.docker_binary, config.container_name
    return {
        "widget_status": CommandSpec(
            group="widget",
            command=[docker, "exec", container, "/usr/bin/widgetctl", "status"],
            timeout=15,
            description="Show widget status",
        ),
    }

If you need a long-running operation (an HTTP admin call, a batch job), make it mode="job" and give it a handler, a thunk that returns a JobHandler (see stormpulse/commands/jobs.py, and stormpulse/garage/set_quota.py for a real one):

        "widget_drain": CommandSpec(
            group="widget",
            command=["widget_drain"],            # sentinel for a job
            timeout=120,
            mode="job",
            handler=lambda params: make_drain_handler(config, params),
            params={
                "target": ParamDef(
                    placeholder="target", default=None,
                    pattern=r"[a-z0-9-]+", description="drain target",
                ),
            },
        ),

4. The integration (stormpulse/widget/integration.py), wire it together and register it:

from stormpulse.integrations import Integration, register_integration
from stormpulse.widget.commands import build_widget_specs
from stormpulse.widget.config import WidgetConfig, parse_widget_config
from stormpulse.widget.state import collect_widget_state

WIDGET_INTEGRATION = Integration(
    id="widget",
    parse_config=parse_widget_config,
    enabled=lambda c: c.enabled,
    specs=build_widget_specs,
    collect_state=collect_widget_state,     # this earns you widget_refresh
)

register_integration(WIDGET_INTEGRATION)

Then one line in stormpulse/agent/integrations_manifest.py:

import stormpulse.widget.integration  # noqa: F401

That is the whole thing. Now a host with this in its config:

[integrations.widget]
enabled = true
container_name = "widget"

gets widget_status, widget_refresh, and (if you added it) widget_drain on the dashboard, its state collected and reported alongside system metrics, and a clean soft-disable if it is misconfigured. You did not touch the kernel.

Read stormpulse/garage/ for the full-featured reference (admin API, jobs with rollback, discovery) and stormpulse/caddy/ for a smaller one (commands, no state loop).

---

Where to look in the code

Concern	File
The integration contract	stormpulse/integrations/registry.py
The command type and its guards	stormpulse/config.py (CommandSpec, ParamDef)
Startup: gate and merge integrations	stormpulse/agent/bootstrap.py
Command dispatch (routes on mode)	stormpulse/agent/dispatch.py
The generic refresh routine	stormpulse/agent/refresh.py
Background jobs	stormpulse/commands/jobs.py
In-tree registration	stormpulse/agent/integrations_manifest.py
Reference integrations	stormpulse/garage/, stormpulse/caddy/

The architecture is enforced, not just described: an import that violates the layering, a command that breaks the whitelist invariants, or a malformed spec fails the build (make check). You do not have to memorize the rules, the tooling holds you to them.