This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Log Shipping

Storm Pulse can tail logs on your server and ship them to the dashboard in real time. This powers the activity logs you see in Storm Cellar and the agent activity view in the developer dashboard.

Three source types are supported:

• docker_stream (recommended, default for new installs) — spawns one long-lived docker logs --follow --timestamps subprocess per container and reads new lines as they arrive. One process per container for its lifetime; zero per-interval fork cost.
• docker — runs docker logs --since <last_ts> on every interval. Simpler, but forks a subprocess per container per cycle. Kept for backward compatibility.
• file — tails a file on disk by byte offset. Use this when the log is produced by a non-Docker process, or when you already have logs rotating into a file.

Common log shapes the dashboard knows how to parse:

• Storage — S3 access logs from Garage (garage_s3 parser)
• Pulse — the agent's own activity log (stormpulse parser)
• Raw container output — any container (docker_raw parser)
• Network — Caddy JSON access logs and TLS cert-lifecycle events (caddy_json parser)

The caddy_json parser is dual-shape: it routes HTTP access lines (with client_ip, method, host, path, status, user_agent, duration_ms, bytes_sent) and TLS cert-lifecycle lines (from any tls.* logger, preserving logger, msg, identifier, names, error) through the same pipeline. The dashboard classifies cert events; the agent does not interpret them. See Caddy Integration — Caddy log shipping for details.

---

Setup

Automatic (recommended): stormpulse logging init

On the VPS, after enrollment:

stormpulse logging init

The wizard:

1. Lists running Docker containers.
2. Asks whether to enable log shipping for all of them, or prompts per container.
3. Writes one [[log_groups]] block per enabled container to ~/.config/stormpulse/stormpulse.toml.
4. Offers to restart the agent.

The generated blocks use source_type = "docker_stream". Parser selection is automatic for known containers:

Container name contains	Parser	Filter
caddy	caddy_json	(none)
garaged / garage	garage_s3	garage_api_common::generic_server
anything else	docker_raw	(none)

docker_raw captures every line verbatim with the Docker timestamp prefix stripped. The other parsers extract structured fields (level, method, path, status, etc.) and build a clean message for the dashboard.

Example interaction:

Checking for Docker containers...
  Found 4 running container(s): web, db, caddy, garaged

Enable log shipping for all 4 containers? [Y/n]:
Docker binary [/usr/bin/docker]:
Ship interval seconds [10]:
  Added: web (docker)
  Added: db (docker)
  Added: caddy (docker)
  Added: garaged (docker)

  4 log group(s) written to ~/.config/stormpulse/stormpulse.toml
Restart stormpulse now? [Y/n]:

The wizard also runs automatically near the end of stormpulse init (main enrollment flow), so new installs pick up container logs without a separate step.

Migrating existing installs to docker_stream

Existing configs with source_type = "docker" keep working unchanged. To opt in to the streaming approach (recommended — much less dockerd churn):

sed -i 's/source_type = "docker"/source_type = "docker_stream"/g' \
    ~/.config/stormpulse/stormpulse.toml
systemctl --user restart stormpulse

Or just re-run stormpulse logging init and write fresh blocks.

Adding a container manually

If you missed a container, or want to add one without re-running the wizard, paste a block like this into ~/.config/stormpulse/stormpulse.toml and restart:

[[log_groups]]
name = "web"
enabled = true
source_type = "docker_stream"
container_name = "web"
docker_binary = "/usr/bin/docker"
filter_contains = ""
parser = "docker_raw"
ship_interval_seconds = 10
max_lines_per_batch = 200
retention_days = 90

Manual / File source

Use the file source when the data isn't in a container, or when you've already set up log rotation on disk.

1. Make sure the agent's user can read the file

The agent runs as your admin user, so any file readable by that user works. Most host logs under /var/log/ are readable by the adm group — playbook 001 puts admin users in adm, so this is usually free. Confirm:

ls -l /var/log/garage/garaged.log   # check the group
groups                              # check you're in it

If the file is mode 0600 and root-owned, either change its perms at the source (preferred) or add a group-read rule via a logrotate create directive.

2. Point the process at a file

For Garage, bind-mount a host directory into the container so it writes logs there, or run a cron that writes docker logs garaged >> /var/log/garage/garaged.log.

3. Add a [[log_groups]] block

[[log_groups]]
name = "storage"
enabled = true
source_type = "file"
source_path = "/var/log/garage/garaged.log"
filter_contains = "garage_api_common::generic_server"
parser = "garage_s3"
ship_interval_seconds = 10
max_lines_per_batch = 200
retention_days = 90

Restart the agent:

systemctl --user restart stormpulse

---

Configuration reference

Field	Required	Applies to	Notes
name	yes	both	Alphanumeric/underscore, 1–50 chars. Must be unique.
enabled	yes	both	Set to false to keep the block but stop shipping.
source_type	yes	both	"docker_stream" (recommended), "docker", or "file".
source_path	file only	file	Absolute path to the log file.
container_name	docker only	docker	Exact Docker container name (as shown in docker ps).
docker_binary	no	docker	Defaults to /usr/bin/docker.
parser	yes	both	docker_raw, garage_s3, stormpulse, or caddy_json.
filter_contains	no	both	Substring filter applied before parsing. Empty = no filter.
ship_interval_seconds	yes	both	How often to read and ship. Must be ≥ 5.
max_lines_per_batch	yes	both	1–200.
retention_days	yes	both	1–365. Enforced by the dashboard.

---

How it works

File source: the agent tails each file, remembers the byte offset and inode in SQLite, and detects rotation automatically.

Docker stream source (docker_stream): the agent spawns docker logs --follow --timestamps --since <last_ts> <container> once and keeps it running. Each interval simply drains whatever lines have arrived on the subprocess's stdout pipe. One process per container, no per-cycle fork. If the container stops or restarts the subprocess exits; the agent detects this on the next read and respawns after a 5-second backoff, using the stored cursor so no lines are missed across the restart.

Docker source (docker, legacy): the agent runs docker logs <container> --since <last_ts> --timestamps on every interval. Simpler, but forks a docker subprocess per container per cycle.

Both docker variants remember the timestamp of the last line shipped; the next run uses that as the --since boundary. No file mount, no rotation concerns. All three paths feed the same shipper: filter → parse → batch → send over the existing mTLS connection. Position is only advanced after the dashboard acknowledges the batch, so a crash or network blip re-ships the last batch rather than losing it.

A note about duplicate lines

docker logs --since is inclusive of its boundary timestamp. When the agent asks for "logs since 13:23:51.766230Z", Docker includes the line that happened at exactly 13:23:51.766230Z — which the agent already shipped in the previous batch.

The dashboard deduplicates on insert (ignore_conflicts=True on the primary key), so this is invisible to you. If you're watching agent -v output closely you'll see the occasional repeat in flight; that's expected.

---

Troubleshooting

Symptom	Check
No logs appearing (docker)	Is the container running? Is container_name spelled exactly as in docker ps? Does docker ps work as the agent's user (rootless dockerd up)?
docker: command not found	Check docker_binary in the log group — it must be an absolute path to an existing docker binary.
Logs appear but are delayed	ship_interval_seconds controls end-to-end latency. Default is 10 seconds; lower values cost more dashboard round-trips.
No logs appearing (file)	Is the source file being written to? Is enabled = true? Does the agent's user have read access to the file?
Agent won't start	Run stormpulse run ~/.config/stormpulse/stormpulse.toml to see config errors.