This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Log Shipping

Storm Pulse can tail logs on your server and ship them to the dashboard in real time. This powers the activity logs you see in Storm Buckets and the agent activity view in the developer dashboard.

Four source types are supported:

• docker_stream (recommended, default for new installs) — spawns one long-lived docker logs --follow --timestamps subprocess per container and reads new lines as they arrive. One process per container for its lifetime; zero per-interval fork cost.
• docker — runs docker logs --since <last_ts> on every interval. Simpler, but forks a subprocess per container per cycle. Kept for backward compatibility.
• file — tails a file on disk by byte offset. Use this when the log is produced by a non-Docker process, or when you already have logs rotating into a file.
• journald — tails a systemd unit's journal by cursor. Use this for anything supervised by systemd. The service needs no StandardOutput= redirect, no log directory, no permissions on it and no rotation policy, and the journal supplies a timestamp for services that print bare lines.

Common log shapes the dashboard knows how to parse:

• Storage — S3 access logs from Garage (garage_s3 parser)
• Pulse — the agent's own activity log (stormpulse parser)
• Raw container output — any container (docker_raw parser)
• Network — Caddy JSON access logs and TLS cert-lifecycle events (caddy_json parser)
• Systemd services — one journal record per line (journald parser)

The caddy_json parser is dual-shape: it routes HTTP access lines (with client_ip, method, host, path, status, user_agent, duration_ms, bytes_sent) and TLS cert-lifecycle lines (from any tls.* logger, preserving logger, msg, identifier, names, error) through the same pipeline. The dashboard classifies cert events; the agent does not interpret them. See Caddy Integration — Caddy log shipping for details.

---

Setup

Automatic (recommended): stormpulse logging init

On the VPS, after enrollment:

stormpulse logging init

The wizard:

1. Lists running Docker containers.
2. Asks whether to enable log shipping for all of them, or prompts per container.
3. Writes one [[log_groups]] block per enabled container to ~/.config/stormpulse/stormpulse.toml.
4. Offers to restart the agent.

The generated blocks use source_type = "docker_stream". Parser selection is automatic for known containers:

Container name contains	Parser	Filter
caddy	caddy_json	(none)
garaged / garage	garage_s3	garage_api_common::generic_server
anything else	docker_raw	(none)

docker_raw captures every line verbatim with the Docker timestamp prefix stripped. The other parsers extract structured fields (level, method, path, status, etc.) and build a clean message for the dashboard.

Example interaction:

Checking for Docker containers...
  Found 4 running container(s): web, db, caddy, garaged

Enable log shipping for all 4 containers? [Y/n]:
Docker binary [/usr/bin/docker]:
Ship interval seconds [10]:
  Added: web (docker)
  Added: db (docker)
  Added: caddy (docker)
  Added: garaged (docker)

  4 log group(s) written to ~/.config/stormpulse/stormpulse.toml
Restart stormpulse now? [Y/n]:

The wizard also runs automatically near the end of stormpulse init (main enrollment flow), so new installs pick up container logs without a separate step.

Migrating existing installs to docker_stream

Existing configs with source_type = "docker" keep working unchanged. To opt in to the streaming approach (recommended — much less dockerd churn):

sed -i 's/source_type = "docker"/source_type = "docker_stream"/g' \
    ~/.config/stormpulse/stormpulse.toml
systemctl --user restart stormpulse

Or just re-run stormpulse logging init and write fresh blocks.

Adding a container manually

If you missed a container, or want to add one without re-running the wizard, paste a block like this into ~/.config/stormpulse/stormpulse.toml and restart:

[[log_groups]]
name = "web"
enabled = true
source_type = "docker_stream"
container_name = "web"
docker_binary = "/usr/bin/docker"
filter_contains = ""
parser = "docker_raw"
ship_interval_seconds = 10
max_lines_per_batch = 200

Manual / File source

Use the file source when the data isn't in a container, or when you've already set up log rotation on disk.

1. Make sure the agent's user can read the file

The agent runs as your admin user, so any file readable by that user works. Most host logs under /var/log/ are readable by the adm group — playbook 001 puts admin users in adm, so this is usually free. Confirm:

ls -l /var/log/garage/garaged.log   # check the group
groups                              # check you're in it

If the file is mode 0600 and root-owned, either change its perms at the source (preferred) or add a group-read rule via a logrotate create directive.

2. Point the process at a file

For Garage, bind-mount a host directory into the container so it writes logs there, or run a cron that writes docker logs garaged >> /var/log/garage/garaged.log.

3. Add a [[log_groups]] block

[[log_groups]]
name = "storage"
enabled = true
source_type = "file"
source_path = "/var/log/garage/garaged.log"
filter_contains = "garage_api_common::generic_server"
parser = "garage_s3"
ship_interval_seconds = 10
max_lines_per_batch = 200

Restart the agent:

systemctl --user restart stormpulse

Journald source

Use this for anything supervised by systemd. Nothing on the service side has to change: no StandardOutput= or StandardError= redirect, no log directory to create, no permissions to grant on it, and no rotation policy to add. It also suits services that print bare lines with no timestamp of their own, since the journal records one for every entry.

1. Confirm the unit's journal is readable by the agent's user

journalctl -u my-daemon.service -n 5 --no-pager

Journal read access is usually granted by membership of systemd-journal or adm. If that command works as your admin user, the agent can read it too.

2. Add a [[log_groups]] block

Name the group after the unit: group is the only thing that tells the dashboard which unit a line came from.

[[log_groups]]
name = "my-daemon"
enabled = true
source_type = "journald"
unit = "my-daemon.service"
filter_contains = ""
parser = "journald"
ship_interval_seconds = 10
max_lines_per_batch = 200

Restart the agent:

systemctl --user restart stormpulse

The first interval after adding a journald group ships nothing: the agent seeds its cursor from the newest existing record so it does not replay the unit's whole retained journal. Lines written from then on ship normally.

---

Configuration reference

Field	Required	Applies to	Notes
name	yes	all	Alphanumeric/underscore, 1–50 chars. Must be unique.
enabled	yes	all	Set to false to keep the block but stop shipping.
source_type	yes	all	"docker_stream" (recommended), "docker", "file", or "journald".
source_path	file only	file	Absolute path to the log file.
container_name	docker only	docker	Exact Docker container name (as shown in docker ps).
docker_binary	no	docker	Defaults to /usr/bin/docker.
unit	journald only	journald	Exact systemd unit name, e.g. "my-daemon.service". No whitespace.
parser	yes	all	docker_raw, garage_s3, stormpulse, caddy_json, django, or journald.
filter_contains	no	all	Substring filter applied before parsing. Empty = no filter.
ship_interval_seconds	yes	all	How often to read and ship. Must be ≥ 2.
max_lines_per_batch	yes	all	1–200.

retention_days was removed: the agent tails and ships, it stores nothing, so the knob enforced nothing. A stale key in an existing config logs a deprecation warning instead of failing. Log retention is a dashboard-side concern.

---

How it works

File source: the agent tails each file, remembers the byte offset and inode in SQLite, and detects rotation automatically.

Docker stream source (docker_stream): the agent spawns docker logs --follow --timestamps --since <last_ts> <container> once and keeps it running. Each interval simply drains whatever lines have arrived on the subprocess's stdout pipe. One process per container, no per-cycle fork. If the container stops or restarts the subprocess exits; the agent detects this on the next read and respawns after a 5-second backoff, using the stored cursor so no lines are missed across the restart.

Docker source (docker, legacy): the agent runs docker logs <container> --since <last_ts> --timestamps on every interval. Simpler, but forks a docker subprocess per container per cycle.

Journald source (journald): the agent runs journalctl --unit <unit> --output json --after-cursor <cursor> each interval. The position marker is the journal's own cursor, so there is no byte offset to keep in sync and no rotation to detect: the journal owns both. On the first run for a group there is no cursor, so the agent seeds from the newest existing record and ships nothing, rather than replaying the unit's entire retained journal. A backlog larger than max_lines_per_batch drains oldest-first across intervals rather than being skipped. A missing journalctl, an unknown unit, a timeout or a non-zero exit all produce an empty batch and leave the stored cursor where it was.

Both docker variants remember the timestamp of the last line shipped; the next run uses that as the --since boundary. No file mount, no rotation concerns. All three paths feed the same shipper: filter → parse → batch → send over the existing mTLS connection. Position is only advanced after the dashboard acknowledges the batch, so a crash or network blip re-ships the last batch rather than losing it.

A note about duplicate lines

docker logs --since is inclusive of its boundary timestamp. When the agent asks for "logs since 13:23:51.766230Z", Docker includes the line that happened at exactly 13:23:51.766230Z — which the agent already shipped in the previous batch.

The dashboard deduplicates on insert (ignore_conflicts=True on the primary key), so this is invisible to you. If you're watching agent -v output closely you'll see the occasional repeat in flight; that's expected.

---

Troubleshooting

Symptom	Check
No logs appearing (docker)	Is the container running? Is container_name spelled exactly as in docker ps? Does docker ps work as the agent's user (rootless dockerd up)?
docker: command not found	Check docker_binary in the log group — it must be an absolute path to an existing docker binary.
Logs appear but are delayed	ship_interval_seconds controls end-to-end latency. Default is 10 seconds; lower values cost more dashboard round-trips.
No logs appearing (file)	Is the source file being written to? Is enabled = true? Does the agent's user have read access to the file?
Agent won't start	Run stormpulse run ~/.config/stormpulse/stormpulse.toml to see config errors.