Shipping logs to Loki, journal first

On October 12 I brought Loki and Grafana Alloy up on the monitoring VM. On October 14 I added collectors to the n8n and Forgejo VMs. I wanted a place to query system logs across those hosts and one useful alert from them. I also wanted to know what I was collecting before application output started accumulating on another machine.

The rollout ended with both hosts shipping their journals, Forgejo shipping reviewed container logs, and n8n staying journal-only. Getting there exposed a permissions problem in the collector and a smaller mistake in my validation configuration.

A separate project on the existing VM

I deployed Loki and Alloy as their own Compose project on the monitoring VM. Prometheus, Alertmanager, and Grafana already lived there. The new project joined the existing Docker network, so Loki could reach Alertmanager and Grafana could reach Loki without moving the existing services into a different Compose file.

That gave the logging stack its own startup and rollback commands. It still shared the VM and the host’s failure domain. I accepted that for this rollout: these logs were diagnostic data, and metrics already depended on the same monitoring host.

I pinned the images by digest and ran Loki’s config verification and Alloy’s validator before starting them. Loki came up first, then Alloy. Loki listened on the LAN address, 192.0.2.94:3100; Alloy’s HTTP interface stayed on loopback.

I provisioned the Grafana datasource as code and left Prometheus as the default. Grafana recovered from its restart in about ten seconds. Explore found the journal marker, and the three provisioned dashboards still returned data.

The 12 GB cap needs a gate

The monitoring VM had a 48 GB root filesystem and about 34 GB free at the initial deployment. I set a 12 GB budget for Loki and kept the configured retention at fourteen days, with the compactor enabled and a two-hour deletion delay.

Calling that a cap needs some care. Loki had no size quota. Retention, measured growth, and a free-space alert were the controls. The packet’s growth check was:

daily growth in GB * 14 * 1.25 <= 12

That works out to roughly 0.68 GB a day. It was a budget to check against measurements, not a promise that the fifteenth gigabyte could never be written.

For the LAN rollout I made the disk check an explicit gate before the first collector and again after each host. Root had to have at least 30 percent free, and loki-data had to be below 12,000,000,000 bytes. Either failure meant stop. Shortening retention just to pass the gate was not part of the procedure.

I also added MonitoringRootFreeLow, a warning below 30 percent free for fifteen minutes. The existing disk warning at 15 percent left too little room for this budget. After the final Forgejo change, root was still 71 percent free and Loki data was about 2.7 MB. That proved the rollout fit at that moment; it did not replace the planned growth and retention checks.

One alert I could prove

The first log alert was SSHAuthFailures. It used the journal, so I could prove alerting before collecting application logs.

The rule counted at least eight matching log entries in ten minutes, grouped by host, with a five-minute hold. The patterns covered failed passwords, invalid users, and authentication failures. I left out the earlier pattern for a connection closed by an authenticating user because that also counted harmless probes.

The rule also mapped the host into an instance label. Alertmanager grouped on alert name and instance, so leaving instance absent could combine failures from different hosts into one notification.

I tested it with eight synthetic failed-password records under the fake host phase43-test. I did not cause real failed logins. The alert was pending after a minute and firing after the five-minute hold. Alertmanager showed it active with the configured ntfy and Discord warning receivers. That gave me evidence for the rule and routing before widening collection.

The journal was the first change

The n8n and Forgejo collectors went into separate Compose projects on their respective VMs. Their first configurations collected allowlisted journal units and kernel records. I removed Docker discovery, the Docker log source, and the socket mount from that first deployment.

The containers had read-only filesystems, all Linux capabilities dropped, and explicit mounts for journal access and state. Container logs were a later change, with a sample review and a separate decision about Docker socket access.

For validation I used a dedicated phase43-validation systemd unit. That mattered because the journal configuration had an allowlist. On the original monitoring deployment, ordinary logger messages did not appear; a marker from the allowed unit did.

The initial Loki deployment also rejected backfill older than seven days. That was an ingestion-age limit, separate from the fourteen-day retention setting. I needed to distinguish an expected rejection from a broken collector.

Root could not write its state

The first n8n collector restart-looped with:

mkdir /var/lib/alloy/data: permission denied

The image owned /var/lib/alloy as UID 473 with mode 770. The collector ran as root, but with every capability dropped it could not traverse that parent directory. The writable state mount below it did not solve the parent-directory permission problem.

The monitoring collector had not exposed this because its Compose configuration did not drop capabilities. I had validated a less restricted runtime and then carried the state path into the stricter one.

I stopped the failed collector and changed both the state mount target and Alloy’s storage argument:

- --storage.path=/var/lib/alloy/data
+ --storage.path=/var/lib/alloy-data

I kept the capability drop. The new path avoided the inaccessible image directory without making the container less restricted. After redeployment, the n8n collector ran with a restart count of zero. Forgejo’s collector passed its configuration checks and ran with zero restarts too. The n8n application itself was untouched.

Markers from both hosts reached Loki within twenty seconds, one stream and one entry each. I checked the disk gate again and confirmed Alertmanager was healthy with its warning route unchanged.

Reading a sample changed the scope

I reviewed the application samples in a private terminal before enabling container collection. Forgejo’s output included request routing, usernames, and repository paths. I accepted those logs for authentication visibility and limited discovery to the Forgejo Compose service.

n8n stayed journal-only. Its execution output could carry workflow payloads and credentials. The decision to collect Forgejo output did not make n8n output acceptable automatically.

Enabling Forgejo collection also meant accepting the Docker socket. The bind mount was read-only, but that did not make the Docker API read-only. Socket access was effectively root-equivalent on that VM. Dropping capabilities from Alloy did not remove that trust decision.

Forgejo produced 38 container log entries in Loki within twenty-five seconds. Then the journal marker disappeared.

The generated journal-only config included phase43-validation, but the repository config used for container collection did not. Other allowlisted units still flowed. I had replaced a working validation allowlist with one that dropped the test itself.

I added the unit to both source configurations and redeployed Forgejo. The marker returned. That check caught a real difference between the two configuration paths that a healthy container alone would have missed.

What I would change next time

I would test the collector’s state path under the final capability and filesystem restrictions before treating the pilot as evidence for the remote rollout. I would also put the validation unit into the source configs from the start, so switching collection modes kept the same end-to-end check.

I would keep the sample review before container collection. It gave me two different answers for two hosts, which was useful. Journal-only is the current collection scope for n8n, not an unfinished checkbox.

The LAN collectors are working. The tailnet listener still waits on its separate decision, and cloud collection is deferred. I still need the measured growth and retention-deletion checks before considering a longer retention window. Those are the next pieces of evidence to collect.