Log Sinks

Pro A space can register itself as a log sink: the knot server mirrors the log records of the space owner’s other spaces in the zone to a log service running inside the sink space. Run VictoriaLogs in a space and watch your other spaces’ logs flow into it, queryable with LogsQL — no server config changes, no admin involvement after the permission is granted.

The Rules

• Owner-scoped, period. A sink only ever receives the logs of spaces owned by the same user — never other users’ logs, no exceptions.
• Tunnels included. A user’s sinks also receive the logs of requests proxied through that user’s web tunnels — tagged with the tunnel name instead of a space — plus the tunnels opening and closing (tunnel_open / tunnel_close records), automatically, no extra configuration. Records carry service: tunnel so they select exactly like space records.
• Off by default. A space becomes a sink only when it explicitly advertises one (below); the base images ship with sink support but don’t enable it.
• Zone-local. Sinks receive logs from spaces in their own zone only.
• Permission-gated. Registration requires the Use Log Sinks permission on the user’s role; without it the registration is ignored (with a warning in the server log). Works in the free built-in Pro tier (2 users) — the licence only lifts the user cap.
• Retry, no disk buffering. Batches and retries on failure, but if the local service is down for long, records are dropped. This is a developer convenience, not a compliance channel — for that, use forward_space_logs on the server (also Pro).

Starting a Sink

A space advertises a sink with environment variables, set on the template or the space. These are agent settings: they are read by the agent inside the space, and setting KNOT_LOG_SINK_PORT is what triggers sink registration. The auth credentials are used only for the agent’s localhost writes — they are never sent to the knot servers or written to the audit log.

The format must match what your log service accepts on that port — the same dialects the agent itself accepts logs in (see sending logs).

Worked example: a VictoriaLogs sink template

The knot-victoria-logs base image bundles VictoriaLogs with the knot agent integration. Sink support is built in but off by default — enable it by setting KNOT_LOG_SINK_PORT in the template. A local-container template spec:

container_name: my-logs
image: "paularlott/knot-victoria-logs:1.52.0"
environment:
  - KNOT_LOG_SINK_PORT=9428
  - KNOT_LOG_SINK_FORMAT=vl
  - VICTORIA_LOGS_RETENTION_PERIOD=30d
volumes:
  - my-logs:/data

For a Nomad template, set the same variables in the task env block:

task "victoria-logs" {
  driver = "docker"
  config {
    image = "paularlott/knot-victoria-logs:1.52.0"
  }
  env {
    KNOT_LOG_SINK_PORT     = "9428"
    KNOT_LOG_SINK_FORMAT   = "vl"
  }
}

Start a space from the template — once the agent registers, your other spaces’ logs start flowing in. Query them by source space or service in the VictoriaLogs web UI on the space’s port 9428:

service="web"            -- all logs from the "web" service, any space
space_id="3f1b..."       -- everything from one space

Any other image works too — point KNOT_LOG_SINK_PORT at any log service that accepts one of the four formats (Loki, Graylog, a custom receiver…).

What Happens, Step by Step

1. Every space’s agent sends its log records to all knot servers in its zone.
2. Each server therefore sees every record, but only the zone leader acts on mirroring — it is the one server that forwards each record to the sinks, so every sink receives exactly one copy of each record, no matter how many servers there are.
3. The leader checks which sinks belong to the same user as the space that produced the record (and skips the sink’s own space). Matching records are batched and pushed to each sink space’s agent, which writes them to the local log service on KNOT_LOG_SINK_PORT in the configured format, retrying transient failures.
4. When a sink space stops or crashes, its agent connection drops — the knot servers detect this immediately, drop any records still waiting in that sink’s buffers, and remove the sink registration. Nothing is spooled to disk. When the space starts again the agent re-registers and mirroring resumes from that point (records emitted while the sink was down are not replayed). Register and deregister appear in the audit log.

Mirrored records are tagged with the source space (space_id and space_name), the space owner (user), service and level, so you can filter by space, owner or service in queries. These tags take precedence: a structured field logged by the application with the same name (service, user, …) is dropped from the mirrored copy rather than overwriting the origin tags. Multiple sinks per user are allowed — each receives its own copy.