# journald

Collect logs from systemd's journal into Kopai using the OpenTelemetry Collector

# Integration

Ship logs from any [systemd](https://systemd.io) service on Linux to Kopai using the OpenTelemetry Collector's [native journald receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver). The collector reads binary journal entries directly — no file tailing, no parsing, no log rotation to manage. It picks up all journald metadata (`_SYSTEMD_UNIT`, `_BOOT_ID`, `_MACHINE_ID`, and dozens more) automatically, and cursor-based checkpointing guarantees no duplicates or loss across collector restarts.

Your application needs **no code changes**. If it already logs to stdout under a systemd unit, journald is already capturing it.

## Prerequisites

- Linux with systemd
- [`otelcol-contrib`](https://github.com/open-telemetry/opentelemetry-collector-releases) (the contrib distribution includes the journald receiver)
- Kopai running locally:

**npm:**

```bash
npx @kopai/app start
```

**pnpm:**

```bash
pnpm dlx @kopai/app start
```

**yarn:**

```bash
yarn dlx @kopai/app start
```

## Journal access

The `journald` receiver runs `journalctl` under the hood, so the process running the collector must be able to read `/var/log/journal`. If it can't, the collector will start, log `Everything is ready`, and then silently ingest no records — there is no error message.

Pick one of the following before starting the collector:

- **Run the collector as root** (simplest for a one-off test):

  ```bash
  sudo otelcol-contrib --config otel-collector/journald.yaml
  ```

- **Add your user to the `systemd-journal` group**, then log out and back in (group membership only takes effect in a new login session):

  ```bash
  sudo usermod -aG systemd-journal "$USER"
  # then log out and log back in, or start a new login shell
  ```

- **Run the collector as a systemd service** with `Group=systemd-journal` in its unit file — recommended for production.

To confirm the collector can see the journal before launching it, run this as the user that will run the collector:

```bash
journalctl -n 1 --no-pager
```

If you see `-- No entries --` or a "you are currently not seeing messages from other users" hint, the collector won't see them either.

## Configure the collector

Point the `journald` receiver at your journal directory and filter by the systemd units you care about. Create `otel-collector/journald.yaml`:

```yaml
receivers:
  journald:
    directory: /var/log/journal
    units:
      - my-service.service
    priority: info
    operators:
      - type: severity_parser
        parse_from: body.PRIORITY
        mapping:
          error: "3"
          warn: "4"
          info: "6"
          debug: "7"
      - type: move
        from: body.MESSAGE
        to: body

processors:
  batch: {}
  resource:
    attributes:
      - key: service.name
        value: "my-service"
        action: insert

exporters:
  otlphttp/kopai:
    endpoint: http://localhost:4318
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [resource, batch]
      exporters: [otlphttp/kopai]
```

Replace `my-service.service` with the systemd unit you actually want to ship to Kopai (e.g. `nginx.service`, `myapp.service`). To tail the entire journal instead of a single unit, omit the `units:` key entirely. The `severity_parser` maps journald `PRIORITY` values (3=error, 4=warning, 6=info, 7=debug) to OTel severity levels.

## Run the collector

```bash
otelcol-contrib --config otel-collector/journald.yaml
```

The collector uses journald's native cursor mechanism to track its read position, so it resumes from the last entry after a restart — no duplicates, no gaps.

:::note
You may see an error-level line at startup mentioning a journal file that "is truncated, ignoring file", possibly with a stack trace. This comes from `journalctl` itself — it's skipping a single corrupted journal file — and the receiver continues normally with the rest of the journal. You can safely ignore it as long as the collector reaches `Everything is ready`.
:::

## Verify

Confirm logs arrived using the Kopai CLI:

**npm:**

```bash
# Recent entries with timestamps
npx @kopai/cli logs search --service my-service \
  --fields Timestamp,Body --sort ASC

# Filter to errors only (severity-min 17 = ERROR)
npx @kopai/cli logs search --service my-service --severity-min 17 --json
```

**pnpm:**

```bash
# Recent entries with timestamps
pnpm dlx @kopai/cli logs search --service my-service \
  --fields Timestamp,Body --sort ASC

# Filter to errors only (severity-min 17 = ERROR)
pnpm dlx @kopai/cli logs search --service my-service --severity-min 17 --json
```

**yarn:**

```bash
# Recent entries with timestamps
yarn dlx @kopai/cli logs search --service my-service \
  --fields Timestamp,Body --sort ASC

# Filter to errors only (severity-min 17 = ERROR)
yarn dlx @kopai/cli logs search --service my-service --severity-min 17 --json
```

## Sending to Kopai.app in the cloud

Swap the exporter endpoint and add a bearer token:

```yaml
exporters:
  otlphttp/kopai:
    endpoint: https://otlp-http.kopai.app
    headers:
      authorization: "Bearer YOUR_BACKEND_TOKEN"
```

Drop the `tls: insecure: true` block — the cloud endpoint uses real TLS.

## Working Example

For a complete runnable example with a demo systemd unit and nix dev shell:

**[Journald Example](https://github.com/kopai-app/kopai-integration-examples/tree/main/journald/python-native)**