Repository navigation
feat(api): expose host system logs at /api/v2/system/logs - #3423
vpetersson-bot wants to merge 9 commits into
Conversation
Anthias had no way to read the host journal from inside the container. This adds a snapshot-only endpoint with one contract on both platforms. - Balena: supervisor POST /v2/journal-logs. No compose or privilege change -- the supervisor-api label is already present. Requests format=json so the payload matches bare metal and one normaliser serves both. - Bare metal: anthias-host-agent over hostcmd, with correlation IDs, line and byte caps, and a Redis TTL on replies. Bare-string hostcmd payloads still work, so rollout order does not matter. - Nothing in the path can block subscriber_loop: no follow on either platform, every read bounded by lines, bytes and a subprocess timeout, and every failure path still writes a reply. - Grant the host agent journal access with SupplementaryGroups=systemd-journal, and refuse the request outright when the system journal is unreadable -- journalctl exits 0 and returns the service account's own user journal otherwise. - Make journald persistent and size-capped in the Ansible role. - Redact secret-looking values (psk, passphrase, tokens) on the way out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
The SonarCloud gate failure comes from three
The new code introduces no actual vulnerabilities. CodeQL and the rest of CI pass, and the three CodeQL exception-exposure alerts were fixed in b069f54. Someone with SonarCloud access needs to mark these three findings as Safe to clear the gate. |
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
CodeQL flagged str(exc) flowing into API responses. JournalUnavailable and JournalRequestError now carry a composed user_message, which the view and serializer return instead. Supervisor and Redis transport errors are logged rather than interpolated into the response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
SonarCloud S2612 flagged the three new `mode:` values. All three match what the OS itself ships, so they are annotated rather than tightened -- but chasing the first one down found a real gap: creating /var/log/journal ourselves skips the `a+` tmpfiles line that grants the `adm` group its read ACL, so adm-based journal access silently would not work on a host where we made the directory. - Run `systemd-tmpfiles --create --prefix=/var/log/journal`, the other half of the recipe documented in journald.conf(5) - NOSONAR with reasons: systemd pins /var/log/journal to 2755 and resets anything narrower; 0755/0644 match /etc/systemd's own files Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
- redact(): the key=value pattern was quadratic. A 48 KB line (journald's default LineMax) took ~a minute and pinned a request worker; measured 6.3s at 16 KB. A lookbehind makes only a real token start a match, taking it back to linear -- 48 KB now 0.003s. - Balena truncation kept the FIRST 4 MB while the host agent kept the last. journalctl emits oldest-first, so the supervisor path was discarding exactly the entries an operator opened the log to read. Trim from the front, and bound total bytes read as well as kept. - `container` means a name on both platforms, but the supervisor only matches CONTAINER_ID_FULL, so forwarding the name returned an empty 200 on Balena. Resolve it via /v2/containerId, pass an ID straight through, and fail loudly on an unknown service. - A stopped host agent made every request block for the full 30s BLPOP. PUBLISH reports its subscriber count; zero means no reply is coming, so fail now. - journalctl output was decoded with `text=True`, i.e. the locale encoding with strict errors -- and a systemd unit has no LANG, so that is ASCII. One non-ASCII byte would raise out of process_message and kill subscriber_loop, the exact failure this feature is built to avoid. - count help_text claimed "most recent first"; entries are oldest first. It is published in the OpenAPI schema. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
They were sitting in the comment block above each `mode:`, where the analyzer does not see them -- SonarCloud kept reporting all three. Inline is also what the working examples in roles/system/tasks use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
388697b to
84190b8
Compare
Translate supervisor body-read failures into the existing 502 response and always close the response. Request full journal fields on bare metal while draining stdout and stderr into bounded buffers under one subprocess deadline; kill and reap the child on failure. Add regressions for streaming failures, long messages, output caps, UTF-8 decoding, and timeout cleanup. Existing redaction performance and newest-entry truncation regressions also pass. Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
`host_agent.subprocess` is an implicit re-export, which mypy rejects under no_implicit_reexport -- it broke run-mypy on the branch. It is the same module object as the stdlib `subprocess` the test already imports, and _run_journalctl resolves `subprocess.Popen` through its own globals, so patching the module directly is identical in effect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
Issues Fixed
Anthias has no way to surface host/system logs.
bin/collect_debug.shshells out tojournalctlon bare metal, but that is a manual operator script — nothing in the app or API can read the journal.Description
Adds
GET /api/v2/system/logs: a snapshot-only journal read with one contract on both supported platforms. Nofollowon either — the bare-metal backend runs inside the host agent's single-threadedsubscriber_loop, where a streaming read would queue reboot/shutdown behind it forever, and offering a live tail on Balena alone would mean two divergent contracts.Parameters:
count(default 500, hard cap 2000),unit,container,since,until.Balena — supervisor
POST /v2/journal-logs. No compose change and no new privilege;anthias-serveralready carriesio.balena.features.supervisor-api. Requestsformat: jsonso the payload is shaped identically to bare metal and one normaliser serves both backends.Bare metal —
anthias-host-agentoverhostcmd, now accepting a JSON payload with a correlation ID alongside the original bare-string commands (so rollout order does not matter). The reply goes onto a correlation-keyed Redis list with a TTL and is collected with a boundedBLPOP— a blocking pop rather than a completion pub/sub, because it has no subscribe-before-publish race and matches theanthias.reply.<id>pattern already used on the viewer bus.Nothing in the path can block
subscriber_loop: bounded by lines, by bytes and by a subprocess timeout, and every failure path still writes exactly one reply so the container's pop resolves instead of timing out.Also here:
SupplementaryGroups=systemd-journalon the host-agent unit, plus a hard refusal when the system journal is unreadable.journalctlexits 0 for a caller outside that group and hands back the service account's own user journal with every system record missing — and under--output=jsonit does not even print its stderr hint, so there is no signal to key off. The agent asks the kernel (os.access) instead, which also honours the ACL journald grants toadm.99-anthias.confso it wins over Raspberry Pi OS's40-rpi-volatile-storage.conf, and followed byjournalctl --flushbecause a restart alone leaves the journal in/rununtil the next boot.persistentis reported in the response rather than assumed, measured from which of/var/log/journaland/run/log/journalholds the live journal. Both cheaper probes give a wrong "yes" on hardware we ship on.psk,passphrase, tokens,Authorization) are blanked on the way out, keeping the key visible.Note for operators:
SystemMaxUse=64Mis enforced by vacuuming, so on an install whose journal already exceeds it the first journald restart discards the excess. A QA x86 board had grown to 1.8 GB. Overrideanthias_journal_max_usewhere the longer history matters more than the card.Checklist
End-to-end on the QA rig (Pi 5 / aarch64 and x86, both Debian 13), host agent and server overlaid from this commit with hashes verified in place: snapshot, every filter, parameter rejection, five concurrent requests each getting their own result, reply TTL, 2000-line read in 0.49s / 0.80s with the next host command answered ~250ms later, and redaction of a real
logger-injected line. Both boards restored to their pinned images afterwards.Verified against the real OS permission state on both boards: with the supplementary group withheld,
journalctl -u docker.servicereturns 0 lines and exit code 0; with it, 50. The agent now refuses that case with an actionable error instead of serving a silently empty view.The Balena backend was exercised over real HTTP inside the server container against a supervisor stub — correct URL, version and API key, correct body,
follow: false, identical normalised output, redaction, and the 4 MiB cap holding on a streamed 9 MB response. It has not been run against a real balenaOS supervisor; the rig has no balena device.🤖 Generated with Claude Code
Review follow-ups
Six findings from a review pass, all fixed and re-verified on the Pi 5 and x86 boards:
redact()was quadratic. A 48 KB line (journald's defaultLineMax) took about a minute and pinned a request worker; measured 6.3 s at 16 KB. A lookbehind means only a real token start can begin a match — 48 KB is now 0.003 s, with every redaction case unchanged.journalctlemits oldest-first, so the supervisor backend was discarding exactly the entries an operator opened the log to read. It now trims from the front, and bounds total bytes read as well as bytes kept.containerreturned an empty 200 on Balena. It means a name on both platforms, but the supervisor only matchesCONTAINER_ID_FULL. It is now resolved via/v2/containerId, an ID is passed straight through, and an unknown service fails loudly instead of looking like a quiet device.BLPOP.PUBLISHreports its subscriber count; zero means no reply is coming. Now 502s in ~0.02 s.text=Truedecoded journalctl output with the locale encoding and strict errors — and a systemd unit has noLANG, so that is ASCII. One non-ASCII byte would have raised out ofprocess_messageand killedsubscriber_loop, the exact failure this feature is built to avoid.counthelp_text claimed "most recent first"; entries are oldest-first. It is published in the OpenAPI schema.Chasing the first SonarCloud finding also turned up a real gap: creating
/var/log/journalourselves skipped thea+tmpfiles line that grants theadmgroup its read ACL, so adm-based journal access would silently not have worked on a host where we made the directory. The role now runssystemd-tmpfiles --create, the other half of the recipe documented injournald.conf(5).