Skip to content

feat(logs-and-metrics): ship container logs and metrics to external providers via Vector - #5624

Merged
imrja8 merged 1 commit into
canaryfrom
feat/logs-and-metrics
Oct 14, 2026
Merged

imrja8 merged 1 commit into
canaryfrom
feat/logs-and-metrics

Conversation

@narcisonunez

@narcisonunez narcisonunez commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Supersedes #5408. Same code base as Log Management, renamed to Logs & Metrics, with metrics added, a single migration and a redesigned agent lifecycle. Docs: Dokploy/website#197 (supersedes Dokploy/website#196).

Adds a Logs & Metrics feature (free tier): Dokploy ships a server's container logs and its host and container metrics to external providers for retention, search, dashboards and alerting. It doesn't replace the live log viewer or the Monitoring page.

  • Runs Vector as one agent per server (dokploy-vector, pinned to timberio/vector:0.58.0-alpine), plus the local agent on the Dokploy host (self-hosted only). Dokploy has no shipping code of its own: a provider is saved credentials translated into Vector sinks.
  • Nine provider types: Grafana Loki (logs); Prometheus remote write, covering Prometheus, VictoriaMetrics, Grafana Mimir and Grafana Cloud (metrics); New Relic (metrics); InfluxDB 2.x/3.x (metrics); Datadog, AWS CloudWatch, Splunk HEC, Elasticsearch/OpenSearch and Better Stack (logs and metrics). Each provider picks what it ships in a required Sends field (Logs, Metrics or Logs & Metrics, depending on the type), and one set of credentials covers both signals.
  • One Servers section: each server card lists every provider as a checkbox labelled with what it sends (Loki prod (logs), Splunk (metrics), Datadog (logs and metrics)). One Deploy applies the selection; Remove stops the agent on that server.
  • Metrics come from Vector's host_metrics plus a prometheus_scrape of a cAdvisor sidecar (dokploy-cadvisor, ghcr.io/google/cadvisor:v0.60.6, listening on 127.0.0.1:4510). cAdvisor runs only on hosts with a metrics provider. Logs and metrics carry dokploy_* tags (organization, project, environment, application, service, server) built from the dokploy.* labels that applications, compose services and databases now get.
  • Desired state plus reconcile: Deploy/Remove save the server's selection (telemetryProviderIds) and reconcileVectorAgent makes the host match the DB: it builds, validates and applies the YAML, deploys Vector, and adds or removes cAdvisor. With nothing left to ship it removes the agent. Editing, disabling or deleting a provider re-applies every server that uses it. A failed deploy keeps the selection saved and the previous config running, shows the redacted error on the card, and Redeploy retries. There is no rollback on purpose: the DB is the source of truth (this replaces the deploy/remove layer Greptile flagged on feat(log-management): ship container logs to external providers via Vector #5408).
  • The local Dokploy host has a single owner organization (vectorAgentOrganizationId), taken and released atomically. Other organizations get a conflict and don't see the local host. Agents on the local host only ship the owner's containers.
  • The generated vector.yaml is validated (vector validate) before it is applied, with secrets redacted from any error. Reconciles of the same host are serialized, and a config-only Redeploy reloads without restarting the task.
  • Connection tests go through an SSRF-guarded fetch (metadata always blocked; private and loopback addresses blocked on Cloud) with a 15 s timeout. The CloudWatch metrics test uses PutMetricData, the only way to check cloudwatch:PutMetricData (new dependency @aws-sdk/client-cloudwatch).
  • Deleting a server, an organization or an account removes their agents.

One migration (0197_logs_and_metrics). Nothing from #5408 reached production, so the rename needs no data migration.

What is this PR about?

Ship container logs and host/container metrics from Dokploy servers to external providers (Loki, Prometheus/Mimir/VictoriaMetrics/Grafana Cloud, New Relic, InfluxDB, Datadog, CloudWatch, Splunk, Elasticsearch/OpenSearch, Better Stack) through a Vector agent managed by Dokploy. It replaces #5408.

Checklist

Before submitting this PR, please make sure that:

  • You created a dedicated branch based on the canary branch.
  • You have read the suggestions in the CONTRIBUTING.md file https://github.com/Dokploy/dokploy/blob/canary/CONTRIBUTING.md#pull-request
  • You have tested this PR in your local instance. If you have not tested it yet, please do so before submitting. This helps avoid wasting maintainers' time reviewing code that has not been verified by you.

Tested locally (Docker Desktop, Swarm, dev server):

  • Loki, Prometheus, VictoriaMetrics, Mimir, InfluxDB 2, Elasticsearch 8.15, OpenSearch 2 and Splunk receive host and container data with the dokploy_* tags.
  • A single Deploy with a mixed selection ships to every backend.
  • Unchecking every metrics provider removes cAdvisor while logs keep flowing.
  • Disabling or deleting a provider re-applies the agent.
  • A password containing $ works.
  • A failed deploy shows the redacted error and Redeploy recovers.
  • Remove cleans up.
  • A second organization gets a conflict on the local host.
  • A compose service in a project named with $ deploys with literal labels.

vector validate runs in a test against the generated configs for every provider type. Grafana Cloud, New Relic, Datadog, AWS CloudWatch and Better Stack still need testing with real accounts.

Issues related (if applicable)

Supersedes #5408.

Screenshots (if applicable)

Screenshot 2026-10-08 at 9 54 29 AM Screenshot 2026-10-08 at 9 54 56 AM Screenshot 2026-10-08 at 9 55 03 AM Screenshot 2026-10-08 at 9 55 20 AM Screenshot 2026-10-08 at 9 55 30 AM

RetriggerConfidence Score: 5/5

The latest changes appear safe to merge; no new actionable issue was established.

What we checked:

  • DNS changes bypass the guard: guardedLookup checks the addresses returned to the socket, rather than doing a separate lookup before the request.

Summary

This PR adds Logs & Metrics shipping through Vector, with provider settings, server selections, container labels, and agent cleanup.

  • The latest changes address all four previous findings.
  • Connection tests now check DNS addresses when connecting.
  • Provider deletion gets affected hosts inside its transaction.
  • Cleanup no longer assumes an empty selection means no agent exists.

Reviews (2) · Last reviewed commit: "feat(logs-and-metrics): ship container l..." · Reviewed by Greptile

Comment thread packages/server/src/services/logs-and-metrics/provider-fetch.ts Outdated
Comment thread packages/server/src/services/logs-and-metrics/service.ts Outdated
Comment thread packages/server/src/services/logs-and-metrics/vector-agent.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts Outdated
@narcisonunez
narcisonunez force-pushed the feat/logs-and-metrics branch from 876896a to b329595 Compare October 8, 2026 14:08
@imrja8
imrja8 merged commit a7d8dff into canary Oct 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants