Skip to content

feat(log-management): ship container logs to external providers via Vector - #5408

Closed
narcisonunez wants to merge 11 commits into
canaryfrom
feat/log-management
Closed

narcisonunez wants to merge 11 commits into
canaryfrom
feat/log-management

Conversation

@narcisonunez

@narcisonunez narcisonunez commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a Log Management feature (free tier, like Vault's secrets providers): Dokploy can ship a server's container logs to an external provider for retention/search/alerting, without replacing the existing live WebSocket log viewer.

  • Runs Vector (Apache-2.0, single binary) as one agent per registered server, plus an optional local agent on the machine Dokploy itself runs on (self-hosted only). Dokploy never implements per-provider HTTP clients — it only translates saved credentials into a Vector sink config.
  • Providers: Grafana Loki, Datadog Logs, Better Stack, Elasticsearch/ OpenSearch, Splunk HEC, AWS CloudWatch Logs — via a LogProviderAdapter registry (packages/server/src/services/log-management/providers), mirroring the existing vaultProvider client-registry pattern. Adding a provider needs no router/UI/orchestrator changes, see docs/log-management.md.
  • Events are tagged with dokploy_organization/project/environment/application via a generated VRL scoping transform, built from the org's project tree and resynced on every log-management change plus every 15 minutes.
  • Two independent toggles gate whether Vector runs at all (per server, and for the local self-hosted host, which a single org "claims" to avoid cross-org credential leakage on a shared instance).
  • Generated vector.yaml is validated (vector validate) before being applied, with secrets redacted from any validation error; writes are serialized per server (in-process queue + a Postgres advisory lock for horizontally scaled Cloud) and applied via a write-then-rename to avoid partial configs.
  • One invalid provider is isolated at config-build time and skipped instead of breaking log shipping for the rest of the org; cross-field constraints an adapter can't express statically (e.g. Elasticsearch requiring a password whenever a username is set) are rejected at save time too.
  • testConnection is permissioned and timeout-bounded like the comparable vaultProvider/registry test-connection paths.

See docs/log-management.md for the full design, known limits (single-node Swarm servers only, no embedded historical log viewer, local-host caveats on a shared self-hosted instance), and the guide for adding a new provider.

What is this PR about?

Please describe in a short paragraph what this PR is about.

Checklist

Before submitting this PR, please make sure that:

  • You created a dedicated branch based on the canary branch.
  • You have read the suggestions in the CONTRIBUTING.md file https://github.com/Dokploy/dokploy/blob/canary/CONTRIBUTING.md#pull-request
  • You have tested this PR in your local instance. If you have not tested it yet, please do so before submitting. This helps avoid wasting maintainers' time reviewing code that has not been verified by you.

Issues related (if applicable)

Screenshots (if applicable)

Screenshot 2026-09-07 at 10 32 45 PM Screenshot 2026-09-07 at 11 10 01 PM Screenshot 2026-09-07 at 11 37 02 PM Screenshot 2026-09-08 at 6 34 48 PM

RetriggerConfidence Score: 4/5

The provider-deletion race still needs fixing before merging because deployment can restore a deleted selection and block later redeploys.

What we checked:

  • Missing agents look deployed: inspectVectorStatus returns stopped when service inspection returns 404. The page treats that status as not deployed.

Summary

Adds provider settings and per-server Vector agents to ship container logs to six external providers without replacing the live log viewer.

  • Organizations can save destinations for their container logs.

Reviews (9) · Last reviewed commit: "fix(log-management): report missing agen..." · Reviewed by Greptile

@narcisonunez
narcisonunez marked this pull request as ready for review September 9, 2026 17:21
Comment thread apps/dokploy/server/api/routers/log-provider.ts
Comment thread apps/dokploy/server/api/routers/settings.ts Outdated
Comment thread packages/server/src/db/schema/web-server-settings.ts
Comment thread apps/dokploy/server/api/routers/server.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts Outdated
Comment thread apps/dokploy/server/api/routers/server.ts Outdated
Comment thread apps/dokploy/server/api/routers/server.ts Outdated
Comment thread apps/dokploy/server/api/routers/server.ts Outdated
Comment thread apps/dokploy/server/api/routers/log-provider.ts Outdated
Comment thread apps/dokploy/server/api/routers/log-provider.ts Outdated
autofix-ci Bot and others added 6 commits September 10, 2026 16:53
- Updated API link conditions to allow skipping batch processing based on context.
- Removed unused VECTOR_RESYNC_CRON_JOB constant from server constants.
- Introduced logProviderIds field in server and webServerSettings schemas to manage log providers.
- Added organization service to handle organization deletions and vector agent removals.
- Enhanced server service to find servers with log management enabled and update log providers.
- Updated vector setup to sync vector configurations based on organization and server log provider IDs.
- Implemented Docker labels for resources to improve tracking and management.
- Refactored vector agent setup and removal processes to streamline operations.
@Siumauricio

Copy link
Copy Markdown
Contributor

@greptile review

Comment thread apps/dokploy/server/api/routers/log-provider.ts
Comment thread apps/dokploy/server/api/routers/log-provider.ts
Comment thread packages/server/src/services/log-management/vector-agent.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts
Comment thread packages/server/src/setup/vector-setup.ts Outdated
Comment thread docs/log-management.md Outdated
Comment thread packages/server/src/services/log-management/vector-agent.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts
Comment thread packages/server/src/services/log-management/vector-agent.ts Outdated
- Check accessedServers on deploy, removal and the server-status list
- Keep the local claim until the agent is removed and serialize deploy/remove per target
- Persist provider assignments only after the Vector deploy succeeds; release the local claim if the first deploy fails
- Remove the agent on server/organization deletion even when no providers remain assigned
- Report agents with no running task as not running
- Refresh logProvider.one after saving, allow clearing saved credentials, keep deployed targets visible with no providers
Comment thread packages/server/src/services/log-management/vector-agent.ts Outdated
Comment thread packages/server/src/setup/vector-setup.ts Outdated
…eleted providers on save

- Inspect the dokploy-vector service before listing tasks; listTasks returns [] for a missing service
- Filter provider assignments to existing providers after a deploy so a provider deleted mid-deploy is not written back
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants