Publish module metrics and module_id labels

Applications need a simple way to publish their own metrics and to raise alerts when one of their services stops, without running an extra exporter or opening a new port. This issue covers only the core and metrics module parts of #8093, so they can be tested and released before the NethVoice part.

Core: a module can ship an export-metrics command. Every 5 minutes a timer on each node runs it and publishes the output through the node exporter, with a module="<id>" label added to each sample.
Metrics module: Prometheus copies the module label to module_id on node series, so module-provided alert rules (feat(alerts): allow module-provided rules ns8-metrics#91) match per-module container, volume and custom metrics. Core containers (module="core") and unattributed ones (module="unknown") are left unchanged.

How it works

A module command export-metrics is run every 5 minutes, and its output appears in the node exporter with the module label.
A failing command keeps the last output and logs the failure; a removed command makes its series disappear at the next run.
Invalid output of one module does not affect the others.
In Prometheus, ns8_container_info{module_id="metrics1"} returns the module containers, and ns8_container_info{module="core",module_id!=""} returns nothing.
Existing installations keep working after the core and metrics updates.

References

Parent issue: NethVoice: Monitor service availability #8093
Core: feat(metrics): publish module metric files ns8-core#1365
Metrics: feat(prometheus): add module_id to node series ns8-metrics#105

Testing versions: core:3.24.0-dev.2, metrics:1.7.0-dev.1

Clean install:

curl https://raw.githubusercontent.com/NethServer/ns8-core/main/core/install.sh > install.sh
bash install.sh ghcr.io/nethserver/core:latest ghcr.io/nethserver/core:3.24.0-dev.2
add-module ghcr.io/nethserver/metrics:1.7.0-dev.1 1

Update:

update-core ghcr.io/nethserver/core:3.24.0-dev.2
update-module ghcr.io/nethserver/metrics:1.7.0-dev.1 metrics1

Test case 1: clean install

Install core and metrics, then add an OpenLDAP domain (it creates openldap1) and an external LDAP domain (it creates ldapproxy1).
Open Grafana and check the Containers dashboard contains data.

Test case 2: update from the current stable versions

On a cluster with stable core and metrics and some apps installed, update core first, then metrics.
Check that the Grafana dashboards still show data and that existing alerts still arrive.

Test case 3: an app publishes its own metric

Add a test command to OpenLDAP:

runagent -m openldap1 sh -c 'printf "#!/bin/sh\necho demo_value 42\n" > ../bin/export-metrics && chmod +x ../bin/export-metrics'

Wait up to 5 minutes, or run runagent -m node refresh-module-metrics.
Run curl -s localhost:9100/metrics | grep demo_value. It shows demo_value{module="openldap1"} 42
In Grafana, open Explore and search demo_value. The value is 42 and carries module_id="openldap1" or do a query using Prometheus UI

Test case 4: the app command fails

Change the command so it prints demo_value 99 and exits with an error (exit 1). After the next run, the value stays 42, and journalctl -u refresh-module-metrics shows a warning for openldap1.
Change the command so it waits 60 seconds. The run stops after 30 seconds, the old value stays, and the log says the command timed out.
Change the command so it prints its own module label, for example demo_value{module="other"} 5. The output is refused, the old value stays, and the log says the label is reserved.

Test case 5: bad output from one app does not affect the others

Add a valid command to ldapproxy1 that prints demo_proxy 7.
Change the OpenLDAP command so it prints this is not valid.
After the next run, demo_proxy{module="ldapproxy1"} 7 is still published, together with all the normal node metrics. Only the OpenLDAP value is missing.
Fix the OpenLDAP command so it prints demo_value 43. After the next run, the value is 43.

Test case 6: removing the command

Delete the command from both apps: runagent -m openldap1 rm -f ../bin/export-metrics and the same for ldapproxy1.
After the next run, demo_value and demo_proxy no longer appear.

QA passed on two Rocky 9 nodes with core:3.24.0-dev.2 and metrics:1.7.0-dev.1, clean install and update from 3.23.2. All test cases are fine and the Containers dashboard shows data.

set verified

Navigation Menu

Repository navigation

Publish module metrics and module_id labels #8269

How it works

References

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Publish module metrics and module_id labels #8269

Description

How it works

References

Activity

gsanchietti commented on Oct 22, 2026

stephdl commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions