Allow applications to provide their own alert rules

NethServer 8 applications can provide monitoring data and Grafana dashboards, but they cannot provide their own Prometheus alert rules through a supported integration.

Application-specific rules must currently be maintained in the Metrics module or added manually to generated configuration files. This ties application changes to Metrics updates, and manual changes can be lost when configuration files are regenerated.

Applications should be able to provide and maintain the alert rules needed to monitor their own services.

Proposed solution

Allow each application instance to publish its alert rules and notify Metrics when they change. Metrics should load these rules into Prometheus and automatically apply additions, updates, and removals. A published source may contain one alert or a grouped rule file.

The integration should meet these requirements:

Each application's rules evaluate only monitoring data belonging to that instance.
Invalid rule content does not interrupt monitoring or affect rules from other applications. If an update is invalid, the previous valid rules remain active.
Removed rules are cleaned up without affecting other applications.
Each published source is validated and replaces its generated file in one operation. Updates to different sources are independent.
Metrics checks that Prometheus has loaded the resulting configuration successfully.
Existing built-in rules, custom alerts, and alert delivery continue to work.

If Prometheus rejects a reload and verification does not recover, Metrics reports an error and leaves the running service in place. Reload-command failures or inconclusive verification use a checked restart fallback.

If the infrastructure needed to prepare application rules fails, Metrics logs a warning and lets monitoring start or reload with the rules available at that point. Some application rules may be stale or incomplete. Files already replaced are not rolled back.

Alternative solutions

Maintain all application-specific rules in Metrics. This requires updating Metrics whenever an application needs to change its rules.

Use the existing custom_alerts mechanism or edit generated configuration files manually. Custom alerts are managed locally by Metrics and do not provide an integration for other applications. Manual file changes can be overwritten during configuration regeneration.

Additional context

Applications would publish rules through Redis, as they already do for other monitoring resources. Metrics would identify the publishing instance from its Redis key and enforce that identity when processing monitoring data, rules, and alerts.

Application-provided rules would be managed separately from built-in rules and existing custom alerts. Prometheus sends alerts directly to the local Alertmanager and also to my Mimir when the enterprise subscription provides the required settings. Critical email notifications remain handled by the local Alertmanager when configured.

Manual QA: module-provided alert rules

The examples target the ns8-traefik module instance, normally traefik1. Reuse the same rules throughout the sequence.

Prepare the QA session

Use a designated QA cluster upgraded to the candidate Metrics version, with Grafana enabled. Record the previous and candidate versions and image digest. Before adding fixtures, confirm that existing Grafana dashboards, built-in rules, and any existing custom alerts still work after the upgrade.

Run the commands as root on the cluster leader in the same Bash session. List the installed modules and set QA_METRICS to the actual Metrics instance ID. The assignment below uses metrics1 as an example. Discover the cluster's default ns8-traefik instance rather than assuming its ID.

runagent --list-modules

QA_METRICS="metrics1" # Replace with the installed Metrics instance ID.
QA_TRAEFIK="$(runagent redis-exec GET cluster/default_instance/traefik)"

QA_DIR="$(mktemp -d /tmp/qa8100.XXXXXX)"
QA_RUN="${QA_DIR##*/}"
QA_RUN="${QA_RUN//./_}"

QA_SINGLE="${QA_RUN}_single"
QA_GROUP="${QA_RUN}_group"
QA_BAD="${QA_RUN}_invalid"
QA_OTHER=""

printf 'Metrics: %s\nTraefik: %s\nQA files: %s\nRun: %s\n' \
    "$QA_METRICS" "$QA_TRAEFIK" "$QA_DIR" "$QA_RUN"

Confirm both module IDs before continuing. If Traefik discovery returns an empty value or (nil), resolve the actual instance ID first.

In Grafana Explore, select the local Prometheus data source and verify that this query returns samples, substituting the discovered Traefik ID:

up{module_id="traefik1"}

The second-module isolation check also requires another test module with scraped up series. Set QA_OTHER to its actual ID when reaching that check.

Use Grafana's Alert rules view with the local Prometheus data source to observe the rules. Prometheus-managed rules are read-only there, so all changes below use Redis and the module event. See the Grafana Prometheus alerting documentation.

Keep Traefik running because it also provides access to Grafana. The fixtures use real up selectors with deliberately false comparisons. Changing the comparison makes an alert fire without stopping the application. Metrics adds the publishing instance's module_id matcher and alert label automatically.

Each Redis hash field holds one complete YAML document. One file below contains a single rule, and the other contains two grouped rules. The unique run marker keeps their names and Redis fields separate from existing configuration.

Create the rule fixtures

Single-rule source: R1

cat > "$QA_DIR/single.yml" <<EOF
alert: QA_Single_${QA_RUN}
expr: sum(up) < 0
for: 0m
labels:
  severity: warning
annotations:
  summary_en: "NS8 module alert QA"
  summary_it: "Verifica alert del modulo NS8"
  description_en: "Single rule, version 1."
  description_it: "Regola singola, versione 1."
EOF

Grouped source: R2 and R3

cat > "$QA_DIR/group.yml" <<EOF
groups:
- name: qa-monitoring
  rules:
  - alert: QA_GroupOne_${QA_RUN}
    expr: max(up) < 0
    for: 0m
    labels:
      severity: warning
    annotations:
      summary_en: "First grouped QA rule"
      summary_it: "Prima regola QA del gruppo"
      description_en: "Grouped rule one, version 1."
      description_it: "Prima regola del gruppo, versione 1."
  - alert: QA_GroupTwo_${QA_RUN}
    expr: min(up) < 0
    for: 0m
    labels:
      severity: warning
    annotations:
      summary_en: "Second grouped QA rule"
      summary_it: "Seconda regola QA del gruppo"
      description_en: "Grouped rule two, version 1."
      description_it: "Seconda regola del gruppo, versione 1."
EOF

The up metric is non-negative, so these rules start non-firing. A comparison of >= 0 fires as long as the publishing module has scraped up series.

Define this convenience function once in the same shell:

qa_publish() {
    if [ "$#" -ne 3 ] || [ -z "${1:-}" ] || [ -z "${2:-}" ]; then
        printf 'Usage: qa_publish <module ID> <Redis field> <YAML file>\n' >&2
        return 2
    fi
    redis-cli -x HSET \
        "module/$1/metrics_alert_rules" "$2" < "$3" || return
    redis-cli PUBLISH \
        "module/$1/event/metrics-alert-rules-changed" '{}'
}

Its arguments are publisher module ID, Redis field name, and YAML file. HSET stores the document and PUBLISH announces the change. An HSET reply of 0 is normal when replacing an existing field. Redis replies do not prove that Prometheus loaded the rules, so use the observations below to judge each case. Allow up to three minutes for reload, evaluation, and Grafana refresh before investigating an unexpected result.

Test 1: publish the three rules

Publish the two source files:

qa_publish "$QA_TRAEFIK" "$QA_SINGLE" "$QA_DIR/single.yml"
qa_publish "$QA_TRAEFIK" "$QA_GROUP" "$QA_DIR/group.yml"

Filter Grafana's rule list by the run marker printed during preparation.

Expected: R1, R2, and R3 appear. R1 has its own generated group, while R2 and R3 share a group. Their names retain their spelling, and their effective expressions and labels identify the actual ns8-traefik instance. All three are initially non-firing, and existing monitoring remains available. No manual Prometheus or Grafana restart is needed.

Test 2: edit existing rules

Edit single.yml, changing its description to version 2. In group.yml, change only R2's description to version 2. Keep the expressions non-firing until Test 4 and preserve the files as the current valid versions.

Republish using the same Redis fields:

qa_publish "$QA_TRAEFIK" "$QA_SINGLE" "$QA_DIR/single.yml"
qa_publish "$QA_TRAEFIK" "$QA_GROUP" "$QA_DIR/group.yml"

Expected: the new descriptions appear automatically without duplicate rules. R3 remains at version 1, and existing monitoring continues working.

Test 3: reject a bad update and recover

Add a new source containing invalid YAML:

redis-cli HSET \
    "module/$QA_TRAEFIK/metrics_alert_rules" \
    "$QA_BAD" 'groups: ['

redis-cli PUBLISH \
    "module/$QA_TRAEFIK/event/metrics-alert-rules-changed" '{}'

Expected: no additional rule appears, and the three valid rules remain.

Make a separate invalid copy of the current grouped source:

cp "$QA_DIR/group.yml" "$QA_DIR/group-invalid.yml"

In group-invalid.yml, change R2's expression to expr: 'up{' and R3's description to This change must not appear. Publish the invalid copy under the existing group field:

qa_publish "$QA_TRAEFIK" "$QA_GROUP" "$QA_DIR/group-invalid.yml"

While the group is invalid, change R1's description in the valid single.yml to version 3 and publish it:

qa_publish "$QA_TRAEFIK" "$QA_SINGLE" "$QA_DIR/single.yml"

Expected: both grouped rules retain their previous valid versions. R3's proposed description does not appear, while R1's version 3 update does. The module journal identifies the rejected sources, and monitoring remains available.

Recover by editing the valid group.yml so R2's description is version 3, keeping both expressions valid and R3's description at version 1. Remove the new malformed source and republish the corrected group:

redis-cli HDEL \
    "module/$QA_TRAEFIK/metrics_alert_rules" "$QA_BAD"

qa_publish "$QA_TRAEFIK" "$QA_GROUP" "$QA_DIR/group.yml"

Expected: R2 now shows version 3 and R3 remains at version 1. The visible revision proves that corrected updates load again.

Test 4: raise and resolve alerts independently

Create a firing version of R1 while keeping single.yml as the valid, non-firing version:

sed 's/^expr:.*/expr: sum(up) >= 0/' \
    "$QA_DIR/single.yml" > "$QA_DIR/single-firing.yml"

For the isolation check, set QA_OTHER to another monitored NS8 module's actual ID. Enter it in the same interactive shell:

read -r -p 'Other monitored NS8 module ID: ' QA_OTHER

Verify in Grafana Explore that up{module_id="<that ID>"} returns samples and that it is a different module from QA_TRAEFIK. The same YAML file can be reused because Metrics scopes it to the publisher's own series.

Run the reference publication only when that prerequisite is met:

qa_publish "$QA_OTHER" "$QA_SINGLE" "$QA_DIR/single-firing.yml"

Raise the Traefik alert:

qa_publish "$QA_TRAEFIK" "$QA_SINGLE" "$QA_DIR/single-firing.yml"

Expected: both modules have firing alerts with the same name and distinct module_id labels. R2 and R3 remain non-firing.

Resolve only Traefik's alert:

qa_publish "$QA_TRAEFIK" "$QA_SINGLE" "$QA_DIR/single.yml"

Expected: Traefik's alert becomes non-firing while the reference alert remains firing. The rule definitions stay listed after recovery.

If the reference rule was published, resolve it too:

qa_publish "$QA_OTHER" "$QA_SINGLE" "$QA_DIR/single.yml"

If the second module is unavailable, record the Traefik firing/recovery observation and mark the isolation part as BLOCKED, rather than passing the complete case.

When notification delivery is included in the QA run, check actual firing and recovery receipt at the configured test destination. For email, set severity: critical in the firing copy before publication and use a configured QA recipient. The default warning fixtures do not exercise critical-alert email delivery. Allow the configured notification intervals. Record delivery as untested if no suitable receiver is available.

Test 5: verify persistence after restart

With all sources valid, restart Prometheus once:

runagent -m "$QA_METRICS" \
    systemctl --user restart prometheus.service

Wait for readiness and refresh Grafana without republishing any rules.

Expected: the three Traefik rules and, if published, the reference rule load automatically with their latest valid definitions. Existing built-in and custom rules remain available. This intentional restart is separate from the earlier publication and editing checks.

Test 6: remove rules and clean up

First, remove R2's complete rule entry from group.yml, leaving R3 in the group, and republish:

qa_publish "$QA_TRAEFIK" "$QA_GROUP" "$QA_DIR/group.yml"

Expected: only R2 disappears. R1, R3, and the reference module's rule remain.

Withdraw the Traefik QA sources:

redis-cli HDEL \
    "module/$QA_TRAEFIK/metrics_alert_rules" \
    "$QA_SINGLE" "$QA_GROUP" "$QA_BAD"

redis-cli PUBLISH \
    "module/$QA_TRAEFIK/event/metrics-alert-rules-changed" '{}'

Expected: no Traefik QA rule definitions remain, while the reference rule and existing monitoring are unaffected.

If the reference module was selected, remove only its QA field:

if [ -n "$QA_OTHER" ]; then
    redis-cli HDEL \
        "module/$QA_OTHER/metrics_alert_rules" "$QA_SINGLE"
    redis-cli PUBLISH \
        "module/$QA_OTHER/event/metrics-alert-rules-changed" '{}'
fi

Confirm that no rule carrying this run marker remains in Grafana, that any QA alerts have cleared, and that the original dashboards and rules still work. Use these withdrawal commands even when an earlier case fails. Remove only the QA fields, not the complete Redis hashes. Retain the local files in QA_DIR until evidence has been collected, then remove that run's temporary directory.

Navigation Menu

Repository navigation

Allow applications to provide their own alert rules #8100

Manual QA: module-provided alert rules

Prepare the QA session

Create the rule fixtures

Single-rule source: R1

Grouped source: R2 and R3

Test 1: publish the three rules

Test 2: edit existing rules

Test 3: reject a bad update and recover

Test 4: raise and resolve alerts independently

Test 5: verify persistence after restart

Test 6: remove rules and clean up

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow applications to provide their own alert rules #8100

Description

Activity

Amygos commented on Oct 13, 2026

Manual QA: module-provided alert rules

Prepare the QA session

Create the rule fixtures

Single-rule source: R1

Grouped source: R2 and R3

Test 1: publish the three rules

Test 2: edit existing rules

Test 3: reject a bad update and recover

Test 4: raise and resolve alerts independently

Test 5: verify persistence after restart

Test 6: remove rules and clean up

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions