Skip to content

Fix the Redis instance average command time staying near zero - #14258

Open
Tanishq-Choudhary wants to merge 1 commit into
apache:masterfrom
Tanishq-Choudhary:fix-redis-avg-command-time
Open

Tanishq-Choudhary wants to merge 1 commit into
apache:masterfrom
Tanishq-Choudhary:fix-redis-avg-command-time

Conversation

@Tanishq-Choudhary

Copy link
Copy Markdown

Fix the Redis instance meter_redis_instance_average_time_spent_by_command staying near zero

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

The instance rule divided the two lifetime counters from redis_exporter first and then took rate('PT1M') of the ratio:

(redis_commands_duration_seconds_total.sum([...]) / redis_commands_total.sum([...])).rate('PT1M')

That ratio is the lifetime average time per command. Its per-second rate is how fast that lifetime average drifts, which is close to 0 on any Redis that has been up for a while, and can go negative. The "Avg Command Time" panel of the Redis instance dashboard reads this metric, so it shows about 0 for every command.

The rule now takes the rate of each counter and divides them, the same shape the BanyanDB rules use for latency per call:

redis_commands_duration_seconds_total.sum([...]).rate('PT1M').safeDiv(redis_commands_total.sum([...]).rate('PT1M'))

safeDiv gives 0 for a command that did not run in the window, instead of NaN.

The MAL script test expected 0.0 for this metric. Its input moves the duration and count counters by the same amount in 2 seconds (25/s each), so the average time per command is 1.0. The expectation is now 1.0; on master the test fails with expected=1.0, actual=0.0.

Values stored before the upgrade keep the old near-zero numbers, so the chart steps at the upgrade.

  • Update the CHANGES log.

  • GraalVM distro impact:

    • None.
    • Impacted, by: bundled rule files. Edits the instance_average_time_spent_by_command expression in otel-rules/redis/redis-instance.yaml; the distro compiles it as it is.

meter_redis_instance_average_time_spent_by_command divided the lifetime
counters redis_commands_duration_seconds_total and redis_commands_total
and then took rate('PT1M') of the ratio. That is how fast the lifetime
average drifts, close to 0, not the average time of a command.

Divide the per-minute rate of the command time by the per-minute rate
of the command count, with safeDiv for commands idle in the window.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant