Repository navigation
Fix the Redis instance average command time staying near zero - #14258
Open
Tanishq-Choudhary wants to merge 1 commit into
Open
Tanishq-Choudhary wants to merge 1 commit into
Tanishq-Choudhary wants to merge 1 commit into
Conversation
meter_redis_instance_average_time_spent_by_command divided the lifetime
counters redis_commands_duration_seconds_total and redis_commands_total
and then took rate('PT1M') of the ratio. That is how fast the lifetime
average drifts, close to 0, not the average time of a command.
Divide the per-minute rate of the command time by the per-minute rate
of the command count, with safeDiv for commands idle in the window.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the Redis instance
meter_redis_instance_average_time_spent_by_commandstaying near zeroThe instance rule divided the two lifetime counters from redis_exporter first and then took
rate('PT1M')of the ratio:That ratio is the lifetime average time per command. Its per-second rate is how fast that lifetime average drifts, which is close to 0 on any Redis that has been up for a while, and can go negative. The "Avg Command Time" panel of the Redis instance dashboard reads this metric, so it shows about 0 for every command.
The rule now takes the rate of each counter and divides them, the same shape the BanyanDB rules use for latency per call:
safeDivgives 0 for a command that did not run in the window, instead of NaN.The MAL script test expected 0.0 for this metric. Its input moves the duration and count counters by the same amount in 2 seconds (25/s each), so the average time per command is 1.0. The expectation is now 1.0; on master the test fails with
expected=1.0, actual=0.0.Values stored before the upgrade keep the old near-zero numbers, so the chart steps at the upgrade.
Update the
CHANGESlog.GraalVM distro impact:
instance_average_time_spent_by_commandexpression in otel-rules/redis/redis-instance.yaml; the distro compiles it as it is.