[for minor release] RSS configurable thresholds by fstagni · Pull Request #8520 · DIRACGrid/DIRAC

fstagni · 2026-04-20T12:56:50Z

closes #8517

test in certification

BEGINRELEASENOTES

*RSS
CHANGE: Added configurable thresholds and fractions for FreeDiskSpace policy evaluation

*Deployment

CHANGE: for users of advanced RSS policies: the configuration is only partially backward compatible. The incompatibility is that the old policy type names (DTOngoing, DTScheduled*, FreeDiskSpaceTB/GB/MB, PropagationPolicy) were consolidated into Downtime, FreeDiskSpace and Propagation. Existing CS sections referencing the old names are silently skipped, so affected deployments must rename those sections/types and move hours/unit into the new args. Additionally, two default behaviors change with no CS change: the FreeDiskSpace policy now also applies fraction thresholds (1% Banned / 5% Degraded), so large SEs may flip to Degraded.

ENDRELEASENOTES

aldbr · 2026-06-11T08:11:15Z

ddev: PR is ready but changes the CS, so we need to make sure who is relying on that to see how smooth the integration could be. @fstagni will check with the known installations

ready to be reviewed

iueda · 2026-06-22T09:21:25Z

I have put a comment in #8517 (comment)

It would be nice if the thresholds can be set by absolute free space (eg. 0.1 TB) and by fraction (free/total, eg. 5%)

I was trying to compare this PR and our code, but decided to comment now, before spending too much time and coming too late.

fstagni · 2026-07-06T15:38:20Z

I have put a comment in #8517 (comment)

It would be nice if the thresholds can be set by absolute free space (eg. 0.1 TB) and by fraction (free/total, eg. 5%)

I was trying to compare this PR and our code, but decided to comment now, before spending too much time and coming too late.

@iueda I have added the "fraction", as IUUC should be in the BelleDIRAC implementation. Please review.

iueda

The code looks good (better than I was preparing).
Only minor suggestions on the comments and the tests.

There may be some remnants of my previous comments, which were in preparation and now obsolete (I tried to delete them, but somehow appears in the list of pending comments...)

chaen · 2026-10-08T17:06:15Z

I have a hard time understand whether this is backward compatible or not, mostly with respect to the config

…ully configurable

- AlwaysBannedForSite2: tests domain-based matching specificity - AlwaysDegradedForResource: tests AlwaysDegraded policy type - SpecificFreeDiskSpace: tests per-policy arg overrides (Unit=GB, custom thresholds) and specificity dedup against generic FreeDiskSpace - PropagationForSite: tests Propagation policy type - PolicyActions sections: LogStatusAction and LogPolicyResultAction - Integration test cases for LogSE WriteAccess (Degraded via specific FreeDiskSpace), LogSE ReadAccess (Active via AlwaysActive), and Site with None status

…es and generic specificity deduplication - Configurations.py: POLICIESMETA now holds plain code-level defaults only; Operations() calls removed (CS is read at runtime by InfoGetter instead) - InfoGetter.getPoliciesThatApply: CS entries without policyType are skipped (they are command-args defaults sections, not policy definitions); non-reserved keys in a CS policy entry are collected as arg overrides, merged into the policy args with case-insensitive key normalisation and type casting - InfoGetter.postProcessingPolicyList: replaces the old FreeDiskSpaceMB/GB/TB hack with a generic rule — when multiple policies of the same policyType match, keep the most specific one (name-match presence > number of matchParams keys) - dirac.cfg: add SpecificFreeDiskSpace example (Unit=GB, Banned_threshold=15, Degraded_threshold falls back to default); fix missing closing brace in Policies - docs: note that command-args sections have no policyType and are not policy definitions - test: new Test_InfoGetter.py with 15 unit tests covering all the above

…d defaults FreeDiskSpaceCommand and DowntimeCommand no longer duplicate default values in .get() fallbacks — defaults live exclusively in POLICIESMETA and are guaranteed to be present in self.args by the time the command runs.

- StateMachine.setState: guard against self.state being None before accessing self.states[self.state], preventing KeyError when the machine transitions from a valid state to None and back. - DowntimeCommand._prepareCommand: change key-existence checks to value checks (if not self.args.get("name")) so that None values from decisionParams defaults are also caught. - Add unit tests for both fixes and a new StateMachine test suite.

- AlwaysBannedForSite2: tests domain-based matching specificity - SpecificFreeDiskSpace: tests per-policy arg overrides (Unit=GB, custom thresholds) and specificity dedup against generic FreeDiskSpace - PropagationForSite: tests Propagation policy type - PolicyActions sections: LogStatusAction and LogPolicyResultAction - Integration test case for LogSE WriteAccess (Error via FreeDiskSpace command when no occupancy data exists in test DB)

Add Banned_fraction and Degraded_fraction config options to the FreeDiskSpace policy so SEs can be flagged based on the fraction of total free space in addition to absolute thresholds. - FreeDiskSpaceCommand: pass fractions through _prepareCommand, doNew, doCache - FreeDiskSpacePolicy: evaluate fraction thresholds alongside absolute ones - Configurations.py: add default fraction values (1% Banned, 5% Degraded) - Tests: update existing tests and add new fraction threshold test cases - Docs: update admin and developer guides with new config options

…WMSAdministrator methods PropagationCommand now returns Unknown status instead of S_ERROR when site elements are missing or DB queries fail, preventing the Error status from overriding other policies in PDP decisions. Removed 9 deprecated no-op RPC methods from WMSAdministratorHandler (setSiteMask, getSiteMask, getSiteMaskStatus, getAllSiteMaskStatus, banSite, allowSite, clearMask, getSiteMaskLogging, getSiteMaskSummary) that only returned S_OK().

DowntimeCommand now returns S_OK(None) or S_OK() instead of S_ERROR when GOCDB queries fail, the downtime cache is unavailable, or command preparation fails (e.g. missing elementType). This prevents the Error status from overriding other policies in PDP decisions in test environments without GOCDB access.

DowntimeCommand.doNew() now returns S_OK(None) instead of S_ERROR when _prepareCommand() fails, matching the integration test fix.

fstagni · 2026-10-21T10:31:39Z

I have a hard time understand whether this is backward compatible or not, mostly with respect to the config

partially: the backward incompatibility is that the old policy type names (DTOngoing, DTScheduled*, FreeDiskSpaceTB/GB/MB, PropagationPolicy) were consolidated into Downtime, FreeDiskSpace and Propagation. Existing CS sections referencing the old names are silently skipped (they no longer exist in POLICIESMETA), so affected deployments must rename those sections/types and move hours/unit into the new args — hence the Deployment release note. Additionally, two default behaviors change with no CS change: the FreeDiskSpace policy now also applies fraction thresholds (1% Banned / 5% Degraded), so large SEs may flip to Degraded;

…times The master-mode DowntimeCommand now takes the look-ahead window from self.args (defaulting to 0, i.e. ongoing downtimes only), but the CacheFeederAgent was passing empty args. As a consequence, scheduled (future) downtimes were no longer stored in the DowntimeCache, and Downtime policies running with hours > 0 (onlyCache=True) would never see them. Explicitly pass hours=120 to restore the pre-existing behavior of the cache feeder.

fstagni requested review from andresailer and atsareg as code owners April 20, 2026 12:56

chrisburr marked this pull request as draft April 20, 2026 13:35

fstagni force-pushed the rss_configurable_thresholds branch 2 times, most recently from 67caea8 to 5689ef1 Compare April 20, 2026 15:48

fstagni marked this pull request as ready for review April 20, 2026 15:49

fstagni force-pushed the rss_configurable_thresholds branch from 5689ef1 to 603dbf3 Compare April 20, 2026 16:10

fstagni force-pushed the rss_configurable_thresholds branch 5 times, most recently from f45acb7 to 17e4b0e Compare June 9, 2026 15:40

iueda mentioned this pull request Jun 23, 2026

[Feature]: Make RSS FreeDiskSpacePolicy actually configurable #8517

Open

1 task

fstagni force-pushed the rss_configurable_thresholds branch from 17e4b0e to 28a1265 Compare July 6, 2026 08:30

iueda approved these changes Sep 8, 2026

View reviewed changes

iueda reviewed Sep 8, 2026

View reviewed changes

Comment thread src/DIRAC/ResourceStatusSystem/Policy/test/Test_RSS_Policy_FreeDiskSpacePolicy.py Outdated

Comment thread src/DIRAC/ResourceStatusSystem/Policy/test/Test_RSS_Policy_FreeDiskSpacePolicy.py Outdated

fstagni force-pushed the rss_configurable_thresholds branch 2 times, most recently from 0998326 to fd47b7f Compare September 8, 2026 09:29

fstagni force-pushed the rss_configurable_thresholds branch 6 times, most recently from aac5874 to ed1f231 Compare September 25, 2026 11:52

fstagni force-pushed the rss_configurable_thresholds branch from ed1f231 to a27c4e5 Compare October 8, 2026 15:32

feat: the thresholds and unit used for the FreeDiskSpace policy are f…

57f10e0

…ully configurable

fstagni added 13 commits October 21, 2026 12:22

feat: the hours for the DowntimePolicy are now configurable

8345349

docs: better explanation of multiple policies configuration

8488be3

chore: removed outdated tests

b3a3920

feat(rss): added default matchParams

c6a5a96

test: update DowntimeCommand tests to expect S_OK on preparation failure

3243434

DowntimeCommand.doNew() now returns S_OK(None) instead of S_ERROR when _prepareCommand() fails, matching the integration test fix.

fstagni force-pushed the rss_configurable_thresholds branch from a27c4e5 to 3243434 Compare October 21, 2026 10:22

fstagni force-pushed the rss_configurable_thresholds branch from 63828d9 to f2bce10 Compare October 21, 2026 13:57

fstagni closed this Oct 22, 2026

fstagni reopened this Oct 22, 2026

Navigation Menu

Repository navigation

[for minor release] RSS configurable thresholds - #8520

[for minor release] RSS configurable thresholds#8520
fstagni wants to merge 15 commits into
DIRACGrid:integrationfrom
fstagni:rss_configurable_thresholds

fstagni commented Apr 20, 2026 •

edited

Loading

Uh oh!

aldbr commented Jun 11, 2026 •

edited

Loading

Uh oh!

iueda commented Jun 22, 2026

Uh oh!

fstagni commented Jul 6, 2026

Uh oh!

iueda left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chaen commented Oct 8, 2026 •

edited

Loading

Uh oh!

fstagni commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

fstagni commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldbr commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iueda commented Jun 22, 2026

Uh oh!

fstagni commented Jul 6, 2026

Uh oh!

iueda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chaen commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fstagni commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fstagni commented Apr 20, 2026 •

edited

Loading

aldbr commented Jun 11, 2026 •

edited

Loading

chaen commented Oct 8, 2026 •

edited

Loading