Skip to content

fix (RSS): restore StorageOccupancy accounting keys and Meta key in FreeDiskSpaceCommand - #8867

Merged
fstagni merged 2 commits into
DIRACGrid:integrationfrom
fstagni:fix/storage-occupancy-accounting-keys
Nov 2, 2026
Merged

fstagni merged 2 commits into
DIRACGrid:integrationfrom
fstagni:fix/storage-occupancy-accounting-keys

Conversation

@fstagni

@fstagni fstagni commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to 9152869 (Endpoint removal from SpaceTokenOccupancyCache), which introduced two regressions in FreeDiskSpaceCommand:

 _storeCommand  (src/DIRAC/ResourceStatusSystem/Command/FreeDiskSpaceCommand.py)
   accountingDict
-    # only "Site" -> StorageOccupancy registers rejected client-side
+    "StorageElement": results["ElementName"]
+    "Endpoint": "Deprecated"

 _cleanCommand
-  {"meta": {"older": [...]}}   # AttributeError: no attribute 'meta'
+  {"Meta":   {"older": [...]}} # SQLAlchemyDB.select only knows "Meta"

checkValues() (BaseAccountingType) requires every definitionKeyFields entry of StorageOccupancy (Site, Endpoint, StorageElement, SpaceType) to be set before gDataStoreClient.addRegister() accepts a record. With Endpoint and StorageElement removed from the dict, all 3 registers per SE (Total/Free/Used) were rejected, i.e. no StorageOccupancy accounting data has been committed since the v12.0.22 deployment.

The DB-side StaleDataError (UPDATE ... expected to update 1 row(s); 2 were matched) was already resolved operationally by deduplicating rows per Token and dropping the Endpoint column, and is therefore not part of this PR.

Evidence

Before (v12.0.22, CacheFeederAgent logs):

ResourceStatus/CacheFeederAgent/FreeDiskSpaceCommand WARN:
  Could not commit register Invalid values: no value for Endpoint, no value for StorageElement
ResourceStatus/CacheFeederAgent/FreeDiskSpaceCommand ERROR: Failed to execute command
  AttributeError: type object 'SpaceTokenOccupancyCache' has no attribute 'meta'

After: all StorageOccupancy.definitionKeyFields are filled, so checkValues() passes and addRegister() accepts the records; _cleanCommand meta conditions are recognized by SQLAlchemyDB.select (params.get("Meta", {})), so doMaster no longer crashes at the end of each cycle.

ruff/pylint on the touched file report only pre-existing findings. The RSS integration tests require a live CS/MySQL and cannot run in CI-less environments.

Note: accounting plots will show a gap covering the v12.0.22 window, since the rejected registers never reached the DataStore.

Merge Danger

Door: two-way

Both commits are small, independent and trivially revertable; no schema or accounting type definition is changed (Endpoint: "Deprecated" is just a placeholder value, consistent with what the cache table has stored since Aug 2024).

Blast Radius: CacheFeederAgent

Only FreeDiskSpaceCommand is touched. Ramifications if wrong: accounting records with a placeholder Endpoint (cosmetic for plots, which group on StorageElement/SpaceType), or a restored crash in the end-of-cycle cleanup (worst case: same as current production behaviour).

9152869 removed StorageElement and Endpoint from the accountingDict
used to fill the StorageOccupancy accounting registers. These two fields
are mandatory key fields of the accounting type (checkValues fails
otherwise), so since then no StorageOccupancy record could be committed,
with CacheFeederAgent repeatedly logging:

  Could not commit register Invalid values: no value for Endpoint,
  no value for StorageElement

Fill them again, with a deprecated placeholder endpoint.
b7f5f47 replaced the selectSpaceTokenOccupancyCache client call with a
direct rmDB.select, but passed the meta conditions under the key 'meta'
(lowercase). SQLAlchemyDB.select only recognizes 'Meta', so 'meta' was
treated as a table column and _cleanCommand crashed with:

  AttributeError: type object 'SpaceTokenOccupancyCache' has no attribute 'meta'

making doMaster fail at the end of every cycle (and skipping the cache cleanup).
@fstagni
fstagni marked this pull request as ready for review October 30, 2026 14:43
@fstagni
fstagni requested a review from atsareg as a code owner October 30, 2026 14:43
@fstagni
fstagni merged commit c7b065f into DIRACGrid:integration Nov 2, 2026
25 checks passed
@DIRACGridBot DIRACGridBot added the sweep:ignore Prevent sweeping from being ran for this PR label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sweep:ignore Prevent sweeping from being ran for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants