feat: add a generic metadata: field to all resources - #9872
feat: add a generic metadata: field to all resources#9872nishantmonu51 wants to merge 5 commits into
metadata: field to all resources#9872Conversation
Adds a top-level `metadata:` map to every resource YAML, parsed generically in `commonYAML` and stored on `ResourceMeta.metadata`, mirroring `tags:`. - Values are strings; non-string scalars are coerced (`tier: 1` -> "1") and nulls become "". Non-scalar values are a parse error naming the key. - Strict validation is otherwise unchanged: unknown fields still error. - Exposed over `GetResource`/`ListResources`/`WatchResources` on `meta`. - Also settable from SQL annotations (`-- @metadata.owner: alice`). - Auto-generated inline explores inherit the metrics view's metadata. Claude-Session: https://claude.ai/code/session_01FvtFG1k64JW9SpV8aCH5Se
Assigning the shared `err` on every successful metadata key could overwrite an error set by an earlier annotation case. Map iteration order is random, so an invalid `-- @name:` or `-- @connector:` could be silently dropped. Claude-Session: https://claude.ai/code/session_01FvtFG1k64JW9SpV8aCH5Se
| // Metadata is user-defined key-value metadata. Parsed generically from any resource YAML's top-level "metadata:" field. | ||
| // Rill does not interpret it; it is exposed here for external tooling. | ||
| // It maps to an Apache Ossie `custom_extensions` entry with vendor_name "RILL" and data `{"metadata": {...}}`. | ||
| map<string, string> metadata = 20; |
There was a problem hiding this comment.
metadata feels too generic and does not specify that Rill ignores this especially given the outer field is also called meta.
Some suggestions:
extensions/ext: Used for vendor specific fields in specs like OpenRTB.
ext_metadata: Again emphasis on the fact that this is external metadata and not Rill specific metadata.
There was a problem hiding this comment.
We can have rill specific metadata as well here e.g. whether the resource is AI generated or edited by human etc.. so, would like to start with metadata for now.
There was a problem hiding this comment.
Keeping metadata per the above, but reworded the docstring (and the schema description, generated docs and TS types) so it no longer claims Rill ignores the field: it now says it is free-form key-value metadata, currently user-defined, and that Rill may populate its own keys later (e.g. provenance).
There was a problem hiding this comment.
Rill putting its own keys in a field that it never parses and stores and forwards as black box seems to be an anti-pattern IMO.
It can also lead to edge cases/complex handling like how do we merge user's metadata with Rill's metadata.
Addresses review feedback: yaml.v3 already coerces non-string scalars into string targets, and the SQL annotation path already goes through mapstructure with WeaklyTypedInput, so the custom decoder was redundant. Also rewords the field docstring and drops the Ossie mapping note from it. Claude-Session: https://claude.ai/code/session_01FvtFG1k64JW9SpV8aCH5Se
| // Tags for organizing and filtering resources. Parsed generically from any resource YAML's top-level "tags:" field. | ||
| repeated string tags = 19; | ||
| // Metadata is free-form key-value metadata for the resource, parsed generically from any resource YAML's top-level "metadata:" field. | ||
| // It is currently user-defined and exposed as-is over the API, but Rill may also populate its own keys in the future (e.g. provenance). |
There was a problem hiding this comment.
As noted in other comment Rill putting its own keys in metadata is an anti pattern and prone to complications :
2 participants
Adds a top-level
metadata:map to every resource YAML file, parsed generically incommonYAMLand stored onResourceMeta.metadata(proto field 20). This mirrors the generictags:field, and gives teams a supported way to attach their own metadata (owner, tier, classification, ticket link) to a resource and read it back out over the API.tier: 1becomes"1"andpii: falsebecomes"false", and a null value becomes"". A nested map or list is a parse error naming the offending key.metainGetResource/ListResources/WatchResources. Nothing in Rill interprets it.-- @metadata.owner: alice), so.sql-only models can carry it too.tagsbehaves.dev:/prod:overrides work.metadata:in arill.yamldefaults:block does not apply, same astags:today; the generated docs say so.Shape note: values are kept a flat
map[string]stringso they serialize losslessly into an Apache Ossiecustom_extensionsentry (vendor_name: RILL,datais a JSON string) if we add an Ossie importer/exporter later.No catalog migration is needed, since
ResourceMetais persisted as an opaque proto blob. The largeresources.pb.godiff is the nestedMetadataEntrymessage renumbering everymsgTypesindex below it.Related to #9794.
Checklist:
https://claude.ai/code/session_01FvtFG1k64JW9SpV8aCH5Se