Repository navigation
Feature: CMR Concept ID <-> Human Readible Name translation tool #24
Description
Hi python-cmr folks,
I'm a developer at GES DISC, and during a hackweek last week my colleague @eni-awowale created a CLI tool (tentatively named CNR: CMR Name Resolver) that translates concept_id's to native_ids (and vise-versa). It's not publically shareable at this point but if you have access to Earthdata bitbucket you can see our repo here: https://git.earthdata.nasa.gov/projects/GDDS/repos/cnr/browse
After discussion with our manager @briannapagan we were wondering if this would be something you'd be interested in merging into python-cmr. Before opening any PRs I wanted to start a discussion on this to make sure 1) this is something you all are interested in having in this project and 2) we're on the same page for how to implement.
The tool is a python script that gets installed as a CLI tool in the ./bin directory of your python install by using [project.scripts] in pyproject.toml. Example usage:
$ cnr M2T1NXSLV_5.12.4
prod C1276812863-GES_DISC
uat C1245662776-EEDTEST
uat C1215802944-GES_DISC
$ cnr -e prod M2T1NXSLV_5.12.4
prod C1276812863-GES_DISC
$ cnr C1215664070-GES_DISC
uat GLDAS_NOAH10_3H_2.0
$ cnr -p MERRA2_100.inst1_2d_asm_Nx.19800101.nc4 -p GES_DISC
prod G1276974976-GES_DISC
uat G1256129374-GES_DISC
sit None
The full list of aceeptable arguments:
$ cnr -h
usage: cnr [-h] [-e {prod,sit,uat}] [-p PROVIDER] [-c] [-g] [-s] id
Get a concept-id or native-id given either
positional arguments:
id Collection ID or Native ID to convert
options:
-h, --help show this help message and exit
-e {prod,sit,uat}, --environment {prod,sit,uat}
Environment to use
-p PROVIDER, --provider PROVIDER
Used for a granule and collection query in CMR with a provider
id
-c, --collection Used to denote a collection query in CMR
-g, --granule Used to denote a granule query in CMR
-s, --service Used to denote a service query in CMR
Activity
owenlittlejohns commentedon Oct 3, 2023
Comment from Slack thread:
Just to expand a little bit - when making a request against the CMR GraphQL interface, you have to specify the fields you get back in the response. What I'm trying to think through is whether:
a) Can this work be backed by CMR GraphQL to make more efficient queries?
b) Does the CMR GraphQL API essentially already handle this use-case and are we primarily missing some communication about the super-powers CMR GraphQL already gives us?
c) Do the parameter sets for the concepts supported by CMR GraphQL include all the things you want to query with (concept ID, for sure, but maybe not things like short name and version, yet)?
One really cool thing CMR GraphQL doesn't give, but this tool automatically does, is results from all of production, UAT and SIT.
eudoroolivares2016 commentedon Oct 3, 2023
@owenlittlejohns
a) you could make python-cmr query against cmr-graphql but, that might be a big change since everything else here is hitting cmr directly so that would probably be outside the scope of this issue
b) I'm almost certain we can make a custom field with graphql that could do this use-case (I've done something like this before on a personal project I think)
c) we do support querying by shortname on cmr-graphql but, not short-name + version. I don't know how much trouble it would be to add that to graphql. I would think it wouldn't be crazy difficult since cmr supports it https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#c-version but, I'm not certain
abbottry commentedon Oct 3, 2023
Querying by short-name + version isn't desirable because you can only do so via a single record. Some time ago the DAACs got together and discussed this, the result was Entry ID which allows users to search to by the combination of short-name and version via a single, supported and indexed field.
Looking through the docs, and testing a few responses, that value does not come back in either json or umm_json and should be added, that would seem to solve a pretty big need.
With respect to the environments, SIT and UAT should be considered testing environments, I would not recommend features in production that provided the user base with test data. That being said, it has come up recently that the use of environments is useful from a curation perspective and there should be some discussion around that.
So, if this library had a feature that would present these results, using supported params and keys that are indexed, that would be a neat feature of a tool.
eni-awowale commentedon Oct 3, 2023
Thanks everyone for sharing your thoughts! To clarify we don't use short_name + version to query collections for concept-ids, we use native-ids, which to some is the same as short_name + version, but this is not consistent across DAACs. I like the idea of using entry-id since that is the agreed upon field for short_name + version searches.
abbottry commentedon Oct 3, 2023
So, it seems like something that would be useful would be filing a ticket with CMR to expose Entry ID (there are multiple teams actually looking to utilize this in the short term)
eni-awowale commentedon Dec 20, 2023
Hey folks! Restarting this thread now that we are moving into the IP. What do you all think are our next steps?
frankinspace commentedon Apr 10, 2024
@eni-awowale I'm late to the party but re-reading the thread it sounds like this got redirected into a feature request for CMR directly? Once searching by Entry ID is supported we could add that to this library and that would solve your original use case?
eni-awowale commentedon Apr 11, 2024
Hi @frankinspace ! Thanks for re-starting this thread. I wasn't aware that this got redirected into a feature request for CMR. Can you share it with me? I do think once searching via Entry ID is supported adding the CL functionality to this library would solve our original use case.
frankinspace commentedon Apr 11, 2024
@eni-awowale That was meant as a question as I was reviewing the discussion on this issue; it sounds like the suggestion is work required by CMR to support Entry ID. I don't feel I have a strong enough grasp of the issue to make the request but maybe I can poke @abbottry to see if he would be willing to fill out that ticket to CMR and link it here?