Repository navigation
Enhance dataset lookup and improve job message handling - #841
Merged
Merged
Conversation
job/get_datasets_by_name read ATLAS_PANDA.Datasets by the names as given, but a dataset's row can be under either scope:name or name. insertDataset skips a dataset whose vuid is already in the table, and the vuid is md5(scope:name) whichever form the name came in, so the table keeps one row per dataset under the form its first writer used: the prodDBlock of an input, which may carry a scope, or the destinationDBlock of an output. A caller asking with the other form got no row, and the live harness had to send both forms itself. Each name is now looked up as given and in its other form: a scoped name also without its scope, and an unscoped one also with the scope extract_scope gives, which is the scope the vuid is made with. Every row says in asked_as which given names it answers. The IN-list cap and the probe budget count these keys, and the answers carry keys_per_name, so a call may give budget // (keys_per_name * partitions) names, 100 on production's 5 partitions instead of 200.
getJobs removes a push job's message from panda_pilot_queue after committing the job, but get_mb_proxy returns None when that queue is not configured or the message broker setup failed in this process, and the call went ahead without checking. The AttributeError reached the outer except after the commit, so getJobs rolled back nothing and returned no jobs, leaving every job committed so far in that call sent but handed to no pilot. The other two senders already check the proxy; this one now does the same and logs that the queue is not configured.
get_mb_proxy returns the MBSenderProxy that start_passive_mode put under the channel, or None, but was declared Any, as were the senders in srv_msg_utils that take its result and the workflow manager's own mb_proxy. They are now MBSenderProxy, imported under TYPE_CHECKING since msg_bkr_utils sets up its loggers and imports stomp on import. CI's mypy does not install panda-common, so there the name is still Any; with panda-common 0.1.11 visible the tree reports nothing new. Three fields whose class was only in a comment now carry it: DaemonMaster.tbif is a TaskBufferInterface or None, AdderGen.add_result an AdderResult or None, and cacheSchedConfig.queueData the list of dicts query_column_sql builds. queueData starts as an empty list rather than None; its only reader, main, calls getQueueData first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request refines the
get_datasets_by_nameAPI to more accurately handle dataset name lookups, ensuring both scoped and unscoped forms are considered, and updates related documentation, error handling, and tests accordingly. It also improves type annotations and code clarity in several modules.Improvements to dataset name lookup logic and API:
pandaserver/api/v1/job_api.py,pandaserver/api/v1/tests/job_api_tests.py) [1] [2] [3] [4] [5] [6] [7] [8]Enhancements to typing and code clarity:
TYPE_CHECKINGto avoid circular imports and runtime issues. (pandaserver/daemons/utils.py,pandaserver/dataservice/adder_gen.py,pandaserver/srvcore/srv_msg_utils.py) [1] [2] [3] [4] [5] [6] [7] [8]Minor code and initialization fixes:
queueDatain the scheduler cache is corrected to an empty list for consistency with expected usage. (pandaserver/daemons/scripts/cache_schedconfig.py) [1] [2]