change(refactor): simplify migration procedure by kpsherva · Pull Request #654 · CERNDocumentServer/cds-migrator-kit

kpsherva · 2026-10-16T12:10:14Z

Logic unchanged, consolidated manual procedures, the whole collection is described in one yaml files shared between all environments (committed here, no changing the configuration per env in openshift anymore)

Details

The working directory of a collection is <root>/<collection>/:

<repo>/migration_cli/rdm-data/        <- the root (default of the local environment)
    users/people.csv                   shared by all collections
    antares/                           <- the working directory of the collection
        dump/                          records dump
        comments/                      optional, comments of the collection

mkdir -p migration_cli/rdm-data/users migration_cli/rdm-data/antares

people.csv is the people authority export of the environment. Copy it to migration_cli/rdm-data/users/people.csv from the EOS users folder, or recreate it as described in migration_cli/migrating-a-collection.md.

2. Configuration

The configuration of a collection is the file migration_cli/collections/antares.yaml of the repository, the same for every environment (all the options in migration_cli/collections/_example.yaml):

name: antares
queries:
  - '<TODO legacy query of the collection>'
flags:
  plots: true
communities:
  - cern-research
  - antares

3. Get the dumps from the legacy system

The legacy tools exist only on cds-migration-01. Run the script there with a scratch folder as root (not on EOS), then download the result:

# on cds-migration-01
S=/opt/cds-migrator-kit/migration_cli      # a checkout that has the configuration of the collection
python $S/collection_migration.py dump records --collection antares --env local --root /tmp/antares-root
ipython $S/collection_migration.py -- dump comments --collection antares --env local --root /tmp/antares-root   # optional

# on your machine
rsync -av cds-migration-01:/tmp/antares-root/antares/dump/ migration_cli/rdm-data/antares/dump/
rsync -av cds-migration-01:/tmp/antares-root/antares/comments/ migration_cli/rdm-data/antares/comments/

The dumps change only when the legacy records change (f.e. when curators , so they do not need to be refreshed on every run. dump files and prepare are not needed locally. Add prepare (duplicates to redirect) if you want to test the redirections, it writes duplicated_pids.json in the working directory of the collection.

5. Migrate

Try each command with --dry-run first.

invenio migration users submitters run --collection antares
invenio migration run --collection antares

# only if you dumped the comments
invenio migration users commenters run --collection antares
invenio migration comments run --collection antares

Then reindex, to see the missing users in the UI:

invenio rdm rebuild-all-indices -o users

6. Statistics (optional in local)

The legacy events are read from the legacy OpenSearch, so this needs access to it: export CDS_MIGRATOR_KIT_SRC_SEARCH_HOSTS as described in migration_cli/step_by_step_documentation.md, then:

invenio migration stats events run --collection antares
invenio migration stats aggregate

kpsherva · 2026-10-23T16:22:11Z

@@ -0,0 +1,422 @@
+#!/usr/bin/env python


this is the main legacy dumper script

* consolidates substeps (check readme and migrating-a-collection.md) # Conflicts: # .gitignore # Dockerfile # cds_migrator_kit/rdm/streams.yaml # cds_migrator_kit/rdm/streams_done.yaml # cds_migrator_kit/rdm/streams_shelved.yaml

palkerecsenyi

The code changes look good to me, I have some questions about the docs. Also, more generally, why are we not putting these docs into the digital-repositories internal docs? It would be nice to have everything in one place if possible. Thanks!

palkerecsenyi · 2026-10-26T15:29:44Z

+
+Prepare the data locally first, see https://digital-repositories.web.cern.ch/cds/cds.cern.ch/migration/ and note the results in the issue:
+
+- [ ] check the permissions references and submissions (Submissions tab of the reference spreadsheet)


Which spreadsheet is this referring to?

palkerecsenyi · 2026-10-26T15:31:44Z

+```
+
+- Write the queries **without** the standard exclusions: the script appends `-690C_:SCICOM -595:*ARCHIVE* -980:ANNOUNCEMENT -980:DELETED -980:HIDDEN -980__a:DUMMY -690C_:movingimages -970__b:MERGE-* -980:GREYBOOK`. `-980__c:MIGRATED` is not excluded on purpose, some records belong to several communities.
+- With several queries, also set `analyze_query`: one query, without `-980:DELETED -980:HIDDEN -980__c:MIGRATED -980__a:DUMMY` (duplicates are searched among them to be redirected to).


What does this mean? What is analyze_query used for?

palkerecsenyi · 2026-10-26T15:33:54Z

+- Optional: `folder` (working directory under the data root, default the name), `comments_queries` (default `queries`) and `reviewers` for the comments, and the `flags` (`update_new_version_publication_date: true` only for administrative documents, e.g. HR; `skip_plot_files`; `workers`; ...).
+
+Merge it and update the checkout on `cds-migration-01` (`/opt/cds-migrator-kit`) and the image of the migration pod before going further.
+


Maybe for completeness here we can mention that at this point you should dry-run the migration locally, add the results to the curation spreadsheet (including where the migrator kit stores the error log CSV), and wait for curation? Just since this might not initially be obvious

palkerecsenyi · 2026-10-26T15:36:34Z

+- [ ] reindex all users `invenio rdm rebuild-all-indices -o users` to add missing migrated accounts
+- [ ] set the community slug and deploy the redirections: [TODO PR with redirection]
+- [ ] merge the Puppet `qa` branch to `master` to activate the submissions/collections/search redirections
+- [ ] run the MIGRATED tagging on old CDS: https://gitlab.cern.ch/cds-team/production_scripts/-/blob/master/cds/set_records_migrated_to_rdm.py


Here we could include a template of the email we send to the content owners after their migration is complete?

palkerecsenyi · 2026-10-26T15:39:28Z

-      missing_users: cds_migrator_kit/rdm/data/users
-      communities_ids:
-        - ""
  at:


Why are we keeping some of the streams here?

@kpsherva could you instead move the done ones to done, it is nice to have the history of what rules were used to run the older ones for reference and in case we need to debug something

And in the future from now onwards, the separate files will be there

palkerecsenyi assigned kpsherva Oct 19, 2026

chore(comments): simplify comments migration

3d7237f

kpsherva changed the title ~~chore(comments): simplify comments migration~~ change(refactor): simplify migration procedure Oct 23, 2026

kpsherva force-pushed the consolidate_comments branch from 5838456 to 33e5c68 Compare October 23, 2026 16:16

kpsherva commented Oct 23, 2026

View reviewed changes

kpsherva force-pushed the consolidate_comments branch 2 times, most recently from 654ea7d to b320486 Compare October 26, 2026 09:02

refactor(global): remove manual steps from migration

b735994

* consolidates substeps (check readme and migrating-a-collection.md) # Conflicts: # .gitignore # Dockerfile # cds_migrator_kit/rdm/streams.yaml # cds_migrator_kit/rdm/streams_done.yaml # cds_migrator_kit/rdm/streams_shelved.yaml

kpsherva force-pushed the consolidate_comments branch from b320486 to b735994 Compare October 26, 2026 09:13

kpsherva removed their assignment Oct 26, 2026

palkerecsenyi reviewed Oct 26, 2026

View reviewed changes

Navigation Menu

Repository navigation

change(refactor): simplify migration procedure - #654

change(refactor): simplify migration procedure#654
kpsherva wants to merge 2 commits into
CERNDocumentServer:masterfrom
kpsherva:consolidate_comments

kpsherva commented Oct 16, 2026 •

edited

Loading

Uh oh!

kpsherva Oct 23, 2026

Uh oh!

palkerecsenyi left a comment

Uh oh!

palkerecsenyi Oct 26, 2026

Uh oh!

palkerecsenyi Oct 26, 2026

Uh oh!

palkerecsenyi Oct 26, 2026

Uh oh!

palkerecsenyi Oct 26, 2026

Uh oh!

palkerecsenyi Oct 26, 2026

Uh oh!

sakshamarora1 Oct 26, 2026

Uh oh!

sakshamarora1 Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		Prepare the data locally first, see https://digital-repositories.web.cern.ch/cds/cds.cern.ch/migration/ and note the results in the issue:

		- [ ] check the permissions references and submissions (Submissions tab of the reference spreadsheet)

		- Optional: `folder` (working directory under the data root, default the name), `comments_queries` (default `queries`) and `reviewers` for the comments, and the `flags` (`update_new_version_publication_date: true` only for administrative documents, e.g. HR; `skip_plot_files`; `workers`; ...).

		Merge it and update the checkout on `cds-migration-01` (`/opt/cds-migrator-kit`) and the image of the migration pod before going further.

Conversation

kpsherva commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

2. Configuration

3. Get the dumps from the legacy system

5. Migrate

6. Statistics (optional in local)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

palkerecsenyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kpsherva commented Oct 16, 2026 •

edited

Loading