Document approaches for site builds on top of EESSI by casparvl · Pull Request #778 · EESSI/docs

casparvl · 2026-06-04T11:18:09Z

No description provided.

… the mkdocs.yml

ocaisa

Found some typos

first attempt at writing the shared fs section

…, after a reboot, the stratum 0 overlay won't mount

…cript. Also, added a full, copyable script combining all the previous components.

Neves-P

Looks really nice! This is my first pass of comments, as of know I only focused on the text. This week I will follow the documentation in practice and review again.

Neves-P · 2026-06-23T08:49:10Z

+
+Another thing to consider is to create a secondary user with `aws iam create-user --user-name <name>` and attach a very limited policy to it (e.g. only read/write/list on buckets, nothing else). Then, create credentials for this user with `aws iam create-access-key --user-name <name>` and provide those credentials to the EESSI build bot and Stratum 0 machines. That way, if that token is compromized, the impact is minimized (e.g. the token can at least not be used to create new IAM idententies, etc).
+
+## Setting up the EESSI build bot


I'd consider separating this into a new page. The build bot is somewhat self contained and the page is quite long already.

casparvl · 2026-07-22T14:34:04Z

+
+Another thing to consider is to create a secondary user with `aws iam create-user --user-name <name>` and attach a very limited policy to it (e.g. only read/write/list on buckets, nothing else). Then, create credentials for this user with `aws iam create-access-key --user-name <name>` and provide those credentials to the EESSI build bot and Stratum 0 machines. That way, if that token is compromised, the impact is minimized (e.g. the token can at least not be used to create new IAM idententies, etc).
+
+## Setting up the EESSI build bot


I realize these docs lack one crucial thing: setting up a site_config_script that sets EESSI_SITE_INSTALL_FORCE and EESSI_SITE_SOFTWARE_PATH_PREFIX

I.e. at SURF, our site_config_script looks like this:

# To build on top of EESSI, we need to software.eessi.io repository to be mounted next to our own repository # The bot/build.sh script does this when the EESSI_SITE_INSTALL_FORCE environment variable is set # Other build scripts will also respect this variable where needed in order to make sure that 'building on top' # of EESSI is possible export EESSI_SITE_INSTALL_FORCE=1 echo "Value of EESSI_SITE_INSTALL_FORCE: $EESSI_SITE_INSTALL_FORCE" # We also need to set a prefix that our installations should end up in # The build scripts should take this prefix, and construct the final EESSI_SITE_SOFTWARE_PATH out of it # that the EESSI-extend module expects export EESSI_SITE_SOFTWARE_PATH_PREFIX=/cvmfs/software.surf.nl/versions/2025.06/ echo "Value of EESSI_SITE_SOFTWARE_PATH_PREFIX: $EESSI_SITE_SOFTWARE_PATH_PREFIX"

One thing to figure out might be how to avoid hard-coding the version in here :)

Scratch that, this should not be needed anymore... I think we do this differently now... We automatically set these here https://github.com/EESSI/software-layer-scripts/blob/ca929cd7ef32a9fcd79fafd4e0d5c362a1fff452/bot/build.sh#L156 . That happens if the repo_name in the repos config file is unequal to software.eessi.io or dev.eess.io essentially.

I think this does warrant a small explanation of why EESSI_SITE_SOFTWARE_PREFIX, EESSI_SITE_INSTALL and friends are not needed with this workflow. Maybe we don't even need an explanation, but just a mention.

I'm saying this because going through the whole docs - especially the shared file system installs - you get the impression that at least doing EESSI_SITE_INSTALL is necessary for site installations (which it is, but that happens behind the scenes).

Neves-P

Just two comments, suggestions. I would have liked to have been more thorough, but overall I did two passes to the text and managed to get to the point of installing the bot (had to wait a bit for the internal S3 and maintenance).

In my mind, after addressing these comments we can either wait a little longer and I spin up a new test CVMFS infrastructure to also test that part, or merge before that and address any thing else we spot in a later PR.

Overall I think the instructions are very nice! I'm not sure I am able to put myself in the shoes of a complete novice to EESSI, but they seem very appropriate for that target audience.

Neves-P · 2026-08-04T10:16:59Z

+
+Another thing to consider is to create a secondary user with `aws iam create-user --user-name <name>` and attach a very limited policy to it (e.g. only read/write/list on buckets, nothing else). Then, create credentials for this user with `aws iam create-access-key --user-name <name>` and provide those credentials to the EESSI build bot and Stratum 0 machines. That way, if that token is compromised, the impact is minimized (e.g. the token can at least not be used to create new IAM idententies, etc).
+
+## Setting up the EESSI build bot


I think this does warrant a small explanation of why EESSI_SITE_SOFTWARE_PREFIX, EESSI_SITE_INSTALL and friends are not needed with this workflow. Maybe we don't even need an explanation, but just a mention.

I'm saying this because going through the whole docs - especially the shared file system installs - you get the impression that at least doing EESSI_SITE_INSTALL is necessary for site installations (which it is, but that happens behind the scenes).

Neves-P · 2026-08-04T15:38:05Z

+``` { .bash .copy }
+curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+unzip awscliv2.zip
+sudo ./aws/install


This can be installed in userland, and ideally I'd do so. That's how I normally install it in the dev.eessi.io bots and it works well.

Neves-P

Another small detail that I spotted now and am adding here so I don't forget.

Neves-P · 2026-09-01T14:39:17Z

The last commit message was supposed to read: "Account for custom prefix installs for .cvmfsdirtab"

After that is reviewed, we need to account for the changes in the tarball ingestion step too. That is blocked until EESSI/filesystem-layer#278 is merged.

Co-authored-by: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com>

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

Neves-P

Suggestions with updates accounting for VERSIONS_SUBPATH workflow in ingest-tarball.sh.

Neves-P · 2026-09-18T12:48:01Z

+DOWNLOAD_DIR=/prefix/for/tarball/staging  # Some directory to temporarily store tarballs on the Stratum 0
+ALLOWED_SIGNERS=/path/to/allowed/signers/file  # Optional, needed in step 4
+REPO_NAME="<repo_name>"
+# a name for a dir in the bucket in which to archive tarballs, so that a subsequent run doesn't re-ingest them


Suggested change

# a name for a dir in the bucket in which to archive tarballs, so that a subsequent run doesn't re-ingest them

# Repository-relative path to the versions directory.

# This must match the versions_subpath used for .cvmfsdirtab.

VERSIONS_SUBPATH="versions"

# For a site prefix such as /cvmfs/name.sitename.tld/eessi, use:

# VERSIONS_SUBPATH="eessi/versions"

# a name for a dir in the bucket in which to archive tarballs, so that a subsequent run doesn't re-ingest them

Neves-P · 2026-09-18T12:54:41Z

+# 3. Update the lmod caches for your site installation prefix
+INGEST_SCRIPT="$PWD/filesystem-layer/scripts/ingest-tarball.sh"
+```
+


Suggested change

`VERSIONS_SUBPATH` must match the `versions_subpath` used when [configuring `.cvmfsdirtab`](#setting-up-your-stratum-0). It is relative to the root of the target CVMFS repository and must include the `versions` directory.

For example, with `EESSI_SITE_SOFTWARE_PREFIX=/cvmfs/name.sitename.tld/eessi`, use `VERSIONS_SUBPATH=eessi/versions`. The default is `versions`.

Neves-P · 2026-09-18T12:55:39Z

+
+**5. Ingest the tarball into the repository**
+
+Here, we leverage a script from `EESSI/filesystem-layer` that ingests tarballs, but also takes care of regenerating the `.cvmfscatalog` files _and_ updates the `Lmod` cache. To update the `Lmod` cache, the script uses the Lmod installation provided by `software.eessi.io`, which is why we explicitly made this available as one of the steps [when we set up our Stratum 0](#setting-up-your-stratum-0)


Suggested change

Here, we leverage a script from `EESSI/filesystem-layer` that ingests tarballs, but also takes care of regenerating the `.cvmfscatalog` files _and_ updates the `Lmod` cache. To update the `Lmod` cache, the script uses the Lmod installation provided by `software.eessi.io`, which is why we explicitly made this available as one of the steps [when we set up our Stratum 0](#setting-up-your-stratum-0)

Here, we leverage [`ingest-tarball.sh`](https://github.com/EESSI/filesystem-layer/blob/main/scripts/ingest-tarball.sh) from `EESSI/filesystem-layer`. The script ingests the tarball, regenerates the `.cvmfscatalog` files, and updates the Lmod cache.

The tarball contains paths relative to the versioned installation directory, such as `2025.06/software/...`. The `--basedir` option determines where that directory is placed relative to the root of the target CVMFS repository. It must therefore match the repository layout configured through `VERSIONS_SUBPATH`.

When updating the Lmod cache, the script uses the selected base directory for the site installation. If the site repository does not provide its own compatibility layer, it automatically falls back to an Lmod installation from `/cvmfs/software.eessi.io/versions`.

Neves-P · 2026-09-18T12:56:13Z

+    echo "Ingesting into CVMFS (${REPO_NAME})..."
+    if $INGEST_SCRIPT "${REPO_NAME}" "${local_tar}"; then
+        echo "Ingest succeeded for ${filename}."
+    else
+        echo "ERROR: cvmfs_server ingest failed for ${filename}." >&2
+        # Keep the files for troubleshooting
+        continue
+    fi


Suggested change

echo "Ingesting into CVMFS (${REPO_NAME})..."

if $INGEST_SCRIPT "${REPO_NAME}" "${local_tar}"; then

echo "Ingest succeeded for ${filename}."

else

echo "ERROR: cvmfs_server ingest failed for ${filename}." >&2

# Keep the files for troubleshooting

continue

fi

echo "Ingesting into CVMFS (${REPO_NAME}) under ${VERSIONS_SUBPATH}..."

if "${INGEST_SCRIPT}" \

--repository "${REPO_NAME}" \

--basedir "${VERSIONS_SUBPATH}" \

"${local_tar}"; then

echo "Ingest succeeded for ${filename}."

else

echo "ERROR: CVMFS ingestion failed for ${filename}." >&2

# Keep the files for troubleshooting

continue

fi

Neves-P · 2026-09-18T12:56:53Z

+
+**6. Regenerate the `.cvmfscatalog` files by publishing an empty transaction**
+
+This is already taken care of by the `$INGEST_SCRIPT` in the step above. If you don't want to use that script, you'll have to implement this step yourself.


Suggested change

This is already taken care of by the `$INGEST_SCRIPT` in the step above. If you don't want to use that script, you'll have to implement this step yourself.

This is already taken care of by `$INGEST_SCRIPT` in the step above. The patterns in `.cvmfsdirtab` must use the same repository-relative versions path as `VERSIONS_SUBPATH`; otherwise the expected nested catalogs will not be created.

Neves-P · 2026-09-18T12:57:22Z

+
+**7. Open a new transaction, update the Lmod cache for your site installs, and publish the transaction**
+
+This is already taken care of by the `$INGEST_SCRIPT` in step 5 above. If you don't want to use that script, you'll have to implement this step yourself.


Suggested change

This is already taken care of by the `$INGEST_SCRIPT` in step 5 above. If you don't want to use that script, you'll have to implement this step yourself.

This is already taken care of by `$INGEST_SCRIPT` in step 5. The cache is updated below `${VERSIONS_SUBPATH}/<eessi_version>` in the site repository. The script will use an Lmod installation from the site repository when one is available, and otherwise falls back to `software.eessi.io`.

Neves-P · 2026-09-18T12:59:29Z

+BUCKET="<bucket_name>"
+DOWNLOAD_DIR=/prefix/for/tarball/staging  # Some directory to temporarily store tarballs on the Stratum 0
+ALLOWED_SIGNERS=/path/to/allowed/signers/file  # Optional, needed in step 4
+REPO_NAME="<repo_name>"


Suggested change

REPO_NAME="<repo_name>"

REPO_NAME="<repo_name>"

# Repository-relative path to the versions directory.

# This must match the versions_subpath used for .cvmfsdirtab.

VERSIONS_SUBPATH="versions"

# For a site prefix such as /cvmfs/name.sitename.tld/eessi, use:

# VERSIONS_SUBPATH="eessi/versions"

Neves-P · 2026-09-18T13:01:02Z

+    # ---- Ingest into CVMFS ----
+    echo "Ingesting into CVMFS (${REPO_NAME})..."
+    if $INGEST_SCRIPT "${REPO_NAME}" "${local_tar}"; then
+        echo "Ingest succeeded for ${filename}."
+    else
+        echo "ERROR: cvmfs_server ingest failed for ${filename}." >&2
+        # Keep the files for troubleshooting
+        continue
+    fi


Suggested change

# ---- Ingest into CVMFS ----

echo "Ingesting into CVMFS (${REPO_NAME})..."

if $INGEST_SCRIPT "${REPO_NAME}" "${local_tar}"; then

echo "Ingest succeeded for ${filename}."

else

echo "ERROR: cvmfs_server ingest failed for ${filename}." >&2

# Keep the files for troubleshooting

continue

fi

# ---- Ingest into CVMFS ----

echo "Ingesting into CVMFS (${REPO_NAME}) under ${VERSIONS_SUBPATH}..."

if "${INGEST_SCRIPT}" \

--repository "${REPO_NAME}" \

--basedir "${VERSIONS_SUBPATH}" \

"${local_tar}"; then

echo "Ingest succeeded for ${filename}."

else

echo "ERROR: CVMFS ingestion failed for ${filename}." >&2

# Keep the files for troubleshooting

continue

fi

Add initial files for each site-building approach

a4d5de8

casparvl changed the title ~~Add initial files for each site-building approach~~ Document approaches for site builds on top of EESSI Jun 4, 2026

Caspar van Leeuwen and others added 5 commits June 4, 2026 17:12

Made a start with the site builds as a CVMFS repo

259e935

Started to write Stratum 0 install docs. Work in progress

6aebcec

Slight rephrasing of site-build overview section and add new pages to…

dcf695f

… the mkdocs.yml

Merge branch 'main' into site_builds

87ccf16

Corrected typo

c887a09

ocaisa requested changes Jun 9, 2026

View reviewed changes

Caspar van Leeuwen added 2 commits June 9, 2026 15:20

More or less completed the docs on how to setup a Stratum 0

729cf88

Finished description for Stratum 0 and Stratum 1

c31746d

casparvl commented Jun 10, 2026

View reviewed changes

Comment thread docs/site_build/site_cvmfs.md

Caspar van Leeuwen and others added 9 commits June 11, 2026 10:09

Finished a few more chapters

e4fac7e

Small update

b9205ca

Started section on installing EESSI build bot on a machine

7b027fc

Further work on deploying bot on machine

3f49e7b

first attempt at writing the shared fs section

8fe6e19

Merge pull request #5 from bedroge/site_builds_shared_fs

0cddb6b

first attempt at writing the shared fs section

Make sure we add instructions to stop _and disable_ autofs. Otherwise…

ed6f41a

…, after a reboot, the stratum 0 overlay won't mount

Merge branch 'site_builds' of github.com:casparvl/docs into site_builds

5aa289b

Add some debugging instructions

d200d12

Neves-P mentioned this pull request Jun 30, 2026

Add documentation on how to build on top of EESSI as a site - and include the use of EESSI_SITE_SOFTWARE_PREFIX in that #768

Open

Caspar van Leeuwen added 2 commits June 30, 2026 16:23

Added an image, rewrote some parts, fixed some formatting

92f27e0

Fixed some typos

6ea3bef

casparvl marked this pull request as ready for review June 30, 2026 14:36

Caspar van Leeuwen and others added 2 commits July 1, 2026 17:36

Added logic to do signature verification to the automatic ingestion s…

acfcb9e

…cript. Also, added a full, copyable script combining all the previous components.

Fixed codespell issues

2bba8e8

Neves-P linked an issue Jul 7, 2026 that may be closed by this pull request

Add documentation on how to build on top of EESSI as a site - and include the use of EESSI_SITE_SOFTWARE_PREFIX in that #768

Open

Neves-P self-requested a review July 7, 2026 08:16

Neves-P requested changes Jul 20, 2026

View reviewed changes

Fix typo

e0724fa

Neves-P added 5 commits July 20, 2026 16:04

Fix missing word

7bef7b8

Fix markdown

51a0d0a

Add line break

0cc3ec7

Bash highlight

bf3f6f8

Use monospaced font for path

e18965a

casparvl commented Jul 22, 2026

View reviewed changes

Neves-P requested changes Aug 4, 2026

View reviewed changes

Neves-P requested changes Aug 7, 2026

View reviewed changes

Comment thread docs/site_build/site_cvmfs.md Outdated

Neves-P mentioned this pull request Aug 28, 2026

Clean up dev.eessi.io-specific code (assume it's doing a site installation) and improve tarball creation step for site installations EESSI/software-layer-scripts#295

Merged

Neves-P added 2 commits September 1, 2026 15:28

Fix link

daeb7b0

Fix link

2bc09af

bedroge reviewed Sep 2, 2026

View reviewed changes

Comment thread docs/site_build/site_cvmfs.md Outdated

bedroge and others added 2 commits September 2, 2026 09:53

Apply batched suggestions from code review

3547ee9

Co-authored-by: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com>

Apply batched suggestions from code review

2296339

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

bedroge reviewed Sep 15, 2026

View reviewed changes

Comment thread docs/site_build/site_cvmfs.md Outdated

Update docs/site_build/site_cvmfs.md

579898c

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>

Neves-P requested changes Sep 18, 2026

View reviewed changes

Neves-P requested a review from bedroge September 18, 2026 13:02


		Another thing to consider is to create a secondary user with `aws iam create-user --user-name <name>` and attach a very limited policy to it (e.g. only read/write/list on buckets, nothing else). Then, create credentials for this user with `aws iam create-access-key --user-name <name>` and provide those credentials to the EESSI build bot and Stratum 0 machines. That way, if that token is compromized, the impact is minimized (e.g. the token can at least not be used to create new IAM idententies, etc).

		## Setting up the EESSI build bot

-# a name for a dir in the bucket in which to archive tarballs, so that a subsequent run doesn't re-ingest them
+# Repository-relative path to the versions directory.
+# This must match the versions_subpath used for .cvmfsdirtab.
+VERSIONS_SUBPATH="versions"
+# For a site prefix such as /cvmfs/name.sitename.tld/eessi, use:
+# VERSIONS_SUBPATH="eessi/versions"
+# a name for a dir in the bucket in which to archive tarballs, so that a subsequent run doesn't re-ingest them



	`VERSIONS_SUBPATH` must match the `versions_subpath` used when [configuring `.cvmfsdirtab`](#setting-up-your-stratum-0). It is relative to the root of the target CVMFS repository and must include the `versions` directory.

	For example, with `EESSI_SITE_SOFTWARE_PREFIX=/cvmfs/name.sitename.tld/eessi`, use `VERSIONS_SUBPATH=eessi/versions`. The default is `versions`.


		5. Ingest the tarball into the repository

		Here, we leverage a script from `EESSI/filesystem-layer` that ingests tarballs, but also takes care of regenerating the `.cvmfscatalog` files _and_ updates the `Lmod` cache. To update the `Lmod` cache, the script uses the Lmod installation provided by `software.eessi.io`, which is why we explicitly made this available as one of the steps [when we set up our Stratum 0](#setting-up-your-stratum-0)

-Here, we leverage a script from `EESSI/filesystem-layer` that ingests tarballs, but also takes care of regenerating the `.cvmfscatalog` files _and_ updates the `Lmod` cache. To update the `Lmod` cache, the script uses the Lmod installation provided by `software.eessi.io`, which is why we explicitly made this available as one of the steps [when we set up our Stratum 0](#setting-up-your-stratum-0)
+Here, we leverage [`ingest-tarball.sh`](https://github.com/EESSI/filesystem-layer/blob/main/scripts/ingest-tarball.sh) from `EESSI/filesystem-layer`. The script ingests the tarball, regenerates the `.cvmfscatalog` files, and updates the Lmod cache.
+The tarball contains paths relative to the versioned installation directory, such as `2025.06/software/...`. The `--basedir` option determines where that directory is placed relative to the root of the target CVMFS repository. It must therefore match the repository layout configured through `VERSIONS_SUBPATH`.
+When updating the Lmod cache, the script uses the selected base directory for the site installation. If the site repository does not provide its own compatibility layer, it automatically falls back to an Lmod installation from `/cvmfs/software.eessi.io/versions`.


		6. Regenerate the `.cvmfscatalog` files by publishing an empty transaction

		This is already taken care of by the `$INGEST_SCRIPT` in the step above. If you don't want to use that script, you'll have to implement this step yourself.

	This is already taken care of by the `$INGEST_SCRIPT` in the step above. If you don't want to use that script, you'll have to implement this step yourself.
	This is already taken care of by `$INGEST_SCRIPT` in the step above. The patterns in `.cvmfsdirtab` must use the same repository-relative versions path as `VERSIONS_SUBPATH`; otherwise the expected nested catalogs will not be created.


		7. Open a new transaction, update the Lmod cache for your site installs, and publish the transaction

		This is already taken care of by the `$INGEST_SCRIPT` in step 5 above. If you don't want to use that script, you'll have to implement this step yourself.

Conversation

casparvl commented Jun 4, 2026

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Neves-P left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

casparvl Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Neves-P Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Neves-P left a comment

Choose a reason for hiding this comment

Uh oh!

Neves-P Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Neves-P left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Neves-P commented Sep 1, 2026

Uh oh!

Uh oh!

Uh oh!

Neves-P left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

casparvl Jul 22, 2026 •

edited

Loading

Neves-P Aug 4, 2026 •

edited

Loading

Neves-P Aug 4, 2026 •

edited

Loading