Skip to content

Fetch all source files before processing easystack file #283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bedroge wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Fetch all source files before processing easystack file #283

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
1fe6501
fetch all sources before installing software from easystack file
bedroge Aug 7, 2026
8337cb9
add easystack
bedroge Aug 7, 2026
8c01b01
add from-commit
bedroge Aug 7, 2026
e9b6583
remove parallel
bedroge Aug 7, 2026
ac3d104
remove failing item
bedroge Aug 7, 2026
d7679ca
remove easystack file
bedroge Aug 7, 2026
d371a51
check if eb --fetch-all is available, otherwise use --fetch
bedroge Aug 7, 2026
38edc87
disable fetching sources if EESSI_SKIP_FETCH_EASYSTACK_SOURCES is set
bedroge Aug 18, 2026
3488625
Merge branch 'main' into fetch_first
bedroge Aug 18, 2026
fe3e3cc
Apply suggestion from @boegel
bedroge Sep 16, 2026
44d0b2f
only fetch sources for added items
bedroge Sep 16, 2026
665d132
clean up temporary easystack file
bedroge Sep 16, 2026
10abf84
let awk print the first line of the easystack as well
bedroge Sep 16, 2026
a69e383
use empty list if there are no additions in an easystack file
bedroge Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions EESSI-install-software.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,37 @@ else
echo_green "All set, let's start installing some software with EasyBuild v${eb_version} in ${EASYBUILD_INSTALLPATH}..."

if [ -f ${easystack_file} ]; then
if [ -z $EESSI_SKIP_FETCH_EASYSTACK_SOURCES ]; then
# run eb --help to determine if --fetch-all is supported (available since 5.3.0), otherwise fall back to --fetch
fetch_option="--fetch"
${EB} --help | grep -q -e "fetch-all" && fetch_option="--fetch-all"
easystack_additions=$(mktemp --suffix=.yml)
# use the PR diff file to find which items have been added to this particular easystack file
# and dump that to a temporary easystack file that we can use to fetch the sources for only these added items
awk -v file="${easystack_file}" '

@boegel boegel Sep 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure awk is available everywhere we need it to be?

@bedroge bedroge Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both the compat layer and container have awk.

/^diff --git / { found=0 }
$0 == "+++ b/" file { found=1; next }
found && /^\+/ && !/^\+\+\+/ {
sub(/^\+/, "")
additions = additions $0 "\n"
count++
}
END {
if (count)
printf "easyconfigs:\n%s", additions
else
print "easyconfigs: []"
}
' "${diff_file}" >> ${easystack_additions}
echo_green "Downloading sources for added items in easystack file ${easystack_file} using eb ${fetch_option}..."
echo '(note: this step can be skipped by setting $EESSI_SKIP_FETCH_EASYSTACK_SOURCES to a non-empty value)'
${EB} ${fetch_option} --easystack ${easystack_additions} --robot
if [ $? -ne 0 ]; then
fatal_error "Could not download all required source files for this easystack file."
fi
rm ${easystack_additions}
fi

echo_green "Feeding easystack file ${easystack_file} to EasyBuild..."

if [[ ${easystack_file} == *"/rebuilds/"* ]]; then
Expand Down
Loading