Skip to content

perf(build): use zstd instead of gzip for weblog image compression #7011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rochdev wants to merge 1 commit into
base: main
Choose a base branch
from
Open

perf(build): use zstd instead of gzip for weblog image compression #7011

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion utils/_context/containers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -958,7 +958,7 @@ def get_image_list(self, library: str | None, weblog: str | None) -> list[str]:
"""Returns images needed to build the weblog"""

# If an image is saved as a file in binaries, we don't need any image
filename = f"binaries/{library}-{weblog}-weblog.tar.gz"
filename = f"binaries/{library}-{weblog}-weblog.tar.zst"
if Path(filename).is_file():
return []

Expand Down
10 changes: 5 additions & 5 deletions utils/build/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ print_usage() {
echo -e " ${CYAN}--default-weblog${NC} Prints the name of the default weblog for a given library and exits."
echo -e " ${CYAN}--binary-path${NC} Optional. Path of a directory binaries will be copied from. Should be used for local development only."
echo -e " ${CYAN}--binary-url${NC} Optional. Url of the client library redistributable. Should be used for local development only."
echo -e " ${CYAN}--save-to-binaries${NC} Optional. Save image in binaries folder as a tar.gz file."
echo -e " ${CYAN}--save-to-binaries${NC} Optional. Save image in binaries folder as a tar.zst file."
echo -e " ${CYAN}--help${NC} Prints this message and exits."
echo
echo -e "${WHITE_BOLD}EXAMPLES${NC}"
Expand Down Expand Up @@ -257,11 +257,11 @@ build() {
fi

# keep this name consistent with WeblogContainer.get_image_list()
BINARIES_FILENAME=binaries/${TEST_LIBRARY}-${WEBLOG_VARIANT}-weblog.tar.gz
BINARIES_FILENAME=binaries/${TEST_LIBRARY}-${WEBLOG_VARIANT}-weblog.tar.zst

if [ -f $BINARIES_FILENAME ]; then
if [ -f "$BINARIES_FILENAME" ]; then
echo "Loading image from $BINARIES_FILENAME"
docker load --input $BINARIES_FILENAME
zstd -d -c "$BINARIES_FILENAME" | docker load
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Load zstd artifacts through Docker directly

In environments that have Docker but not the zstd CLI, the downloaded weblog artifact can no longer be loaded because this path now shells out to zstd before docker load; the run-end-to-end.yml/debug-harness.yml test jobs download binaries/ and call ./build.sh without installing that tool. Docker's own load command documents that it can read tar archives compressed with gzip, bzip2, xz, or zstd from --input/STDIN, so using docker load --input "$BINARIES_FILENAME" here would preserve the new artifact format without adding a new runtime dependency to every artifact consumer.

Useful? React with 👍 / 👎.

else

if [[ $TEST_LIBRARY == python ]]; then
Expand Down Expand Up @@ -339,7 +339,7 @@ build() {

if [[ $SAVE_TO_BINARIES == 1 ]]; then
echo "Saving image to $BINARIES_FILENAME"
docker save system_tests/weblog | gzip > $BINARIES_FILENAME
docker save system_tests/weblog | zstd > "$BINARIES_FILENAME"
fi
fi
elif [[ $IMAGE_NAME == lambda-proxy ]]; then
Expand Down
Loading