Skip to content

Use vg WDL that gives RTG a defined memory limit #4918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamnovak wants to merge 8 commits into
base: master
Choose a base branch
from
+11 −6
Open

Use vg WDL that gives RTG a defined memory limit #4918

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
64152eb
Use vg WDL that gives RTG a defined memory limit
adamnovak May 29, 2026
54568a2
Pass SDF through and avoid making it again
adamnovak Jun 1, 2026
3183c02
Try to make build use simpler logic to get at build args
adamnovak Jun 1, 2026
2917249
Add extra ARG in case Docker docks lie about scoping
adamnovak Jun 1, 2026
1e13643
Remove extra echo and explain arg shenanigans
adamnovak Jun 1, 2026
4bb31f7
Move system Node out of the way of new Node
adamnovak Jun 1, 2026
21ad36c
Upgrade nodejs used for vg testing
adamnovak Jun 1, 2026
9114e13
Remove system Node packages that break new Node install
adamnovak Jun 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ RUN apt-get -qq -y update && apt-get -y install --no-upgrade \
###DEPS_END###

FROM packages AS build
# TODO: We're *supposed* to inherit ARGs from our base stage according to the
# Docker documentation, but we don't.
ARG THREADS=8

RUN echo build > /stage.txt

Expand All @@ -61,17 +64,19 @@ RUN if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then sed -i s/m
RUN find . -name CMakeCache.txt | xargs rm -f
# Build the dependencies
COPY Makefile /vg/Makefile
RUN CXXFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" CFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" make -j $((THREADS < $(nproc) ? THREADS : $(nproc))) deps
# Remember that THREADS is an ARG and not a Bash variable, and it gets
# substituted in using Bash-ish syntax *before* the command gets to Bash.
RUN CXXFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" CFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" make -j $(( $THREADS < $(nproc) ? $THREADS : $(nproc) )) deps

# Bring in the sources, which we need in order to build.
COPY src /vg/src

# Build all the object files for vg, but don't link.
# Also pass the arch here
RUN CXXFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" make -j $((THREADS < $(nproc) ? THREADS : $(nproc))) objs
RUN CXXFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" make -j $(( $THREADS < $(nproc) ? $THREADS : $(nproc) )) objs

# Do the final build and link, knowing the version. Trim down the resulting binary but make sure to include enough debug info for profiling.
RUN CXXFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" make -j $((THREADS < $(nproc) ? THREADS : $(nproc))) static && strip -d bin/vg
RUN CXXFLAGS="$(if [ -z "${TARGETARCH}" ] || [ "${TARGETARCH}" = "amd64" ] ; then echo " -march=nehalem "; fi)" make -j $(( $THREADS < $(nproc) ? $THREADS : $(nproc) )) static && strip -d bin/vg

# Ship the scripts
COPY scripts /vg/scripts
Expand All @@ -84,7 +89,7 @@ ARG THREADS=8

RUN echo test > /stage.txt

RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - && apt-get -qq -y install nodejs && npm install -g txm@7.4.5
RUN apt-get -qq -y remove libnode-dev libnode72 && curl -sL https://deb.nodesource.com/setup_20.x | bash - && apt-get -qq -y install nodejs && npm install -g txm@7.4.5

# Fail if any non-portable instructions were used
RUN /bin/bash -e -c 'if objdump -d /vg/bin/vg | grep vperm2i128 ; then exit 1 ; else exit 0 ; fi'
Expand All @@ -97,7 +102,7 @@ COPY README.md /vg/

# Run tests in the middle so the final container that gets tagged is the run container.
# Tests may not actually be run by smart builders like buildkit.
RUN /bin/bash -e -c "export OMP_NUM_THREADS=$((THREADS < $(nproc) ? THREADS : $(nproc))); make test"
RUN /bin/bash -e -c "export OMP_NUM_THREADS=$(( $THREADS < $(nproc) ? $THREADS : $(nproc) )); make test"


############################################################################################
Expand Down
2 changes: 1 addition & 1 deletion vgci/vgci.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ def _giraffe_dv_run(self, sample_name, chrom, gbz_path, fq_path,
job_store = self._jobstore(tag)
out_store = self._outstore(tag)

workflow_spec = "https://raw.githubusercontent.com/vgteam/vg_wdl/5006823c6693ba5f6e41a4206dcf970a9eaa0b11/workflows/giraffe_and_deepvariant.wdl"
workflow_spec = "https://raw.githubusercontent.com/vgteam/vg_wdl/09739c8eb160777073a95ceea2f36347a8d79df3/workflows/giraffe_and_deepvariant.wdl"

output_json = os.path.join(out_store, "output.json")
input_json = "input.json"
Expand Down
Loading