`--rlimit_fsize soft` results in `write error: File too large`

[mkst]

Hi folks,

We’ve been using nsjail 3.4 for a long time in decomp.me. I’m testing an update to nsjail 3.6 and I’m seeing commands fail with:

/bin/bash: line 1: echo: write error: File too large

In another run, after setting --rlimit_fsize 100, the process terminated with signal 25 (SIGXFSZ).

The surprising part is that our existing config used:

--rlimit_fsize soft

Inside the jail, that reports a near-unlimited value:

cat /proc/self/limits | grep -i "file size"
Max file size 18446744073708503040 18446744073708503040 bytes

Whereas an explicit value works as expected:

--rlimit_fsize 1000

reports:

Max file size 1048576000 1048576000 bytes

Setting --rlimit_fsize 1024 avoids the error for this case.

We are running nsjail inside Docker. The sandbox work directory is under a Docker tmpfs mount:

tmpfs:
  - /sandbox/tmp:exec,uid=1000,gid=1000,size=64M,mode=0700

and we bind a temporary subdirectory of that into the jail as /tmp, /var/tmp, and /run/user/1000.

Example invocation:

WINE=wine \
WIBO=wibo \
WIBO_PATH=/backend/compilers/common/wibo_dlls \
INPUT=/tmp/code.c \
OUTPUT=/tmp/object.o \
COMPILER_DIR=/backend/compilers/ps1/gcc2.5.7-psx \
COMPILER_FLAGS=-O2 \
FUNCTION='' \
MWCIncludes=/tmp \
TMPDIR=/tmp \
/bin/nsjail36 \
  --mode o \
  --chroot /sandbox/root \
  --bindmount /sandbox/tmp/tmp24qbotun:/tmp \
  --bindmount /sandbox/tmp/tmp24qbotun:/run/user/1000 \
  --bindmount /sandbox/tmp/tmp24qbotun:/var/tmp \
  --bindmount_ro /dev \
  --bindmount_ro /bin \
  --bindmount_ro /etc/alternatives \
  --bindmount_ro /etc/fonts \
  --bindmount_ro /etc/passwd \
  --bindmount_ro /lib \
  --bindmount_ro /lib32 \
  --bindmount_ro /lib64 \
  --bindmount_ro /usr \
  --bindmount_ro /proc \
  --bindmount_ro /sys \
  --bindmount_ro /backend/compilers \
  --bindmount_ro /backend/libraries \
  --env PATH=/bin:/usr/bin \
  --cwd /tmp \
  --rlimit_fsize soft \
  --rlimit_nofile soft \
  --bindmount_ro /tmp/wine:/wine \
  --bindmount /sandbox/tmp/tmp24qbotun/Temp:/wine/drive_c/users/ubuntu/Temp \
  --env WINEDEBUG=-all \
  --env WINEPREFIX=/wine \
  --disable_proc \
  --really_quiet \
  --bindmount_ro /backend/compilers/ps1/gcc2.5.7-psx \
  --env WINE=wine \
  --env WIBO=wibo \
  --env WIBO_PATH=/backend/compilers/common/wibo_dlls \
  --env INPUT=/tmp/code.c \
  --env OUTPUT=/tmp/object.o \
  --env COMPILER_DIR=/backend/compilers/ps1/gcc2.5.7-psx \
  --env COMPILER_FLAGS=-O2 \
  --env FUNCTION= \
  --env MWCIncludes=/tmp \
  --env TMPDIR=/tmp \
  -- \
  /bin/bash -euo pipefail -c '
    /usr/bin/cpp -E -lang-c -nostdinc "${INPUT}" -o "${INPUT}".i &&
    eval "${COMPILER_DIR}/gcc ${COMPILER_FLAGS} -c -pipe -B${COMPILER_DIR}/ -o "${OUTPUT}" "${INPUT}".i"
  '

Inside the jail, /tmp is backed by the Docker tmpfs as expected:

findmnt /tmp
TARGET SOURCE              FSTYPE OPTIONS
/tmp   tmpfs[/tmp24qbotun] tmpfs  rw,nosuid,nodev,relatime,size=65536k,mode=700,uid=1000,gid=1000,inode64

The same setup worked with nsjail 3.4.

Questions:

1. Did the handling of --rlimit_fsize soft change between 3.4 and 3.6?
2. Is the near-UINT64_MAX value expected for an inherited unlimited soft limit?
3. Could the mount handling changes in 3.6 affect this case, given that /tmp is a bind mount of a subdirectory inside a Docker tmpfs?
4. Would a git bisect between 3.4 and 3.6 be useful?

Happy to provide more logs or test a patch.

[mkst]

Just been looking through the commits and came across dc5338c. Which looks like the smoking gun.

I'm trying to find the "correct" value for --rlimit_fsize -> if I set "too low", e.g., 100MB, then the compiler(s) will exit with EFBIG / SIGXFSZ.

EDIT:

• Setting --rlimit_fsize soft and running ulimit -f returns unlimited on nsjail 3.4
• Setting --rlimit_fsize soft and running ulimit -f returns 18014398509480960 on nsjail 3.6

[robertswiecki]

Does it still happen, there were some changes inbetween which might have fixed it - maybe.