Skip to content

test: ui-smoke phase 2, g-code execution and endpoint check#4054

Open
grandixximo wants to merge 10 commits into
LinuxCNC:masterfrom
grandixximo:ui-tests-phase2
Open

test: ui-smoke phase 2, g-code execution and endpoint check#4054
grandixximo wants to merge 10 commits into
LinuxCNC:masterfrom
grandixximo:ui-tests-phase2

Conversation

@grandixximo
Copy link
Copy Markdown
Contributor

@grandixximo grandixximo commented May 24, 2026

Phase 2 of the GUI test work tracked in #3756. Stacked on #3999 (now merged).

Summary

Extends each per-GUI ui-smoke test landing in #3999 with an end-to-end g-code execution stage:

  • Estop reset, machine on, home all (via c.home(-1), respects HOME_SEQUENCE).
  • MODE_AUTO, program_open on a shared _lib/smoke.ngc, auto(AUTO_RUN, 0).
  • Poll linuxcnc.stat until interp_state == INTERP_IDLE and queue == 0 for 5 consecutive 10 ms polls (settle window guards the inter-line moment where interp briefly reports IDLE while queueing the next move).
  • Assert stat.position[:3] delta against --expect-delta-mm 1,1,0, converted via stat.linear_units so the same arg works on inch (axis, touchy) and mm (gmoccapy, qtdragon) sims.

No new test directories. The four existing tests/ui-smoke/{axis,touchy,gmoccapy,qtdragon}/test.sh now also pass the --run-program and --expect-delta-mm flags through run-gui.sh. The Phase 1 connect-and-settle path stays as-is when those flags are omitted.

smoke.ngc

G21
G91
G0 X1 Y1
G90
M2

Forces mm input units (G21) and uses relative motion (G91) so the same file is sim-agnostic. The driver records stat.position[:3] after homing and checks (final - start) against the expected delta converted to machine units. This sidesteps each sim's HOME offset (axis homes to 0/0/0, qtdragon to 20/20/-10, etc).

State machine handling

linuxcnc.command.wait_complete() on a state or mode change only proves the NML message was acked, not that task_state / task_mode has transitioned. Polling the stat fields is the only deterministic signal. Two GUIs (gmoccapy reliably, qtdragon intermittently) re-issue their own mode commands during their own startup and revert task_mode AUTO -> MANUAL immediately after the driver sets it; without retry, auto(AUTO_RUN, 0) is then rejected and interp_state stays at IDLE.

ensure_state / ensure_mode helpers implement an issue + wait + stability-check + retry pattern, up to STATE_RETRY_BUDGET = 6 attempts. Intermediate timeouts use a quiet wait variant so spurious UI_SMOKE_FAIL lines do not pollute the log during retries (checkresult.sh greps for ^UI_SMOKE_FAIL on any line, so a retry that ultimately succeeds must not emit the marker).

qtdragon-specific bits

The qtdragon sim needs three CI-only workarounds layered in tests/ui-smoke/qtdragon/test.sh:

  1. Writable config mirror. qtvcp's INI-driven LOG_FILE is rooted in the config dir. CI mounts the workspace read-only for the runtime user, so a relative LOG_FILE = qtdragon.log resolves to a path qtvcp cannot create and hal_bridge exits before the driver can attach. The test now mirrors configs/sim/qtdragon/qtdragon_xyz/ into mktemp -d and rewrites LOG_FILE to ~/qtdragon.log.
  2. Offscreen Qt platform. qtvcp under xvfb + xcb on Ubuntu 24.04 segfaults during widget construction (no backtrace). Setting QT_QPA_PLATFORM=offscreen and LINUXCNC_OPENGL_PLATFORM=offscreen renders entirely in memory; xvfb-run still wraps the call so scripts/linuxcnc's X-display assumptions hold.
  3. Block QtWebEngine import. qtdragon's UI embeds WebWidget (QWebEngineView, Chromium). Chromium's browser-process init segfaults inside the qtvcp PID under offscreen + xvfb on the Ubuntu runner even with --no-sandbox --single-process --disable-gpu (Chromium logs Sandboxing disabled by user. and then crashes). Rather than chase Chromium flags for a widget the smoke test never touches, the test drops a sitecustomize.py on PYTHONPATH that installs a sys.meta_path finder blocking qtpy.QtWebEngineWidgets (and PyQt5.QtWebEngineWidgets). lib/python/qtvcp/widgets/web_widget.py already has a fail-safe path that swaps the QWebEngineView for a plain QWidget when that import fails, so the UI loads cleanly with the Web tab inert.

debian/control

Adds python3-zmq under the <!nocheck> profile. Without it qtdragon's hal_bridge fails on startup and qtvcp tears down before the driver can attach.

Performance

ui-smoke total wall time on this machine:

Tests Wall Marginal over Phase 1 baseline (~2 min)
Phase 1 only (#3999) 4 ~2 min baseline
Phase 1 + Phase 2 (this PR) 4 ~2m43s +~40s

Phase 2 reuses each test's existing linuxcnc startup and xvfb instance; CI cost is the Phase 2 sequence per GUI (estop + home + mode + run + verify, ~5 to 16 s depending on GUI), not the full ~30 s startup/shutdown overhead each test pays once.

Sequential by necessity: every ui-smoke test launches linuxcnc which claims a fixed set of SHM keys (SHM_KEYS in _lib/cleanup-runtime.sh), so parallel ui-smoke runs would race. Per-test SHM-key isolation would be a separate refactor.

Test plan

  • 5 consecutive local scripts/runtests tests/ui-smoke runs, 4/4 pass, 0 shmem errors, clean tree
  • Per-test diagnostics confirm gmoccapy hits the mode-revert path on every run and the retry recovers cleanly
  • scripts/shellcheck.sh clean on changed scripts
  • python3 -m py_compile clean on drive.py
  • CI: rip-and-test, rip-and-test-clang, rip-rtai, and all 6 package-arch / 3 package-indep jobs green on run 26361844630

Out of scope

@grandixximo grandixximo marked this pull request as ready for review May 25, 2026 03:13
@grandixximo grandixximo marked this pull request as draft May 27, 2026 12:10
@grandixximo
Copy link
Copy Markdown
Contributor Author

I will do more testing locally on an ubuntu 24.04 before marking as ready

grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
grandixximo added a commit to grandixximo/linuxcnc that referenced this pull request May 28, 2026
Three debug-only additions to diagnose the rip-and-test-clang failure
(homing timeout after 60s + Fatal glibc pthread mutex assertion on
shutdown) which my local docker cannot reproduce:

- launch.sh: PYTHONFAULTHANDLER=1, ulimit -c unlimited, LIBC_FATAL_
  STDERR_=1, MALLOC_CHECK_=3 so SIGABRT/SIGSEGV in any Python child
  prints a Python+native stack to stderr (visible via linuxcnc.err).
- qtvcp.py: faulthandler.enable() + register on SIGUSR1 so the smoke
  driver can dump qtvcp's interpreter stack without killing it.
- drive.py: on homing timeout, dump per-joint state, halui machine
  pin, locate qtvcp processes and send SIGUSR1; sleep briefly so the
  stack dump lands in the log before we tear down.

Will be reverted once the clang-only failure mode is understood.
@grandixximo
Copy link
Copy Markdown
Contributor Author

CI will fail on this branch until a fix for the qtdragon ui-smoke dbus
segfault lands on master. Root cause is a half-initialized
dbus.SessionBus in qtvcp/lib/sys_notify.py whose mainloop-attached
QSocketNotifier keeps the C connection alive after init raises
ServiceUnknown; the next Qt event-loop tick dispatches into
_dbus_list_unlink and SEGVs. Captured under gdb, full backtrace in
#4064 which proposes the one-line fix. Idling this PR (no further
pushes) until #4064 or an equivalent fix is merged, then I will rebase.

@grandixximo
Copy link
Copy Markdown
Contributor Author

Addendum to my quit-path question above: I built it, so it is concrete now.

Shared harness plus per-GUI tests (touchy, gmoccapy, qtdragon): boot the GUI, wait for the task, send SIGTERM to the GUI alone, assert it exits in a short grace. Validated on a uspace RIP build under xvfb: touchy fails without its handler and passes with it; qtdragon and gmoccapy pass with theirs. They depend on #4076 / #4077 / #4078, so they fail on current master by design until those land.

Worth noting: neither phase catches a GUI wedged in its constructor. gmoccapy on a fresh profile blocks in a modal startup dialog (#4072) and never reaches its main loop, yet both phases report UI_SMOKE_OK since only the NML task is checked. The quit test catches it: a constructor-blocked GUI never installs its handler, so it cannot exit on SIGTERM. So it guards two things, signal handling and "the GUI reached its event loop". I did only the SIGTERM path; a clean GUI-native quit is per-toolkit and messier, left for later.

Packaging: fold into this PR, or a separate follow-up after the three SIGTERM fixes? I lean follow-up to keep this PR focused, but either works.

c-morley pushed a commit that referenced this pull request May 30, 2026
qtvcp registered self.shutdown as the SIGTERM/SIGINT handler, but two
things stopped it working: Qt's C++ event loop (APP.exec()) does not
return to Python often enough to run Python signal handlers, and
shutdown() cleans up without quitting the loop, so even when reached the
process kept running. A caller tearing qtvcp down therefore had to
escalate to SIGKILL.

Quit the event loop on the signal (APP.quit()) so APP.exec() returns and
the existing post-exec shutdown() runs the normal cleanup once, and add a
periodic no-op QTimer so the interpreter gets a chance to run the signal
handler during exec(). Quitting the loop bypasses the window-close
confirmation dialog, which is correct for a terminate request and leaves
that interactive feature untouched.

Fixes the hang for all qtvcp screens (qtdragon, qtplasmac, ...). Verified
qtdragon exits cleanly on SIGTERM and SIGINT.

Surfaced by the ui-smoke tests (PR #4054).
@grandixximo
Copy link
Copy Markdown
Contributor Author

Just rebasing, extended UI smoke quit will probably be a followup PR

@grandixximo
Copy link
Copy Markdown
Contributor Author

grandixximo commented Jun 1, 2026

There has been a regression in #4088 for headless systems, the phase 2 CI would have caught it, since it fails now, will send PR to fix it, once merged phase 2 shall pass again, the smoke test is actually already catching stuff, doing its job properly :-)

BsAtHome pushed a commit that referenced this pull request Jun 1, 2026
touchy.ini was the only sim config shipping a non-zero DEBUG (0x10 =
EMC_DEBUG_TASK_ISSUE). That flag makes task log every issued NML command
as an ASCII string via emcCommandBuffer->msg2str(), which lazily creates
a CMS_DISPLAY_ASCII_UPDATER and prints its scary 'may not function
properly due to range limitations' banner on startup.

The emcCommand buffer is correctly xdr-encoded; the ASCII updater is only
a temporary string-conversion helper for the debug log. Setting DEBUG=0
(matching axis and gmoccapy sims) stops the task-issue logging and the
spurious banner. No functional change to touchy.

Surfaced by the ui-smoke tests (#4054).
BsAtHome pushed a commit that referenced this pull request Jun 1, 2026
gtk_action.py had 'return npath' inside a finally: block. A return in
finally swallows any in-flight exception and overrides earlier returns,
so the except branch's 'return None' (the error path) was always
clobbered: a failed save still returned the path and hid the exception.

Dedent the return out of the finally so the finally only closes the
file. On success the function returns npath; on a write error it now
correctly returns None. Also silences the Python 3.12+
SyntaxWarning: 'return' in a 'finally' block.

Surfaced by the ui-smoke tests (PR #4054).

Fixes #4067
BsAtHome pushed a commit that referenced this pull request Jun 1, 2026
touchy ignored SIGTERM (GLib/PyGObject absorbs the signal), so a caller
tearing it down had to escalate to SIGKILL. Install a SIGTERM handler
that quits the GTK main loop, deferring the quit to the loop via
GLib.idle_add since GTK calls are not safe directly from a signal
handler. atexit cleanup (child processes, prefs) still runs.

This does not touch the interactive window-close path. SIGTERM is a
system terminate request and should not require confirmation.

Note: PyGObject still prints a benign KeyboardInterrupt on this shutdown;
that is pre-existing (touchy printed it on SIGTERM before too, it just
did not exit) and is emitted from PyGObject's C layer, so it is not
suppressible from Python here.

Surfaced by the ui-smoke tests (PR #4054).
andypugh pushed a commit that referenced this pull request Jun 2, 2026
gmoccapy did not exit on SIGTERM; callers had to escalate to SIGKILL.

Root cause: PyGObject's signal bridge surfaces a termination signal into
the GTK main loop as a Python KeyboardInterrupt. That propagated to the
installed sys.excepthook, which unconditionally popped a modal
'Found an error ... KeyboardInterrupt' dialog and blocked in the dialog's
nested loop, so the process never quit.

Handle KeyboardInterrupt at the top of excepthook: log it and quit the
main loop instead of showing the modal error dialog. Verified that
gmoccapy now exits cleanly on SIGTERM (no SIGKILL escalation, no spurious
error-dialog traceback). The previous broken 'except KeyboardInterrupt'
on the window lookup (which could never fire there) is removed.

Surfaced by the ui-smoke tests (PR #4054).

Fixes #4069
@andypugh
Copy link
Copy Markdown
Collaborator

andypugh commented Jun 2, 2026

It seems to be failing CI on QTDragon.

It's awkward because this looks to be good code, doing the job of finding errors. But that very fact makes it fail CI.

(It's also failing clang, it seems?)

@grandixximo
Copy link
Copy Markdown
Contributor Author

I need to rebase it, I think

Each per-GUI test now also drives estop reset, machine on, home all,
mode auto, program_open + auto(RUN) on a tiny shared smoke.ngc, waits
for sustained INTERP_IDLE, and asserts stat.position delta against
--expect-delta-mm 1,1,0 converted via stat.linear_units so the same
arg works on inch (axis, touchy) and mm (gmoccapy, qtdragon) sims.

State/mode commands use ensure_state/ensure_mode helpers with a
retry-and-stability pattern: gmoccapy and qtdragon re-issue their own
mode commands during startup and can revert task_mode AUTO -> MANUAL
right after we set it. The helpers wait for the desired state, then
re-check after STATE_STABILITY_S; on revert they retry up to
STATE_RETRY_BUDGET times. Intermediate timeouts use a quiet variant
so spurious UI_SMOKE_FAIL lines do not pollute the log during retries
(checkresult.sh greps for ^UI_SMOKE_FAIL on any line).

smoke.ngc is G21 G91 G0 X1 Y1 G90 M2 - relative move in mm, sim-
agnostic. The driver snapshots stat.position[:3] after homing and
checks (final - start) against the converted delta, sidestepping each
sim's HOME offset.

Adds python3-zmq and python3-opencv to debian/control.top.in under
!nocheck: qtdragon's hal_bridge and the camview widget segfault on
startup without them, which is invisible to the connect-only Phase 1
smoke but breaks the run-program path before the program can start.

5 consecutive local runs all green at 2m43s wall each.
@BsAtHome
Copy link
Copy Markdown
Contributor

BsAtHome commented Jun 2, 2026

There still is a RT/non-RT race somewhere in the homing. This error seems to be quite transient and appears to have a timing sensitivity:

[0](https://github.com/LinuxCNC/linuxcnc/actions/runs/26823637681/job/79084814062?pr=4054#step:7:341)
        === ui-smoke.err ===
        WARN: re-issued home(-1); homed=[0, 0, 0] task_state=4 task_mode=1
        WARN: re-issued home(-1); homed=[0, 0, 0] task_state=4 task_mode=1
        UI_SMOKE_FAIL: timeout waiting for all joints homed after 60.0s; homed=[0, 0, 0] task_state=4 task_mode=1 exec_state=2 njoints=3

It was also seen other places when CI was busy. This does indicate a timing race.

@grandixximo grandixximo force-pushed the ui-tests-phase2 branch 2 times, most recently from fdd3f2f to 5d56a14 Compare June 3, 2026 01:23
CI run hit a PermissionError in qtvcp's logger when it tried to open
configs/sim/qtdragon/qtdragon_xyz/qtdragon.log for write: the GitHub
Actions workspace is mounted read-only for the docker build user, and
qtvcp resolves LOG_FILE = qtdragon.log into the config dir. hal_bridge
then exits, linuxcnc tears down, and the driver retries ESTOP_RESET
until the budget is exhausted.

qtdragon test.sh now mirrors the qtdragon_xyz config dir to a mktemp
directory, seds LOG_FILE to ~/qtdragon.log, and passes the absolute
INI path to run-gui.sh. run-gui.sh treats any path starting with /
as absolute; everything else still resolves under configs/sim. Trap
cleans the tmp dir on exit so the working tree stays clean.

Does not touch the shipped qtdragon config to avoid changing default
behaviour for real users. The same fix would work for any other
config that turns out to write into its own dir on CI.
Ubuntu 24.04 rip-and-test runs hit a qtvcp segfault after the log-
permission fix let qtvcp get further than Phase 1 had. Debian
package-arch passes the same code. Two known asymmetries match:

- python3-opencv on Ubuntu pulls Qt5 GUI bits whose cv2/qt/plugins
  directory overrides the system PyQt5 platform plugin path under
  xvfb (opencv-python issue LinuxCNC#572, Qt Forum 119109). qtvcp's
  camview_widget tolerates ImportError on cv2 and just logs a
  warning, so dropping the dep restores the harmless fallback path
  Phase 1 was already exercising.
- xcb_glx is the historical fragile integration under xvfb
  (Launchpad #1761708, QTBUG-67537); xcb_egl is what software-GL
  stacks expect anyway. Set as defense in depth.

Local 4/4 still green with both changes.
xvfb + xcb + xcb_egl was not enough for Ubuntu 24.04 rip-and-test:
qtvcp still segfaults during widget construction even with opencv
and qtwebengine paths quiet, and the same code passes on Debian
package-arch. Offscreen renders entirely in memory and exercises a
different Qt plugin entirely, dodging the xcb-stack instability.

scripts/linuxcnc itself forces QT_QPA_PLATFORM=xcb unless
LINUXCNC_OPENGL_PLATFORM is set to a non-glx value, so pin both.
Only qtdragon needs this; axis (Tk), touchy and gmoccapy (GTK) are
unaffected. Trade-off: no Phase 3 screenshot from qtdragon under
this config; Phase 3 would need an opt-out for offscreen tests.
qtdragon embeds QWebEngineView. On rip-and-test (gcc) CI it racy-crashed
during Chromium browser-process spawn under offscreen + xvfb, no GPU,
no user namespaces. rip-and-test-clang got past it by luck.

Force --no-sandbox --single-process --no-zygote --disable-gpu so the
renderer runs in-process with software rendering.
QtWebEngine browser-process init segfaults inside the qtvcp process on
Ubuntu 24.04 CI even with --no-sandbox --single-process --disable-gpu.
The smoke test never touches the WebWidget, so block the
qtpy.QtWebEngineWidgets import via a sitecustomize meta_path finder;
WebWidget already has a fallback that swaps in a plain QWidget when
that import fails. No Chromium spawn, no segfault.

The previous chromium-flags attempt was retracted: 'Sandboxing disabled
by user.' confirmed Chromium got the flags but still crashed during
init, so we are not going to win that race.
Adds a quit-path smoke test per GUI that boots the GUI, waits for the
NML task to come up, sends SIGTERM to the GUI process alone, and asserts
the GUI exits on its own within a short grace. This guards the
clean-shutdown handlers: a GUI that absorbs SIGTERM and has to be
SIGKILLed fails the test.

The new _lib/quit-launch.sh shares the headless environment (software
GL + audio silencing) with launch.sh by sourcing a new
_lib/launch-env.sh rather than copying it, so the two launchers cannot
drift apart. Results go through _lib/checkresult-quit.sh (pass on
UI_SMOKE_QUIT_OK). The GUI process is identified by matching a python
argv[0], so the linuxcnc launcher and xvfb-run wrappers that also carry
the GUI name on their command line are not mistaken for it. Per-GUI
dirs: touchy-quit, gmoccapy-quit, qtdragon-quit.

The qtdragon quit test needs the same CI workarounds the qtdragon smoke
test already carries (writable config mirror with a patched LOG_FILE,
the offscreen Qt platform, and the QtWebEngine import shim). Those move
out of qtdragon/test.sh into _lib/qtdragon-prepare.sh, sourced by both
qtdragon test.sh files, so the quit test reuses them instead of leaving
qtvcp to crash on startup.

Requires the SIGTERM handlers in LinuxCNC#4076 (gmoccapy), LinuxCNC#4077 (touchy) and
LinuxCNC#4078 (qtvcp); without them the GUIs ignore SIGTERM and these tests
fail by design.

(cherry picked from commit aff5991)
@grandixximo
Copy link
Copy Markdown
Contributor Author

It's green now, sent #4114 as a fix, will roll back the fix I did here, and do some improvements to the smoke test, then we can run again after the fix merges, I want to catch the errors with better messaging.

The driver polled NML for up to 60s after a GUI crash, then blamed
whatever stage timed out (e.g. homing); a dead task keeps serving its
last stat buffer. Watch the launcher PID and fail in ~1s pointing at
the crash. Enable PYTHONFAULTHANDLER for a Python traceback on fatal
signals.
PYTHONFAULTHANDLER stops at the Python frame; a Qt/dbus/GL segfault
needs the C stack. Arm a core dump in the launchers and, if the GUI
leaves a core, gdb its backtrace into the log. Failure-path only, so
green runs pay nothing.
quit-launch.sh catted ui-smoke.out but not ui-smoke.err, hiding the
driver's failure reason (e.g. the GUI-crashed message) on a quit-path
failure. Cat it like launch.sh already does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants