ICARUS CI test failures - "empty waveform"

[SFBayLaser]

The ICARUS CI tests are failing which is preventing us from keeping up to date with both LArSoft and SBNcode releaeses. This error message is new to us and the only thing we are sure of is that it is generated in wirecell toolkit code. The specific dump we see is the following:
2669: %MSG-s ArtException: PostEndJob 15-Jul-2026 15:10:41 CDT ModuleEndJob
2670: ---- EventProcessorFailure BEGIN
2671: EventProcessor: an exception occurred during current event processing
2672: ---- ScheduleExecutionFailure BEGIN
2673: Path: ProcessingStopped.
2674: ---- StdException BEGIN
2675: An exception was thrown while processing module WireCellToolkit/decon2droiWW run: 1 subRun: 0 event: 2
2676: ../../source/wirecell-0.36.1/util/src/Waveform.cxx(96): Throw in function percentile
2677: Dynamic exception type: boost::wrapexceptWireCell::ValueError
2678: [tag_stacktrace*] = 0# WireCell::Waveform::percentile(std::vector<float, std::allocator > const&, float) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellUtil.so
2679: 1# WireCell::SigProc::OmnibusSigProc::restore_baseline(Eigen::Array<float, -1, -1, 0, -1, -1>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2680: 2# WireCell::SigProc::OmnibusSigProc::decon_2D_tighterROI(int) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2681: 3# WireCell::SigProc::OmnibusSigProc::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2682: 4# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
2683: 5# WireCell::Pgraph::Function::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
2684: 6# WireCell::Pgraph::Graph::call_node(WireCell::Pgraph::Node*) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
2685: 7# WireCell::Pgraph::Graph::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
2686: 8# WireCell::Pgraph::Pgrapher::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
2687: 9# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_36_1/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
2688: 10# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft-ib.opensciencegrid.org/LArSoft/LArSoft_custom_slf7_e26_prof_20260715_142111/localProducts_LArSoft_LArSoft_lar_ci_e26_prof/larwirecell/v10_03_05/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
2689: 11# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic&, std::atomic&, std::atomic&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
2690: 12# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
2691: 13# 0x000014F8BEA407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
2692: 14# tbb::detail::d1::function_taskhep::concurrency::SerialTaskQueue::QueuedTask::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
2693: 15# 0x000014F8C32424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
2694: 16# 0x000014F8C3241437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
2695: 17# 0x000014F8C32408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
2696: 18# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
2697: 19# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
2698: 20# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
2699: 21# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
2700: 22# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
2701: 23# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
2702: 24# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
2703: 25# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_deleteart::OptionsHandler >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_deleteart::OptionsHandler > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
2704: 26# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
2705: 27# main in lar
2706: 28# __libc_start_main in /lib64/libc.so.6
2707: 29# 0x00000000004012D0 in lar
2708:
2709: [WireCell::tag_errmsg*] = empty waveform
2710: ---- StdException END
2711: Exception going through path path
2712: ---- ScheduleExecutionFailure END
2713: ---- EventProcessorFailure END
2714: %MSG
2715: Art has completed and will exit with status 1.
2716:

Setup:

From an icarus gpvm do:
source /cvmfs/icarus.opensciencegrid.org/products/icarus/setup_icarus.sh
setup icaruscode v10_20_09p01 -q e26:prof

The set of CI tests run and with failures can be found at this link.

fcl to run:

icarus_ci_single_reco0_seq_test_icaruscode.fcl

Example of lar command from CI tests:
lar --rethrow-all -n 5 --TFileName hist-single_reco0_test_icaruscode.root -o single_reco0_test_icaruscode_Current.root --config icarus_ci_single_reco0_seq_test_icaruscode.fcl ../single_detsim_seq_test_icaruscode/single_detsim_test_icaruscode_Current.root

input file:

The full output of the CI tests has been stored at "/exp/icarus/data/users/vito/ci_tests/ci_14907/ICARUS"

A specific file to use that will fail on the second event is:
/exp/icarus/data/users/vito/ci_tests/ci_14907/ICARUS/single_detsim_seq_test_icaruscode/single_detsim_test_icaruscode_Current.root

Because the full signal processing chain takes a long time I would suggest skipping directly to the second event:

lar -c icarus_ci_single_reco0_seq_test_icaruscode.fcl -s /exp/icarus/data/users/vito/ci_tests/ci_14907/ICARUS/single_detsim_seq_test_icaruscode/single_detsim_test_icaruscode_Current.root -e 1:0:1

[SFBayLaser]

I forgot to mention... we are not seeing this problem in production which is done along the SBN2025A branch which is using wirecell toolkit v0_30_6. At this state, icaruscode v10_20_09p01 is wirecell toolkit v0_36_1.

[SFBayLaser]

Interestingly... in trying to help debug an issue with dunesw I set up the latest version of dunesw and it also crashes in wirecell albeit in a somewhat different manner:

%MSG-s ArtException: PostEndJob 17-Jul-2026 10:29:59 CDT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- StdException BEGIN
An exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40222 subRun: 1 event: 43679
../../source/wirecell-0.37.0/util/inc/WireCellUtil/Exceptions.h(58): Throw in function raiseWireCell::RuntimeError
Dynamic exception type: boost::wrapexceptWireCell::RuntimeError
[tag_stacktrace*] = 0# void WireCell::raiseWireCell::RuntimeError(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
1# WireCell::SigProc::L1SPFilterPD::operator()(std::shared_ptr<WireCell::IFrame const> const&, std::shared_ptr<WireCell::IFrame const>&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellSigProc.so
2# WireCell::IFunctionNode<WireCell::IFrame, WireCell::IFrame>::operator()(boost::any const&, boost::any&) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellIface.so
3# WireCell::Pgraph::Function::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
4# WireCell::Pgraph::Graph::call_node(WireCell::Pgraph::Node*) in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
5# WireCell::Pgraph::Graph::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
6# WireCell::Pgraph::Pgrapher::execute() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellPgraph.so
7# WireCell::Main::operator()() in /cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_37_0/Linux64bit+3.10-2.17-e26-prof/lib/libWireCellApps.so
8# wcls::WCLS::process(art::Event&) in /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_03_08/slf7.x86_64.e26.prof/lib/liblarwirecell_Tools_WCLS_tool.so
9# art::detail::Producer::doEvent(art::EventPrincipal&, art::ModuleContext const&, std::atomic&, std::atomic&, std::atomic&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Core.so
10# art::Worker::runWorker(art::EventPrincipal&, art::ModuleContext const&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
11# 0x00007F3DA18407E3 in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Principal.so
12# tbb::detail::d1::function_taskhep::concurrency::SerialTaskQueue::QueuedTask::execute(tbb::detail::d1::execution_data&) in /cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof/lib/libhep_concurrency.so
13# 0x00007F3DA64424F1 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
14# 0x00007F3DA6441437 in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
15# 0x00007F3DA64408BA in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
16# tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) in /cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26/lib/libtbb_debug.so.12
17# void art::EventProcessor::process<(art::Level)4>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
18# void art::EventProcessor::process<(art::Level)3>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
19# void art::EventProcessor::process<(art::Level)2>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
20# void art::EventProcessor::process<(art::Level)1>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
21# void art::EventProcessor::process<(art::Level)0>() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
22# art::EventProcessor::runToCompletion() in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_EventProcessor.so
23# art::run_art(int, char**, boost::program_options::options_description&, std::vector<std::unique_ptr<art::OptionsHandler, std::default_deleteart::OptionsHandler >, std::allocator<std::unique_ptr<art::OptionsHandler, std::default_deleteart::OptionsHandler > > >&&) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
24# artapp(int, char**, bool) in /cvmfs/larsoft.opensciencegrid.org/products/art/v3_14_04/slf7.x86_64.e26.prof/lib/libart_Framework_Art.so
25# main in lar
26# __libc_start_main in /lib64/libc.so.6
27# 0x00000000004012D0 in lar

  [WireCell::tag_errmsg*] = L1SPFilterPD: unexpected input
---- StdException END
Exception going through path produce

---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG