PBSPRO executor should use `qstat -B -f` to prevent failure on truncated server names

[jscgh]

Bug report

Expected behavior and actual behavior

The pbspro executor should be able to poll PBS job status and detect scheduler-side failures, so that Nextflow can apply the configured errorStrategy and maxRetries.

On the UNSW Katana HPC cluster, Nextflow 25.10.4 constructs this command:

qstat -f $( qstat -B | grep -E -v '(^Server|^---)' | awk -v ORS=' ' '{print "@"$1}' )

UNSW Katana runs OpenPBS 23.06.06. Its default qstat -B output is a fixed-width status table which truncates the server name:

Server             Max   Tot   Que   Run   Hld   Wat   Trn   Ext Status
---------------- ----- ----- ----- ----- ----- ----- ----- ----- -----------
kman.restech.un*     0 42979  2501  2318   103     0     0     1 Scheduling

Consequently, Nextflow queries @kman.restech.un*, which fails:

Unknown Host.
qstat: cannot connect to server kman (errno=15008)

The actual server name is kman.restech.unsw.edu.au. It is available from the documented long-format server query:

qstat -B -f

...
Server: kman.restech.unsw.edu.au
    server_state = Scheduling
    server_host = kman.restech.unsw.edu.au
...

Using the complete name works:

qstat -f @kman.restech.unsw.edu.au

As a result, a scheduler-side job failure is recorded correctly by PBS but is never observed by Nextflow. The task remains SUBMITTED until timing out, and its retry/failure handling is not triggered.

This is related to #5528, but differs from the blank case described there. The proposed PR #5529 filters empty records only; it does not address truncation in the fixed-width qstat -B display.

Steps to reproduce the problem

On a PBS Pro/OpenPBS cluster where the PBS server name exceeds the default qstat -B table's server-name column width:

1. Configure a process with:

process.executor = 'pbspro'
process.errorStrategy = 'retry'
process.maxRetries = 1

2. Run a failing task:

process FAIL {
    script:
    '''
    exit 42
    '''
}

workflow {
    FAIL()
}

3. Compare the outputs:

qstat -B
qstat -f @<truncated-name-from-qstat-B>
qstat -B -f
qstat -f @<full-name-from-qstat-B-f>

Program output

WARN: [PBSPRO] queue status cannot be fetched
- cmd executed: bash -c set -o pipefail; qstat -f $( qstat -B | grep -E -v '(^Server|^---)' | awk -v ORS=' ' '{print "@"$1}' ) | { grep -E '(Job Id:|job_state =)' || true; }
- exit status : 255
- output      :
  Unknown Host.
  qstat: cannot connect to server kman (errno=15008)

Environment

• Nextflow version: 25.10.4
• Java version: OpenJDK 1.8.0_492
• Operating system: Rocky Linux 8.10, kernel 4.18.0-553.123.1.el8_10.x86_64
• Bash version: GNU bash 4.4.20(1)-release
• OpenPBS version: 23.06.06

Additional context

A possible fix is to derive server names only from long-format records:

qstat -f $(
  qstat -B -f |
  awk '$1 == "Server:" {print "@" $2}'
)

References:

• Cannot fetch PBSPRO queue status #5528
• Fix PBSPRO queue status fetch #5529
• https://github.com/openpbs/openpbs/blob/v23.06.06/doc/man1/qstat.1B#L93-L103

[bentsherman]

Two questions:

1. Is this the same issue as Cannot fetch PBSPRO queue status #5528 ?
2. How do you know whether you need to use pbs or pbspro for OpenPBS? Just wondering because it's not obvious to me from the name

[jscgh]

Two questions:

1. Is this the same issue as [Cannot fetch PBSPRO queue status #5528](https://github.com/nextflow-io/nextflow/issues/5528) ?

2. How do you know whether you need to use `pbs` or `pbspro` for OpenPBS? Just wondering because it's not obvious to me from the name

1. The underlying issue is the same as Cannot fetch PBSPRO queue status #5528 but the breakage is different. In Cannot fetch PBSPRO queue status #5528, the qstat -B output produced an empty server token (@). On Katana, the server name is truncated in the table (kman.restech.un*). Both result in an invalid operand to the outer qstat -f command, so Nextflow cannot fetch task states.
   Fix PBSPRO queue status fetch #5529 would address the empty-token case only. Parsing the documented long server-status format (qstat -B -f) and selecting only Server: records addresses both cases.

2. Yes, it is confusing based on names alone. Nextflow's pbs executor does not use commands compatible with modern OpenPBS. OpenPBS and PBS Pro have shared a common core since 2020: OpenPBS and Version 20.0. Attempting to set the executor to pbs on our OpenPBS 23.06.06 results in commands being rejected as malformed.

I have tested the same on the NCI Gadi cluster, which uses PBSPro version 2024.1.2.20241017100211. On Gadi both commands emit @gadi-pbs, so the proposed fix preserves the existing behaviour.

$ qstat -B -f |   awk '$1 == "Server:" {print "@" $2}'
@gadi-pbs
$ qstat -B | grep -E -v '(^Server|^---)' | awk -v ORS=' ' '{print "@"$1}'
@gadi-pbs