Two-stream solver crash in historic FATES run with land use

[JessicaNeedham]

Offline historic FATES simulation crashes when it is restarted in 1850 with transient land use with an error message about the two-stream solver. I'm not sure if this is a problem with my set up or a real bug. I am on my own branches (see details below) and am pointing to an old surface data file (CESM couldn't find the correct file or download it - @mvdebolskiy has now added the correct files).

General bug information

CTSM version you are using: [output of git describe]
I am on my own CTSM and FATES branches that are branched from current noresm and FATES but with recent pending PRs included. On the CTSM side: #211 and #209. On the FATES side: 57 (NorESMhub/fates#57), 55 (NorESMhub/fates#55), 52 (NorESMhub/fates#52), 56 (NorESMhub/fates#56) and 40 (NorESMhub/fates#40).

CTSM branch here: https://github.com/JessicaNeedham/ctsm/tree/ncsrevise
FATS branch here: https://github.com/JessicaNeedham/fates/tree/ncsrevise

Does this bug cause significantly incorrect results in the model's science? [Yes / No] Yes

Configurations affected: [Fill this in if known.]
FATES runs with land use

Details of bug

Poor forward solution on two-stream solver
380: isol (1=beam or 2=diff): 1
380: i (equation): 6
380: band index (1=vis,2=nir): 1
380: error (tau(i) - omega(i,:)*lambda(:)) -1.631609623423152E-002
380: Dumping Two-stream elements for band 1
380:
380: rbeam atm: 1.00000000000000
380: rdiff_atm: 1.00000000000000
380: alb grnd diff: 8.000000000000000E-002
380: alb grnd beam: 8.000000000000000E-002
380: cosz: 0.345449349549086
380: snow fraction: 0.000000000000000E+000
380: -- 1 1 --
380: pft: 1
380: area: 1.00000000000000
380: lai,sai: 3.80035171846969 0.380035172006965
380: Kb: 1.16160406009915
380: Kb leaf: 1.17776446611587
380: Kd: 1.03519339863251
380: Rb0: 1.00000000000000
380: om: 0.173727272728699
380: betad: 0.580805965469380
380: betab: 0.474166953283628
380: a: 0.954104398321600
380: RDiff Down @ bottom: -4.167520312229959E-003
380: RDiff Up @ bottom: 1.458110854053115E-004
380: Rbeam @ bottom: 7.781902878575531E-003

Important details of your setup / configuration so we can reproduce the bug

Crash is on Betzy. My case is here: /cluster/work/users/jessica/ncsrevise_runs/noresm-fates-f19-LU-PPE-1850-1900-control_oldsurdat.2026-04-15

I ran an AD and postAD spin up and then used the finitdat to start this run. The crash occurred immediately on the first time step. The postAD spin up continues fine, suggesting it is something to do with the transient simulation.

[mvdebolskiy]

Can you run with DEBUG=TRUE, please?

[JessicaNeedham]

@mvdebolskiy - case with DEBUG=TRUE is here: /cluster/work/users/jessica/ncsrevise_runs/noresm-fates-f19-LU-PPE-1850-1900-control_oldsurdat_debug.2026-04-16

Looks like an issue with the LUPFT history variables.

I see you made some changes to those before they were merged in 56 (NorESMhub/fates#56). I merged the latest changes from noresm branch into my FATES branch and that case is here (DEBUG=FALSE) /cluster/work/users/jessica/ncsrevise_runs/noresm-fates-f19-LU-PPE-1850-1900-control_oldsurdat_newhist.2026-04-16

Now there is a mass balance error detected. But now the spin up and historic 1850 are on different commits so I'm not sure what to make of this.

[JessicaNeedham]

I don't think we should spend time on this since we have updates coming in soon and we can run historic simulations with those.

[kjetilaas]

I have been running a couple of historical cases with this setup also, and have also run into some problems. Like @JessicaNeedham, I am not sure we should spend more time on this, since this is with combination of PRs that has not yet been approved. I am also not convinced that the problems I am having are not betzy problems. But I try to document here for future reference.

These are all historical simulations starting after 400yr ADspinup and 300yr postADspinup with the same setup. All runs start without problems.

First run is described with this run script. This runs for 55 years, but crashes just before finishing the first submit and writing restars:

  96: --------------------------------------------------------------------------
  96: MPI_ABORT was invoked on rank 96 in communicator MPI_COMM_WORLD
  96: with errorcode -1.
  96: 
  96: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
  96: You may or may not see output from other processes, depending on
  96: exactly when Open MPI kills them.
  96: --------------------------------------------------------------------------

Case dir here: /cluster/work/users/kjetisaa/ihist.ne16pg3_tn14.ncsrevise.CPLHIST.2026-04-17
Run directory here: /cluster/work/users/kjetisaa/noresm/ihist.ne16pg3_tn14.ncsrevise.CPLHIST.2026-04-17/run

In a second run I tested a short test with restarts every year. This completed without problems (4 yrs):
/cluster/work/users/kjetisaa/ihist.ne16pg3_tn14.ncsrevise.CPLHIST_1yrRestart.2026-04-19

In a third run I tested running the full historical without restarts. This crashed in yr 1939, much later than the initial crash, with the following cesm log message:

638:  ENDRUN:
638:  ERROR in TwoStreamMLPEMod.F90 at line 392   

/cluster/work/users/kjetisaa/noresm/ihist.ne16pg3_tn14.ncsrevise.CPLHIST_170yrs.2026-04-19/run

[kjetilaas]

My historical test with out-of-the-box beta14 also crashed at the end of the first submit. It seems to me that it crashed while writing datm restart file. This is the end of the atm log file:

(shr_orb_params) -----------------------------------------
(shr_strdata_readstrm)  closing  : /cluster/work/users/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23/cpl/hist//n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23.cpl.hx.atm.1h.inst.0424-12-31-03600.nc
(shr_strdata_readstrm)  opening   : /cluster/work/users/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23/cpl/hist//n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23.cpl.hx.atm.1h.inst.0425-01-01-03600.nc
(shr_strdata_readstrm)     reading file ub: /cluster/work/users/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23/cpl/hist//n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23.cpl.hx.atm.1h.inst.0425-01-01-03600.nc
(shr_strdata_advance)      Stream: 1 reading new data with  LB ymd,tod = 18691231  85500   UB ymd,tod = 18700101  2700
(shr_strdata_readstrm)  closing  : /cluster/work/users/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23/cpl/hist//n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23.cpl.hx.atm.1h.avrg.0424-12-31-03600.nc
(shr_strdata_readstrm)  opening   : /cluster/work/users/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23/cpl/hist//n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23.cpl.hx.atm.1h.avrg.0425-01-01-03600.nc
(shr_strdata_readstrm)     reading file ub: /cluster/work/users/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23/cpl/hist//n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23.cpl.hx.atm.1h.avrg.0425-01-01-03600.nc
(shr_strdata_advance)      Stream: 4 reading new data with  LB ymd,tod = 18691231  85500   UB ymd,tod = 18700101  2700
(dshr_restart_write)  writing ihist.ne16pg3_tn14.noresm_3_0_beta14.CPLHIST.2026-04-24.datm.r.1870-01-01-00000.nc

Run folder is here:
/cluster/work/users/kjetisaa/noresm/ihist.ne16pg3_tn14.noresm_3_0_beta14.CPLHIST.2026-04-24/run

The experience from earlier is that this is not triggered when running shorter simulations (e.g. 1 yr). I will start it again with debug=true. This is a 20yr simulation, so that will take some time, but should be ready by Monday...

[mvdebolskiy]

@kjetilaas I don't think debug will help. It's an integer overflow error in typecasting. Meaning somewhere a big float64 is converted to int16 or in32. Not sure how it is happening though. Will look if there are other issues somewhere with this in DATM.

[kjetilaas]

Right! That's useful to know. I am testing now with 10yr intervals and, and with crujra forcing instead. Just to see if I still run into this problem with also then.

[mvdebolskiy]

@kjetilaas
You already had this problem before.In December. Reduce the nummber of stream files that are being read.

[kjetilaas]

Thanks a lot for looking into this, @mvdebolskiy! That makes sense. And it is great to know that this is not a problem with the historical forcing files or setup. Both my CRUJRA historical and the test with only using 10yrs of CPLHIST forcing seems to run without problems (both have run ~65 years), which agrees well with what you write.

Still, it seems like the number of stream files we can have is decreasing. I think I reduced this from 100 to 50 years earlier. Now I must reduce even further. And it is less for historical than AD-/postAD spinup (they both were using 50yr CPLHIST). Not sure if you have any idea of why that is?

[mvdebolskiy]

@kjetilaas you also might have files missing for cphist. CRUJRA has monthly files so there are ot a lot of them. You had a crash in december with the same problem with i1850clphist running for 55 years. We should change the setup for aux hist files or postprocess them by concatenating in the compress job.

[mvertens]

@kjetilaas @JessicaNeedham - I am really confused about the setup of this case.

• It says that it's a historical simulation - but you are cycling over pre-industrial atm data (DATM_CPLHIST_CASE is n1850.ne16pg3_tn14.noresm3_0_beta10.Run6.2026-01-23).
• You are starting the run at year 1850 - but you are setting DATM_YR_ALIGN at year 1. I think the DATM_YR_ALIGN should be 1850 (your first year of forcing data should align with the model year 1850).
  See https://escomp.github.io/CDEPS/versions/master/html/streams.html#data-model-stream-input:
  The simulation year corresponding to yearFirst.
  A common usage is to set this to the first year of the model run (for CIME-CCS this would correspond to the xml variable RUN_STARTDATE). With this setting, the forcing in the first year of the run will be the forcing of year yearFirst.
  Another usage is to align the calendar of transient forcing with the model calendar. For example, setting yearAlign = yearFirst will lead to the forcing calendar being the same as the model calendar. The forcing for a given model year would be the forcing of the same year. This would be appropriate in transient runs where the model calendar is setup to span the same year range as the forcing data.
• I'm still perplexed that the number of files is triggering the error you are seeing. A much simpler configuration to try to understand why the number of files would trigger the problem you would be:

--compset HIST_DATM%CPLHIST_SLND_SICE_SOCN_SROF_SGLC_SWAV_SESP --res ne16pg3_tn14

[kjetilaas]

@mvertens, I understand that you are confused. And thanks for showing interest. :)

This is a bit of a Frankenstein setup, which is why we originally didn't think this would be worth debugging. But since both @JessicaNeedham and I had problems that only appeared when turning on the transient LU, we used this issue to keep track of it.

From my side, it was an attempt to test the new code and parameters that are coming in to beta14-16, and check

• 1. that the LU stream files work, and the transition from constant to transient landuse doesn't crash. (Before we run full historical coupled again)
• 2. if we still have a too large land source of CO2 in the 20th century with the new parameters. This will be different when climate is also evolving, but since we don't have historical CPLHIST files, I only test with evolving CO2 and LU for now which is still a useful test for us.
     This is why I didn't care about the DATM alignment for now.

[mvertens]

@kjetilaas - thanks for the clarification. In the meantime I'm digging a bit into the crash using a simplified compset:
--compset HIST_DATM%CPLHIST_SLND_SICE_SOCN_SROF_SGLC_SWAV_SESP --res ne16pg3_tn14
and I can reproduce the crash very quickly.