msolve CI failures

[fredrik-johansson]

The msolve job on CI fails or times out very frequently, in my recent experience probably 50% of runs. This is a waste of time and resources.

If some specific msolve test is the culprit (I have not looked too carefully), maybe we could manually disable just that test?

[albinahlback]

I'll try to grab a hold of one of the maintainers within the coming hours!

[albinahlback]

@jerebertho told me that as of this morning he knows somewhat where the bug is located, and that they are trying to solve it. It seems like something in msolve is wrong, and that the random seed is sometimes generating a "bad prime". In the future, they will allow setting the seed during the check (such as make check SEED=0xdeadbeef).

In the mean time, we could just disable the CI.

[vneiger]

In the mean time, we could just disable the CI.

Yes, I agree, it's been failing annoyingly often recently. Let's say that if we (msolve developers) haven't solved it by early next week, then I will disable this part of FLINT's CI (if you prefer not to wait until early next week, no problem, please proceed).

[jerebertho]

Hi,
We pushed several fixes today to msolve. Furthermore, it seems that a culprit, in our workflow file (and that is also in yours) was the line
echo "MAKE=make -j$(expr $(nproc) + 1) --output-sync=target" >> $GITHUB_ENV
I suggest that you change this to simply
echo "MAKE=make -j2 --output-sync=target" >> $GITHUB_ENV

[vneiger]

To explain things a bit: we are not sure yet, why things go better with -j2. We have no good reason to prefer -j2 upon the previous version, apart from the fact that we observe much more unexplained failures with -j$(expr $(nproc) + 1).

The possibly failing test should now be restricted to a single test in those of msolve's make check. If it still fails too much, we might disable that test only, as initially suggested here.

[vneiger]

FYI, recent fixes in msolve have improved the situation a little further (and -j2 is not necessary anymore). Hopefully failures will be quite more rare now, let's see if this is confirmed over the next few days.