Consider recommending stricter standards about merging in submissions to hubs

[lshandross]

This came up in the context of evaluating multiple targets in the dashboard, but is useful to consider in general both for the evaluations dashboard and good practice.

When requesting relative metrics, the evaluations dashboard's data will fail to build if there are model submissions but no baseline for a particular reference date (see https://github.com/hubverse-org/hubPredEvalsData/issues/20) that is included in any evaluation set. It is possible to get around this issue by explicitly specifying which dates to include in the evaluation set(s), but this solution is slightly cumbersome and does not work when a user wants to exclude a particular date for one target but not another.

The problem becomes irrelevant if hubs are stricter about the submissions they merge in, so that there are never other forecasts for a reference date without baseline forecasts. This solution seems reasonable since we don't expect hubs need to collect forecasts on dates that they do not plan on generating a baseline for.

The one potential source of friction is that many hubs collect forecasts for multiple targets in a single model output file; this means that on weeks the baseline won't be generated for target X, hub administrators will need to communicate to teams that they are only collecting forecasts for targets other than target X and potentially manually check this rule is being followed.

What do others think about this proposal?

[nickreich]

To me, this feels like maybe an unnecessary restriction. It seems like the main issue that brought this up is computation of relative scores. But methods exist (and could be used) to implement relative scores even when the baseline isn't submitted for a few prediction tasks. See existing issues about this:
hubverse-org/hubPredEvalsData#14
hubverse-org/predevals#18

I'd be in favor of implementing fixes to those issues which then would make the source of the friction that generated this discussion disappear.