fix(qlib): isolate holdout evaluation from iterative research feedback

[tolivetoearn-debug]

Summary

• Run iterative Qlib factor evaluations on the validation window, rather than the configured test/holdout window.
• Add an explicit final-holdout path that runs only after the research loop has stopped and stores its result separately.
• Add regression coverage for search-vs-holdout window selection and for preserving search results after final evaluation.

Root Cause

Qlib's SignalRecord and PortAnaRecord report metrics from the dataset test segment. The factor runner mapped that segment directly to the configured test period, and read_exp_res.py exported those metrics into exp.result. QlibFactorExperiment2Feedback then used exp.result to construct iterative LLM feedback and replacement decisions.

Change

develop() now binds Qlib's internal evaluation segment to the configured validation interval. The configured test period is only used by evaluate_holdout(), which writes holdout_result and holdout_stdout without replacing exp.result or triggering feedback/recording. main(..., run_final_holdout=True) invokes that final evaluation after RDLoop.run() returns.

The final-holdout method is private so LoopMeta does not register it as an iterative workflow step.

Validation

• python -m compileall -q rdagent/scenarios/qlib/developer/factor_runner.py rdagent/scenarios/qlib/experiment/factor_experiment.py rdagent/app/qlib_rd_loop/factor.py test/qlib/test_factor_holdout.py
• python -m ruff check --no-fix --select E9,F821,F822,F823 rdagent/scenarios/qlib/developer/factor_runner.py rdagent/scenarios/qlib/experiment/factor_experiment.py rdagent/app/qlib_rd_loop/factor.py test/qlib/test_factor_holdout.py

The focused pytest module could not be collected locally because this checkout's Python environment is missing the project's litellm runtime dependency. No Qlib, Docker, or model API execution was performed during validation.

---

📚 Documentation preview 📚: https://RDAgent--1442.org.readthedocs.build/en/1442/

[tolivetoearn-debug]

@microsoft-github-policy-service agree [company="{your company}"]

[tolivetoearn-debug]

@microsoft-github-policy-service agree