[Show and Tell] Programmer from Argentina built a live agentic trading system — permutation tests killed my signals, looking for real feedback (online now)

[raveslayer666]

Hey all — programmer here, not a quant. Based in Argentina, live account,
small size. Been building for a few weeks and wanted to share where I landed
and get honest feedback from people who've done this longer.

---

What I built

Agentic system with signal-only mode — nothing auto-executes. All signals go
to a review queue, human approves before anything hits the broker. Four
strategies running in parallel: sector momentum, pairs trading, volatility
harvesting, event-driven.

Added multiple validation layers before a signal reaches the queue:

• Historical sanity check (recent hit rate + return)
• Statistical significance test (shuffle-based)
• Economic rationale gate: "why would someone consistently take the
  other side of this trade?" — no clear answer = discard

Also running 60+ trading skills integrated with the broker API,
an orchestrator that routes analysis requests to the right agents
automatically, and live market data flowing into the reasoning layer.

---

What the stats found

Ran significance tests on all active signals this week.
None passed the threshold.

My interpretation: the signals are fundamentals-based value plays,
not momentum signals. The shuffle test validates momentum edge —
so maybe I'm testing the wrong thing. Or maybe I'm rationalizing.
Genuinely unsure.

---

The pairs problem

System kept generating a pair signal between two bond ETFs.
Ran actual cointegration: p-values near 1. Not cointegrated.
Killed it.

Today's autonomous analysis (multi-agent, real broker data,
ran in under 3 minutes) identified a potential real pair:
two semiconductors exposed to the same AI capex cycle,
but with dramatically different performance over the past year.
One up ~3x on narrative premium, the other down ~20% from highs.
DCF says the first is significantly overvalued, the second is fairly priced.

Stop defined, target defined, size constraint is the main issue
(small account).

---

What I know is wrong

• Significance test window might be too short — haven't tested longer horizons
• Fundamental model built on public data with manual assumptions,
  could be way off
• Account too small for real diversification
• No persistent memory between sessions yet (workaround in place but static)

---

What I'm looking for

1. Is a shuffle/permutation test the right framework for validating
   fundamentals-driven signals, or is there a better approach?

2. For the pair: does a large volatility difference between the two legs
   break the thesis, even if the ratio divergence is historically extreme?

3. Anyone else using an LLM as the reasoning layer over a broker API —
   not for execution, just for analysis and signal generation?
   What actually works?

Happy to go deep on architecture, validation approach, or anything else.
Online now if anyone wants to discuss in real time.

rs

[warren618]

Welcome, @raveslayer666 — genuinely enjoyed this. What you built (signal-only, everything through a human review queue, execution gated) is exactly the philosophy we care about here, and the "why would someone consistently take the other side?" gate is the kind of check that's hard in a pure quant pipeline but natural once an LLM sits in the reasoning layer.

I won't turn your Show-and-Tell into a methodology thread — your three questions (permutation tests on value vs momentum signals, vol-mismatched pair legs, LLM-as-reasoning-layer) are good ones, and I'd rather see how others here who've run these setups longer would answer. Anyone want to weigh in?

One light pointer instead of a full answer: signal-only LLM-over-broker is basically this project's default mode, so the built-in validation tools (Monte Carlo / bootstrap / walk-forward) might be a useful second opinion to cross-check what your shuffle test is telling you. Happy to go deeper if you open a focused discussion on any one of the three.

[raveslayer666]

thanks for this — and for not burying the lede on the walk-forward comparison. the shuffle test result that's been bothering me most: YPF came back with IC = -0.056 for a BUY signal, which my system flagged as "wrong sign." but the DSR was 1.0 and subsample stability was 3/3 across 5-year windows. i ended up treating it as a value/mean-reversion signal rather than a momentum signal and kept the position — but i'm not confident that distinction is principled versus convenient. the question i'd want to go deeper on is the third one (LLM-as-reasoning-layer), specifically: when the LLM is sitting in the reasoning layer and it surfaces something real (in this case public insider selling data that changed a decision), how do you separate "the LLM found signal" from "the LLM rationalized a prior belief with publicly available noise"? your monte carlo / walk-forward tools — do they run against the LLM's outputs or only against the underlying price signals? El El sáb, 20 jun 2026 a la(s) 07:14, Haozhe Wu ***@***.***> escribió:

…

Welcome, @raveslayer666 <https://github.com/raveslayer666> — genuinely enjoyed this. What you built (signal-only, everything through a human review queue, execution gated) is exactly the philosophy we care about here, and the *"why would someone consistently take the other side?"* gate is the kind of check that's hard in a pure quant pipeline but natural once an LLM sits in the reasoning layer. I won't turn your Show-and-Tell into a methodology thread — your three questions (permutation tests on value vs momentum signals, vol-mismatched pair legs, LLM-as-reasoning-layer) are good ones, and I'd rather see how others here who've run these setups longer would answer. Anyone want to weigh in? One light pointer instead of a full answer: signal-only LLM-over-broker is basically this project's default mode, so the built-in validation tools (Monte Carlo / bootstrap / walk-forward) might be a useful second opinion to cross-check what your shuffle test is telling you. Happy to go deeper if you open a focused discussion on any one of the three. — Reply to this email directly, view it on GitHub <#275?email_source=notifications&email_token=CFMUL2CB5FFMNWXF5LW7UY35AZPX5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZTG4ZTKNZTUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#discussioncomment-17373573>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CFMUL2DPMWQTSYHR4KXQNTT5AZPX5AVCNFSNUABJKJSXA33TNF2G64TZHMYTCOJYGMYDMOBRGI5UI2LTMN2XG43JN5XDWMJQGI4DSMZQHGQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CFMUL2DOZDPUAYCUZSNSAND5AZPX5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZTG4ZTKNZTUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/CFMUL2DT6ZXBJQIGBBUIVZL5AZPX5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZTG4ZTKNZTUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[warren618]

Hey @raveslayer666 — sorry for the gap, this fell through the cracks.

On the concrete question about our tools: they run against the realized trade/equity series, not against any LLM output directly. Looking at the implementation (agent/backtest/validation.py): monte_carlo_test takes the completed TradeRecord list and shuffles PnL order; bootstrap_sharpe_ci and walk_forward_analysis take the equity_curve produced by the backtest run. All three are downstream of "whatever the SignalEngine actually decided to do" — so if your LLM is generating entries/exits, its decisions are exactly what's being tested, but only as a black-box decision sequence. None of the three look at the LLM's stated reasoning, its context, or the input data it saw. They can tell you "this decision sequence has more structure than a random shuffle of the same trades" or "performance is stable across rolling 5yr windows" — they cannot tell you why, and specifically cannot distinguish "the reasoning was causally load-bearing" from "the reasoning was a plausible-sounding rationalization sitting on top of a decision sequence with real edge from something else."

That's really the crux of your question, and I think it's a structural limit of the shuffle/bootstrap/walk-forward family, not something that gets patched by running the same tests harder. Two things I'd actually try, both compatible with what's here:

1. Reasoning ablation, not just decision-sequence stats. Hold the model, universe, and price data fixed; strip the insider-selling (or whichever qualitative) input from context on the exact same historical windows; rerun. Then paired-bootstrap the difference between the two equity curves (with-reasoning vs. without), not each curve's own CI separately. If the delta isn't distinguishable from noise, the reasoning wasn't load-bearing for that DSR = 1.0, whatever the LLM said it was doing.
2. Held-out event generalization, not just held-out time. Walk-forward's rolling windows tell you a fixed rule is stable over time, but an LLM's reasoning isn't a fixed rule — it can quietly overfit to recurring surface patterns in how insider-selling stories get reported. What you actually want is performance on event types the model hasn't been prompted with before, which walk-forward alone won't surface.

On the YPF case — negative short-horizon IC on a BUY-labeled signal next to a stable, positive equity curve isn't automatically "wrong sign," it's a horizon-mismatch flag worth chasing directly: if your IC window is short and the thesis is mean-reversion/value, check IC at a longer lag before deciding the value-vs-momentum relabeling was ad hoc. If IC stays negative at every horizon you have data for and the curve is positive only because of a handful of trades, that's a different, less comfortable finding.

Curious whether you've already tried the ablation — that's the one that would actually answer your third question directly.

[raveslayer666]

hey Haozhe, no worries about the gap, and honestly this is the reply i was hoping for short version: you're right that it's a structural limit, and it's most of why i stopped leaning on the backtest at all. but your causal point is only half of why. the other half is the one that actually broke my trust in the DSR: the model has the price history in its weights. anything with a 2025 cutoff already knows how YPF or NVDA moved through the exact window i was testing on, so a chunk of what the shuffle test read as "structure" is just recall, and no data-pipeline audit catches it because the leak isn't in the pipeline — it's inside the weights. there's a real line of work on this now (look-ahead-bench, the "summoning the oracle" paper) and once it clicked, the DSR = 1.0 i'd been quoting stopped feeling like a number i earned. so i tore that part out and rebuilt around forward measurement only — every signal source scored purely on realized forward return from the moment the info was actually public, no pre-cutoff data, no backtest left to overfit. and doing that i walked straight into a concrete version of exactly your worry. i'd been crediting the "superinvestors bought this" signal at the quarter-end date. a 13F isn't public until ~45 days after quarter-end. i moved the entry to the actual filing date and the signal dropped from ~+30% / 90% hit to ~+6% / 46%. same decisions, honest timestamp, most of the "edge" was a look-ahead in my own harness. your "real edge from something else" made painfully literal — the something else was me. on the model, this is the "it's different now" part. since june i moved the decision/filter layer off the cheaper model it was on onto a frontier thinking model, and for your exact question that's not cosmetic. the CoT-faithfulness people have measured post-hoc rationalization rates across models and they're wildly different — double digits for the small cheap ones, ~0.04% for a frontier reasoning one. that doesn't prove the reasoning is load-bearing, you still have to run the ablation, but it moves the prior a lot, and it means the june version i showed you was materially more likely to be rationalizing than what's running now. so your ablation is worth running now because there's a plausible causal story to test, not just a null-result exercise. which — straight with you — i haven't run end-to-end yet. but it drops cleanly onto the forward-scoring scaffolding i already have: hold model/universe/prices fixed, strip the insider input on the same windows, paired-bootstrap the delta between the two curves instead of each CI on its own. that's causal mediation on the input, which is the right frame. i'll run it and send you the delta, positive or negative. your second point is the deeper one and i don't have a clean answer for it. walk-forward tests a fixed rule over time; the LLM isn't a fixed rule, it can quietly overfit to how insider-selling stories get written, and my qualitative universe is small and recurring — insider clusters, 13Fs, the same handful of korean-memory narratives — so i'm exposed precisely there. held-out event types is the right test and i'm not running it yet. the only thing between that failure mode and actual capital right now is that the qualitative layer isn't allowed to touch sizing — it's context into a human queue, and size is deterministic off measured skill, so a confident rationalization can't silently lever up a position. that's a blast-radius limit, not a substitute for the test, and i won't pretend it's more. on YPF — good catch, and you're right it's a horizon-mismatch flag, not a wrong sign. it was always in the book as a value / mean-reversion name (vaca muerta production growth and breakeven, not momentum), so short-lag IC of -0.x next to a positive curve is roughly what i'd expect. but "it was always value" is exactly what i'd say whether or not it's true, so i take the point: the honest move is to check IC at every lag i have and see if the curve survives dropping the best two or three trades. i haven't done the drop-the-best-trades stress on YPF by itself. i'll do that too — if it stays negative at every horizon and the curve only lives on a handful of names, that's the less comfortable finding, and i'd rather know it than not. for what it's worth i've been reading around while rebuilding — tradingagents and the debate-style multi-agent stuff, the live benchmarks like livetradebench that score forward instead of backtesting. almost all of the published ones auto-execute; i've stayed deliberately on the signal-only / human-gated side, which is the part you liked to begin with. the causal question you're pushing is, to me, the actual frontier — the shuffle/bootstrap/walk-forward family genuinely can't answer it, and where the field seems to be heading (anonymize the ticker so the model can't recall, held-out events, causal mediation on the inputs) is exactly where i'm trying to go. so thanks for pushing on it instead of being nice about the number. El dom, 12 jul 2026 a las 6:37, Haozhe Wu ***@***.***>) escribió:

…

Hey @raveslayer666 <https://github.com/raveslayer666> — sorry for the gap, this fell through the cracks. On the concrete question about our tools: *they run against the realized trade/equity series, not against any LLM output directly.* Looking at the implementation (agent/backtest/validation.py): monte_carlo_test takes the completed TradeRecord list and shuffles PnL order; bootstrap_sharpe_ci and walk_forward_analysis take the equity_curve produced by the backtest run. All three are downstream of "whatever the SignalEngine actually decided to do" — so if your LLM is generating entries/exits, its decisions are exactly what's being tested, but only as a black-box decision sequence. None of the three look at the LLM's stated reasoning, its context, or the input data it saw. They can tell you "this decision sequence has more structure than a random shuffle of the same trades" or "performance is stable across rolling 5yr windows" — they cannot tell you *why*, and specifically cannot distinguish "the reasoning was causally load-bearing" from "the reasoning was a plausible-sounding rationalization sitting on top of a decision sequence with real edge from something else." That's really the crux of your question, and I think it's a structural limit of the shuffle/bootstrap/walk-forward family, not something that gets patched by running the same tests harder. Two things I'd actually try, both compatible with what's here: 1. *Reasoning ablation, not just decision-sequence stats.* Hold the model, universe, and price data fixed; strip the insider-selling (or whichever qualitative) input from context on the exact same historical windows; rerun. Then paired-bootstrap the *difference* between the two equity curves (with-reasoning vs. without), not each curve's own CI separately. If the delta isn't distinguishable from noise, the reasoning wasn't load-bearing for that DSR = 1.0, whatever the LLM said it was doing. 2. *Held-out event generalization, not just held-out time.* Walk-forward's rolling windows tell you a fixed rule is stable over time, but an LLM's reasoning isn't a fixed rule — it can quietly overfit to recurring surface patterns in how insider-selling stories get reported. What you actually want is performance on event *types* the model hasn't been prompted with before, which walk-forward alone won't surface. On the YPF case — negative short-horizon IC on a BUY-labeled signal next to a stable, positive equity curve isn't automatically "wrong sign," it's a horizon-mismatch flag worth chasing directly: if your IC window is short and the thesis is mean-reversion/value, check IC at a longer lag before deciding the value-vs-momentum relabeling was ad hoc. If IC stays negative at every horizon you have data for and the curve is positive only because of a handful of trades, that's a different, less comfortable finding. Curious whether you've already tried the ablation — that's the one that would actually answer your third question directly. — Reply to this email directly, view it on GitHub <#275?email_source=notifications&email_token=CFMUL2DS3SHBDOX5ESWZ7JL5ENL4XA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZWGEYTGMRTUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#discussioncomment-17611323>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CFMUL2DO6MRFXTONZHWKZZD5ENL4XAVCNFSNUABJKJSXA33TNF2G64TZHMYTCOJYGMYDMOBRGI5UI2LTMN2XG43JN5XDWMJQGI4DSMZQHGQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CFMUL2HST722YBIMQ7JC3I35ENL4XA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZWGEYTGMRTUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/CFMUL2AQBXTZXGWJBYAYEAD5ENL4XA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZWGEYTGMRTUZZGKYLTN5XKO3LFNZ2GS33OUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[warren618]

Coming back to this properly, and apologies for the gap.

Your second point is the one I under-weighted at the time. A pipeline audit can't catch a leak that lives in the weights. If the model has seen the price history for the exact window you're testing on, a shuffle test can read recall as structure, and nothing in the data plumbing will ever surface it. That's a different failure mode from the causal-structure limit I raised, and for anything that puts an LLM in the signal path it's the harder one to defend against.

The consequence, as I read it: a permutation or DSR number computed over a period inside the model's training window isn't evidence of much, and the honest move is to say so rather than report the statistic. Forward-only evaluation on data postdating the cutoff is the version that survives scrutiny, with the obvious cost that sample accumulates slowly.

I haven't read the specific papers you mention closely enough to endorse them, so I'll leave the citations to you — but the mechanism you're describing seems real to me and worth writing down properly. If you were up for it, a short writeup of how you rebuilt the evaluation forward-only would be valuable here. Taking apart a result you'd already published a number for is not a comfortable thing to do, and it's worth more to this community than another strategy post would be.