"Training world models over word models." — Saining Xie (谢赛宁), founding manifesto of AMI Labs
A curated, timestamped map of every research paper, model, and system mentioned by Saining Xie (谢赛宁) — co-founder & Chief Scientist of AMI Labs (with Yann LeCun) — during his first-ever interview: a 6-hour-45-minute marathon with podcaster Zhang Xiaojun (张小珺), recorded in Brooklyn after the 2026 Spring Festival.
Its centerpiece is a moment around 2:21:40 where Saining tries to name the "~20–25 papers that deeply shaped deep learning and AI" — the works Saining considers genuine 代表作 (masterpieces) — while insisting "那我一篇都没有" ("I don't have a single one of those"). This repo reconstructs that canon, separates it from his own work, and decodes the names the auto-transcription garbled.
| Format | Link |
|---|---|
| 📺 Video (Bilibili) | https://www.bilibili.com/video/BV1tew5zVEDf/ |
| 🎧 Audio (小宇宙 / Xiaoyuzhou) | https://www.xiaoyuzhoufm.com/episode/69b77577f8b8079bfa8eb837 |
| 🎧 Apple Podcasts (ep. 133) | https://podcasts.apple.com/cn/podcast/id1634356920?i=1000755501308 |
| 📝 Text version (official) | 《对话谢赛宁：逃出硅谷！》 — WeChat 公众号 "语言即世界 language is world" |
Every ▶ timestamp below deep-links into the video at that exact second.
- Saining Xie (谢赛宁) — co-founder & Chief Scientist, AMI Labs; faculty at NYU. 🌐 sainingxie.com · 𝕏 @sainingxie · GitHub @s9xie
- Zhang Xiaojun (张小珺) — award-winning tech journalist; host of 《商业访谈录》, founder of "语言即世界" studio. 𝕏 @zhang_benita · ▶ YouTube @benitazxj
Saining's own spontaneous enumeration of the deep-learning canon ("我可能数不全" — "I can't list them all"). Saining keeps his own work out of this tier: asked "DiT 不算吗?" he answers "算 0.25" — only a quarter.
The first convolutional neural network trained end-to-end with backpropagation, built to read handwritten digits (famously deployed on bank checks). It introduced the conv → pool → conv stack that every modern vision model still echoes — arriving a decade before the data and GPUs that would finally make it shine.
Widely credited with sparking the modern deep-learning boom. Winning ImageNet 2012 by a wide margin — using GPUs, ReLUs and dropout — it convinced the field that learned deep features, not hand-crafted ones, were the future.
Not a model but the dataset that made everything else possible: ~14M human-labeled images across 1000+ categories. Fei-Fei Li's bet that data, not just algorithms, was the real bottleneck — and the benchmark that drove a decade of progress.
Residual ("skip") connections fixed the vanishing-gradient problem and let networks scale to 100–1000+ layers. One of the most-cited papers in all of science, the default vision backbone for years, and the conceptual ancestor of the residual stream inside Transformers.
Brought CNN features to object detection — region proposals scored by a CNN. The recipe that launched the modern detection pipeline behind self-driving and medical imaging.
Made R-CNN end-to-end and far faster — a single pass over the image with RoI pooling, jointly predicting class and box. The basis for Faster R-CNN and Mask R-CNN.
Threw out recurrence and convolution, keeping only attention. It made training massively parallel and is the single architecture beneath GPT, BERT, CLIP, ViT and diffusion models — most of today's frontier AI.
8. Attention · Bahdanau, Cho & Bengio, 2014 ⚠️ garbled in transcript — see decoding
Introduced the attention mechanism for neural machine translation: instead of cramming a sentence into one vector, let the model "look back" at the relevant words. The seed idea that Transformers later scaled into everything.
At 175B parameters, it showed that scale alone unlocks emergent few-shot ability: describe a task in the prompt and the model just does it. The empirical proof of the scaling-law thesis and the direct ancestor of ChatGPT.
Bidirectional masked-language pretraining — hide words, predict them from both sides. It advanced the state of the art across NLP, made "pretrain then fine-tune" standard, and directly inspired Saining's own MAE for images.
Trained on 400M image–text pairs to align pictures and language in one space, enabling zero-shot classification and becoming the visual backbone of the text-to-image era. (Saining also notes its blind spots at 3:18.)
Showed that a plain Transformer, fed image patches as tokens, beats CNNs once data and compute are large enough — unifying vision and language under one architecture.
A generator and a discriminator locked in a minimax game. The framework that first made photorealistic synthesis possible and dominated generative modeling before diffusion.
Represents a 3D scene as a tiny neural network of color and density, rendering photorealistic novel viewpoints from a handful of photos. Reignited neural 3D and inverse graphics.
Replaces NeRF's slow volumetric sampling with millions of optimized 3D Gaussians, hitting real-time, high-fidelity rendering — quickly the new default for neural 3D.
Generate by learning to reverse a gradual noising process. DDPM made diffusion practical and surpassed GANs in quality and stability — the engine behind today's image and video generators.
Ran diffusion in a compressed latent space, cutting cost enough to make high-resolution generation fast — and, by open-sourcing it, kicked off the consumer generative-AI wave.
The interview is Chinese with English terms mixed in, and the auto-transcription mangled two paper names. Both are decoded here with evidence:
- "Tension Solution Unit" (~2:22:04) → an attention paper. High confidence on "attention," medium on the exact paper. This transcript reliably renders self-attention as "software attention" / "Software Tension" (2:58:30, 2:58:45) — so "Tension" = attention. Since Saining lists Transformer separately at #6, the best fit is the original attention paper, Bahdanau et al. 2014.
⚠️ Confirm by ear at ~2:22:04–2:22:06 — issues/PRs welcome.
Discussed in depth, but Saining keeps these out of the "great" tier (he rates DiT "0.25").
"Deep supervision" — attach classifiers/losses to intermediate layers so gradients reach early ones directly. A recurring motif in his work; he notes REPA is structurally a deeply-supervised net.
Reframed edge detection as image-to-image prediction with multi-scale deep supervision, setting a new bar. The PhD work he says he's "proud of."
Added "cardinality" — parallel grouped transformations — as a new scaling axis beyond depth and width. Kaiming He named it ("X = next, the next-gen ResNet"); it placed 2nd in the ImageNet challenge.
Replaced the U-Net in diffusion with a Transformer, so image generators inherit the clean scaling behavior of LLMs. The backbone behind Sora and modern video models.
Recasts DiT through stochastic interpolants / flow matching — a more flexible, often better-performing training formulation, now a common academic baseline alongside DiT.
Built a large, consistent dictionary for contrastive self-supervised learning — the first time self-supervised features rivaled supervised ImageNet pretraining on vision tasks. (Kaiming He–led; Saining is a co-author.)
Mask 75% of an image and reconstruct it — a simple, scalable self-supervised recipe that brought BERT's idea to vision. (Kaiming He–led; Saining is a co-author.)
A vision-centric multimodal LLM and a study of how visual representations should connect to language models — Saining's argument that vision deserves first-class treatment, not a bolt-on.
Align a diffusion Transformer's internal features to a strong self-supervised encoder, dramatically speeding up training. "Another keyword: representation," as Saining puts it.
Use a powerful pretrained representation as the generative model's encoder/foundation — pushing the "representation-first" thesis one step further.
Mentioned in passing or as influences — the RL & world-model lineage, plus the systems and concepts Saining name-drops.
Sutton's model-based RL — learn a world model and plan with it; the reactive vs. model-based "System 1 / System 2" framing Saining invokes. (Shown: Richard Sutton — Dyna's 1991 diagram has no clean digital figure.)
General methods that scale with compute beat hand-crafted knowledge — Saining's foil, who argues LLMs are "anti–Bitter Lesson." (Shown: Richard Sutton — the essay has no figure.)
LeCun's blueprint for autonomous machine intelligence — predict in abstract representation space, not pixels or tokens. AMI Labs' north star. (Figure: the I-JEPA instantiation.)
A foundation world model trained from unlabeled internet videos that generates playable, interactive environments.
Encoder–decoder with skip connections for image-to-image tasks — the diffusion backbone DiT replaced.
Contrastive Predictive Coding — learn representations by predicting the future in a latent space; an early self-supervised milestone.
Adds a mask branch to Faster R-CNN for pixel-level instance segmentation. (Part of the detection line; mentioned, not in Tier 1.)
Down-weights easy examples so dense one-stage detectors aren't swamped by background — powering RetinaNet.
Contemporary systems / tools name-dropped Sora (openai.com/sora, cites his DiT) · SeeDance · Nano Banana · Gemini / ChatGPT · a video model likely Veo · PyTorch (pytorch.org)
Concepts / awards (not papers) Neural Architecture Search (NAS, reflected on critically, 1:13:52) · Scaling Law (4:10:33) · Marr Prize nomination for his PhD paper (0:50:29)
Cultural references: 《金刚经》 Diamond Sutra (on research taste); 《万神殿》 Pantheon, TV series by Ken Liu 刘宇坤.
- Pulled the full verbatim transcript — 4,926 timestamped segments — of the episode.
- Scanned every segment for model / paper / architecture names, then read each hit in context.
- Profiled the transcription's failure modes (e.g. self-attention → "software attention", inductive bias → "index bias") to decode garbled names.
- Separated three tiers: (1) papers Saining himself calls great, (2) his own work, (3) everything else referenced.
This matters because online recaps were wrong: they listed ConvNeXt and V-JEPA as mentioned, but those names never appear in the actual audio (0 hits) — they were imported from Saining's CV. The verbatim transcript fixes that.
- ✅ Every paper with an official / arXiv link and a ▶ timestamp that deep-links into the video at that second.
- ✅ Three tiers — separates "papers Saining calls great" from "Saining's own work" from "merely referenced."
- ✅ Garbled-name forensics — decoded with evidence and the exact second to verify by ear.
- ✅ Built from the complete transcript, not secondhand summaries.
Content licensed CC BY 4.0. Paper figures are © their respective authors, shown for educational reference; LeNet, ImageNet, and the Richard Sutton portraits (Dyna, Bitter Lesson) are from Wikimedia Commons. Interview by Zhang Xiaojun (张小珺) with Saining Xie (谢赛宁). Unofficial, fan-made study aid; all rights to the interview belong to its creators.
Spotted a paper that's missing, or can confirm the garbled name at ~2:22:04? Open an issue or PR.