Skip to content

baseline.md

Latest commit

 

History

History
650 lines (487 loc) · 26.4 KB

baseline.md

File metadata and controls

650 lines (487 loc) · 26.4 KB
Raw

Baselines

This repo ships multiple reference baselines for ATMBench under:

  • Run scripts: scripts/QA_Agent/<Baseline>/
  • Baseline implementations: memqa/qa_agent_baselines/<Baseline>/

This document explains:

  1. what each baseline is measuring,
  2. why it is implemented the way it is (design rationale), and
  3. the important knobs (CLI flags and environment variables) you may want to change.

If you are looking for the expected local data layout, start with docs/data.md.

Baseline Principles (Why they look similar)

1) Evidence-grounded answering

ATMBench is an evidence-grounded benchmark. Most baselines use prompts that:

  • instruct the model to answer only from retrieved evidence, and
  • return Unknown when evidence is insufficient.

This reduces “hallucinated correctness” and makes retrieval quality measurable.

2) Text-only vs raw-media modes

Many baselines accept:

  • --media-source batch_results: treat images/videos as text via precomputed batch_results.json (captions/OCR/tags).
  • --media-source raw: load raw images/videos and insert them into the prompt (multimodal inference).

Rationale:

  • batch_results is cheaper, faster, and easier to reproduce.
  • raw is closer to “true multimodal QA”, but it is slower and depends on a multimodal LLM endpoint.

3) “Memory/index model” vs “answerer model”

Several baselines are two-stage or two-model:

  • a memory/index model for ingestion / extraction / note building
  • an answerer model for final QA

Rationale:

  • ingestion is often the expensive “O(#memories)” stage; you can use a smaller model here,
  • QA is “O(#questions)”; you can use a larger model for higher answer quality.

In this repo’s canonical run scripts, the defaults follow:

  • 2B model for memory/indexing
  • 8B model for answering

You can override models via environment variables in the run.sh scripts.

4) Avoiding evaluation contamination

Some memory systems update memory after answering a question (e.g., appending the QA itself). That contaminates evaluation because later questions can “cheat”.

Where applicable we implement:

  • per-question isolation (fresh instance restored from a checkpoint), and/or
  • no-update evaluation mode (skip post-answer writes).

5) Reproducibility and caching

Baselines write only to output/ and are designed to be restartable:

  • most runs are keyed by a --method-name (output subdirectory),
  • long stages cache indexes/checkpoints,
  • many scripts will skip work when outputs already exist.

Setup (One-time)

From repo root:

conda create -n atmbench python=3.11 -y
conda activate atmbench

pip install -r requirements.txt
pip install -e .

Baseline-specific dependencies are intentionally not all in root requirements.txt. Install what you need:

pip install -r memqa/qa_agent_baselines/HippoRag2/requirements.txt
pip install -r memqa/qa_agent_baselines/MemoryOS/requirements.txt
pip install -r memqa/qa_agent_baselines/A-Mem/requirements.txt
pip install -r memqa/qa_agent_baselines/mem0/requirements.txt

Notes:

  • Some baselines have heavy/native deps (e.g., python-igraph, faiss-*). Install them per your platform.
  • Some baselines vendor upstream code under third_party/ (see third_party/README.md).

Environment Recommendations

  • Core baselines (MMRAG, Oracle, NIAH) are tested in the main atmbench environment.
  • Third-party memory-system baselines in this repo are:
    • A-Mem
    • HippoRAG2
    • mem0
    • MemoryOS
    • MemPalace
  • MemoryOS and MemPalace are strongly recommended to run in separate conda environments. MemoryOS uses a FAISS / sentence-transformers stack, while MemPalace uses ChromaDB / ONNX-backed local embeddings; isolating them avoids dependency collisions with the core baseline environment and each other.
  • A-Mem, HippoRAG2, and mem0 are tested to be compatible with the core baseline environment, but separate environments are still safer for dependency isolation.
  • Setup for the vendored baselines is documented under:
    • third_party/A-mem/
    • third_party/HippoRAG/
    • third_party/mem0/
    • third_party/MemoryOS/
  • MemPalace is installed from PyPI (mempalace==3.3.5); see memqa/qa_agent_baselines/Mempalace/README.md.
  • OpenClaw, OpenCode, and Codex baselines are compatible with this repo’s evaluation workflow, but each requires its own third-party software installation.

Vendored upstream code (third_party)

For reproducibility, some baselines vendor upstream code directly into this repo under third_party/.

What to expect in a vendored subtree:

  • LICENSE (upstream license)
  • UPSTREAM.md (source URL + pinned commit/tag)
  • optional ATMBench.patch (local changes applied on top of upstream snapshot)

Baselines that rely on vendored code:

  • HippoRAG 2: third_party/HippoRAG/
  • MemoryOS: third_party/MemoryOS/ (baseline can also use a pip-installed memoryos package)
  • A‑Mem: third_party/A-mem/
  • Mem0: third_party/mem0/ (baseline can also use a pip-installed mem0 / mem0ai)

Common Runtime Knobs

Endpoints and keys

Most scripts assume an OpenAI-compatible Chat Completions endpoint:

  • VLLM_ENDPOINT (default: http://127.0.0.1:8000/v1/chat/completions)
  • VLLM_API_KEY (optional; picked up by code if your endpoint requires auth)
  • OPENAI_API_KEY (required only for baselines using --provider openai or for LLM-judge eval)

This repo supports providers:

  • --provider vllm: OpenAI-compatible HTTP endpoint (vLLM, etc.)
  • --provider openai: OpenAI API (Responses API preferred when available)
  • --provider vllm_local: local vLLM Python runtime (used by some baselines)

Data paths

All canonical scripts assume the docs/data.md layout:

  • QA files:
    • data/atm-bench/atm-bench.json
    • data/atm-bench/atm-bench-hard.json
  • Text evidence for media:
    • output/image/qwen3vl2b/batch_results.json
    • output/video/qwen3vl2b/batch_results.json
  • Optional emails:
    • data/raw_memory/email/emails.json

Generating batch_results.json (required for --media-source batch_results)

Most baselines are text-first and expect batch_results.json to exist. In this repo, the default baseline scripts read directly from:

  • output/image/qwen3vl2b/batch_results.json
  • output/video/qwen3vl2b/batch_results.json

Note on GPS / reverse-geocoding:

  • By default, the processors reverse-geocode GPS coordinates via a public provider (OpenStreetMap Nominatim).
  • Public geocoding endpoints are rate-limited (often strict per-IP requests/minute) and do not tolerate high concurrency.
  • If you have a pre-extracted GPS cache bundle, place the *_location_name.json entries under data/raw_memory/geocoding_cache/image and data/raw_memory/geocoding_cache/video, then copy them into your processor cache directory before running the memory processors so geocoding is skipped and runs don’t stall.
  • GPS cache files are keyed by the media filename stem, so the cache bundle must match the current image/video filenames.

Example (Qwen3-VL-2B captioner via vLLM):

# Optional (recommended): copy pre-extracted GPS cache to skip geocoding calls
python memqa/utils/copy_gps_info.py data/raw_memory/geocoding_cache/image output/image/qwen3vl2b/cache
python memqa/utils/copy_gps_info.py data/raw_memory/geocoding_cache/video output/video/qwen3vl2b/cache

# Generate batch results
bash scripts/memory_processor/image/memory_itemize/run_qwen3vl2b.sh
bash scripts/memory_processor/video/memory_itemize/run_qwen3vl2b.sh

The run_qwen3vl2b.sh wrappers write these files directly under output/image/qwen3vl2b/ and output/video/qwen3vl2b/, which is what the QA baselines read by default.

Output directories

Outputs live under output/QA_Agent/<Baseline>/.... If you want to re-run with different settings, prefer changing:

  • --method-name (creates a new subdir), or
  • --output-dir-base (separates experiment families).

Baselines

Oracle (Upper Bound Answering)

What it is: An upper bound baseline that uses ground-truth evidence IDs from the dataset and asks the model to answer.

What it measures: Answer synthesis quality given perfect retrieval (not a retrieval baseline).

Default script:

  • bash scripts/QA_Agent/Oracle/run_oracle_qwen3vl8b_raw.sh

Related scripts:

  • scripts/QA_Agent/Oracle/run_oracle_no_evidence_qwen3vl8b.sh (answers with no evidence; sanity check)
  • scripts/QA_Agent/Oracle/run_oracle_gpt5.sh (OpenAI answerer variant)

Implementation:

  • memqa/qa_agent_baselines/oracle/oracle_baseline.py

Key design choices:

  • Uses dataset-provided evidence IDs, so retrieval recall is not meaningful here.
  • Typically uses --media-source raw so the model sees the true pixels for image/video evidence.

Important knobs (script-level):

  • VLLM_ENDPOINT: answerer endpoint
  • --model: answerer model
  • --max-workers: QA concurrency

Common CLI flags (implementation):

  • --qa-file: QA JSON input
  • --output-file: where to write answers JSONL
  • --use-niah-pools: used by NIAH wrapper to swap in fixed evidence pools
  • --media-source {raw,batch_results}: raw pixels vs text-only evidence
  • --provider {vllm,openai,vllm_local} / --vllm-endpoint / --model: answerer backend
  • --timeout / --max-workers: reliability + throughput

Outputs (typical):

  • <output>/oracle_*.jsonl (answers)

MMRAG (Retrieve → Answer Baseline)

What it is: A strong “standard” baseline: retrieve top-k evidence with an embedding retriever, then answer with an evidence-only prompt.

What it measures: End-to-end retrieval + answering, with minimal “memory system” behavior.

Default script:

  • bash scripts/QA_Agent/MMRAG/run.sh

Key design choices:

  • Default mode is batch_results (text-only) to keep runs cheap and reproducible.
  • Retrieval is pluggable (--retriever) to support text-only and VL embedding retrieval.
  • Uses resume/checkpoint JSONL files so partial runs can be resumed.

Important knobs:

  • TOP_K (script env var): number of retrieved items used for answering
  • TEXT_EMBED_MODEL (script env var): text embedding model for sentence_transformer retriever
  • --retriever: sentence_transformer, text, qwen3_vl_embedding, clip, vista
  • --reuse-retrieval-results/--no-reuse-retrieval-results: cache retrieval outputs
  • --insert-raw-images: optionally attach raw images even when retrieval is text-only
  • --critic-answerer: draft + critic verification (slower; can reduce unsupported answers)

Common CLI flags (most used):

  • Data:
    • --qa-file
    • --media-source {batch_results,raw}
    • --image-batch-results, --video-batch-results, --email-file
  • Retrieval:
    • --retriever, --retrieval-top-k
    • --text-embedding-model (text retrievers)
    • --vl-embedding-model / --clip-model / --vista-* (VL retrievers)
    • --force-rebuild, --index-cache, --reuse-retrieval-results
  • Answering:
    • --provider, --vllm-endpoint, --model
    • --no-evidence (answer without evidence; debug)
    • --max-workers, --timeout

Where to look for full CLI: memqa/qa_agent_baselines/MMRag/mmrag_retrieve_answer.py

Baseline README: memqa/qa_agent_baselines/MMRag/README.md

Outputs (typical):

  • <output>/mmrag_answers.jsonl
  • <output>/retrieval_recall_details.json
  • <output>/retrieval_recall_summary.json

HippoRAG 2 (Graph Memory Baseline)

What it is: A graph-based memory baseline (HippoRAG 2) that builds a knowledge graph via OpenIE and retrieves via Personalized PageRank + reranking.

What it measures: Whether structured graph memory improves retrieval compared to pure embedding baselines.

Default script (2B index/OpenIE, 8B answerer):

  • bash scripts/QA_Agent/HippoRag2/run.sh

Key design choices:

  • We vendor upstream under third_party/HippoRAG and default --hipporag-repo points there.
  • Two-stage pipeline:
    • --stage build: build HippoRAG index (OpenIE, graph construction)
    • --stage answer: run QA using the cached index
  • Index cache is keyed on embedding model + OpenIE model + augmentation level + corpus hash.

Important knobs:

  • MODEL (script env var): index/OpenIE model (default 2B)
  • ANSWERER_MODEL (script env var): QA/rerank model (default 8B)
  • AUGMENTATION_LEVEL: how much text is included from batch_results (caption-only vs caption+tags+ocr)
  • --openie-workers: parallelism for OpenIE calls
  • --qa-top-k: number of retrieved passages provided to the answerer

Common CLI flags (most used):

  • Pipeline:
    • --stage {build,answer,all}
    • --force-rebuild (rebuild index)
    • --force-openie (re-run OpenIE extraction)
  • Upstream integration:
    • --hipporag-repo (defaults to third_party/HippoRAG)
  • Indexing/retrieval:
    • --embedding-model
    • --augmentation-level
    • --retrieval-top-k, --linking-top-k, --qa-top-k
    • --openie-mode, --openie-workers
  • Answering:
    • --provider, --vllm-endpoint, --model (index/OpenIE model)
    • --answerer-model, --answerer-endpoint (QA/rerank model)
  • Output:
    • --index-cache (index cache root)
    • --output-dir-base, --method-name

Dependencies:

  • Requires python-igraph (and other deps in memqa/qa_agent_baselines/HippoRag2/requirements.txt).

Where to look for full CLI: memqa/qa_agent_baselines/HippoRag2/hipporag2_baseline.py

Baseline README: memqa/qa_agent_baselines/HippoRag2/README.md

Outputs (typical):

  • <output>/hipporag2_answers.jsonl
  • <output>/retrieval_recall_details.json

MemoryOS (Tiered Memory System Baseline)

What it is: An OS-inspired tiered memory system (STM/MTM/LPM) adapted for batch QA.

What it measures: Whether tiered memory management + session/page retrieval helps on long-horizon memory QA.

Default script (2B index, 8B answerer):

  • bash scripts/QA_Agent/MemoryOS/run.sh

Key design choices:

  1. Chronological indexing
  • We merge emails/images/videos and sort by timestamp before ingestion.
  1. Per-question isolation (prevents contamination)
  • By default we restore from a checkpoint for each QA so the QA itself is not added to memory for later QAs.
  1. Full-history mode (benchmark variant)
  • Paper-default capacities (e.g., MTM≈2000) can aggressively summarize/evict historical items.
  • For large memory banks, this can produce very poor retention and can degrade ingestion/runtime behavior.
  • We therefore provide --memoryos-full-history-mode which enforces floors on MTM capacity and session search.

In this repo’s canonical run.sh, full-history mode is enabled by default:

  • FULL_HISTORY_MODE=1 (script-level)
  • method name is suffixed as memoryos_fullhistory to make the variant explicit.

Paper defaults vs full-history variant (what changes):

  • Paper-ish defaults (see memqa/qa_agent_baselines/MemoryOS/config.py):
    • memoryos_short_term_capacity=10
    • memoryos_mid_term_capacity=2000
    • small retrieval/search depths (paper uses small queue/session counts)
  • Full-history variant (this repo’s benchmark setting):
    • memoryos_mid_term_capacity >= 20000
    • memoryos_top_k_sessions >= 200
    • memoryos_heat_threshold >= 1e9 (suppresses MTM→LPM promotion churn)

Practical implication on large memory banks:

  • paper-ish defaults can retain only ~MTM capacity worth of page-level history and can become slow/unstable;
  • full-history mode retains substantially more historical items and keeps ingestion behavior stable.

Important knobs:

  • TOP_K: retrieval queue capacity (pages) used during evaluation
  • TOP_K_SESSIONS: MTM session search depth
  • EVAL_NO_UPDATE: if 1, passes --memoryos-eval-no-update to skip post-answer writes
  • FULL_HISTORY_MODE: enable/disable the benchmark full-history variant

Common CLI flags (most used):

  • Models:
    • --provider, --vllm-endpoint
    • --memoryos-index-llm-model (indexing LLM)
    • --memoryos-answer-llm-model (answering LLM)
    • --memoryos-embedding-model
  • Contamination control:
    • --memoryos-use-per-qa-instance (isolate each QA)
    • --memoryos-eval-no-update (skip post-answer writes)
  • Checkpointing:
    • --memoryos-checkpoint-dir
    • --memoryos-reuse-checkpoint/--no-memoryos-reuse-checkpoint
    • --memoryos-save-checkpoint/--no-memoryos-save-checkpoint
    • --memoryos-force-rebuild
  • Retrieval/capacity:
    • --memoryos-retrieval-queue-capacity
    • --memoryos-top-k-sessions
    • --memoryos-full-history-mode (+ its --memoryos-full-history-* floors)

Where to look for full CLI: memqa/qa_agent_baselines/MemoryOS/memoryos_baseline.py

Baseline README: memqa/qa_agent_baselines/MemoryOS/README.md

Outputs (typical):

  • <output>/memoryos_answers.jsonl
  • <output>/retrieval_recall_details.json

A‑Mem (Agentic Memory Baseline)

What it is: An “agentic memory” system that analyzes each memory item with an LLM to create structured notes (keywords/context/tags) and optionally evolves links.

What it measures: Whether LLM-augmented memory construction + link traversal improves retrieval/QA.

Default script (2B memory, 8B answerer, text-only construction):

  • bash scripts/QA_Agent/A-Mem/run.sh

Key design choices:

  1. Two-stage pipeline with explicit caching
  • --stage build constructs memories and caches them to disk.
  • --stage answer loads the cache and answers questions.
  1. Batch-results (text-only) by default in this repo
  • The default run script builds from batch_results.json for speed/reproducibility.
  • You can switch to raw-media construction/answering with:
    • --construct-from-raw (Stage 1)
    • --answer-from-raw (Stage 2)

What we used in development experiments:

  • The primary A‑Mem scripts used raw image construction (--construct-from-raw) with minimal text fields (ID/timestamp/location only), and then answered with a larger model.
  • An older archived variant built from batch_results (--caption-only --use-short-caption). This repo keeps raw mode as an optional variant because it is slower and more endpoint-dependent.

Important knobs:

  • MEMORY_MODEL: memory/index model (default 2B)
  • ANSWER_MODEL: answerer model (default 8B)
  • --index-cache: cache directory (must be shared between Stage 1 and Stage 2)
  • --disable-evolution: disable link evolution (faster, simpler)
  • --follow-links/--no-follow-links: whether retrieval traverses memory links
  • --caption-only: minimal text mode (useful when comparing captioners)
  • --construct-from-raw / --answer-from-raw: raw image variants (slower; endpoint-dependent)

Correctness pitfall (common):

  • Stage 2 must reuse the same cache key as Stage 1. That means cache-affecting args must match: --image-batch-results, --video-batch-results, --email-file, --embedding-model, and memory-LLM settings like --memory-model / --memory-provider.

Common CLI flags (most used):

  • Pipeline:
    • --stage {build,answer,all}
    • --index-cache (cache dir)
    • --resume, --force-rebuild, --checkpoint-interval
  • Memory construction:
    • --memory-provider, --memory-vllm-endpoint, --memory-model
    • --memory-workers (parallel note construction)
    • --disable-evolution, --evo-threshold
  • Retrieval/answer:
    • --retrieve-k, --follow-links
    • --provider, --vllm-endpoint, --model (answerer)
    • --max-workers (QA concurrency)
  • Text mode:
    • --caption-only, --use-short-caption
    • fine-grained --include-* toggles

Where to look for full CLI: memqa/qa_agent_baselines/A-Mem/amem_baseline.py

Baseline README: memqa/qa_agent_baselines/A-Mem/README.md

Outputs (typical):

  • <output>/amem_answers.jsonl
  • <output>/retrieval_recall_details.json

Mem0 (Memory Extraction + Vector Store Baseline)

What it is: A mem0-backed baseline that indexes memories into a vector store and retrieves top-k “memories” for QA.

What it measures: Whether mem0’s memory representation helps retrieval/QA on ATMBench-style evidence.

Important note about mem0 extraction:

mem0 supports an LLM-based extraction/update pipeline (--mem0-infer) that is optimized for conversational personal facts. For structured batch-result style items, the upstream extraction prompt often returns empty facts, producing a near-empty index.

Therefore, the canonical script defaults to:

  • MEM0_INFER=0 → --no-mem0-infer (store memories directly as embeddings)

If you want to try extraction, set:

  • MEM0_INFER=1
  • USE_CUSTOM_EXTRACTION_PROMPT=1 (uses an ATMBench-adapted extraction prompt)

Default script (8B answerer, local ST embedder, no-extraction indexing):

  • bash scripts/QA_Agent/mem0/run.sh

Key design choices:

  1. Stable indexing by explicit paths
  • The script pins:
    • --mem0-vector-path (Chroma dir)
    • --mem0-history-db-path
    • --mem0-progress-path
    • --mem0-collection-name so that atmbench and hard do not accidentally double-index into the same collection.
  1. Local embedding by default
  • Uses --mem0-embedder-provider local + --mem0-local-retriever sentence_transformer with all-MiniLM-L6-v2 for reproducibility.

Important knobs:

  • TEXT_EMBEDDING_MODEL: sentence-transformers model used for indexing/retrieval
  • MEM0_LOCAL_DEVICE: cpu or cuda (if you want local embeddings on GPU)
  • MEM0_LLM_MODEL + MEM0_LLM_BASE_URL: used only when MEM0_INFER=1 (extraction)

Common CLI flags (most used):

  • Indexing storage (avoid accidental double-indexing):
    • --mem0-collection-name
    • --mem0-vector-path
    • --mem0-history-db-path
    • --mem0-progress-path
    • --mem0-user-id
  • Extraction behavior:
    • --mem0-infer/--no-mem0-infer
    • --use-custom-extraction-prompt
    • --mem0-llm-model, --mem0-llm-base-url
  • Embedding behavior:
    • --mem0-embedder-provider local
    • --mem0-local-retriever sentence_transformer
    • --text-embedding-model
  • QA:
    • --mem0-top-k
    • --provider, --vllm-endpoint, --model
    • --max-workers, --timeout

Where to look for full CLI: memqa/qa_agent_baselines/mem0/mem0_baseline.py

Baseline README: memqa/qa_agent_baselines/mem0/README.md

Outputs (typical):

  • <output>/mem0_answers.jsonl
  • <output>/retrieval_recall_details.json

MemPalace (Hybrid Drawer + Closet Retrieval Baseline)

What it is: A wrapper around the upstream mempalace PyPI package (3.3.5) that stores each ATM-Bench memory item as a chunked "drawer" plus a regex topic/entity "closet" pointer index, then retrieves with hybrid BM25 + cosine reranking, closet boost, and drawer-grep neighborhood enrichment.

Default script (vLLM 8B answerer, local ONNX embedding, hybrid retrieval):

  • bash scripts/QA_Agent/Mempalace/run.sh

Key design choices:

  1. All retrieval primitives come from upstream mempalace
  • chunk_text (CHUNK_SIZE=800 / OVERLAP=100 / MIN=50), _build_drawer_metadata, build_closet_lines, purge_file_closets, upsert_closet_lines, and search_memories are called directly. Nothing is re-implemented.
  1. Virtual source files (atmbench://<item_id>)
  • ATM-Bench items are not files on disk, so process_file() cannot be used. We inline its pipeline (chunk_text → _build_drawer_metadata → batched collection.upsert → build_closet_lines → upsert_closet_lines) so the resulting Chroma collection is byte-equivalent to what process_file would write for an on-disk corpus.
  1. Wing = modality, room = sanitized item ID
  • Upstream detect_room walks a project directory, which is meaningless here. We use modality (email/image/video) for the wing and the item ID for the room.
  1. Over-fetch for recall reporting
  • --n-results defaults to 100 (upstream default is 5). The full ranked list is persisted in retrieval_recall_details.json so R@{1,5,10,25,50,100} can be computed; only the top --retrieve-k (default 10) items are passed to the answerer.

Important knobs:

  • TOP_K / --retrieve-k: evidence items per question
  • N_RESULTS / --n-results: rerank candidate pool size
  • RETRIEVAL_K_VALUES: comprehensive retrieval eval k values (default: 1,5,10,25,50,100)
  • --candidate-strategy {vector,union}: pure vector candidates vs vector ∪ BM25
  • --max-distance: cosine distance cutoff (0.0 disables filtering)
  • ANSWERER_MODEL / --model: answerer LLM
  • REBUILD_INDEX=1 / --force-rebuild: rebuild the palace cache

Correctness pitfall:

  • The build and answer stages must use the same cache-affecting args (batch-results paths, email file, all --include-* toggles, and --limit when used for a debug index). Changing any of them produces a new cache_key and the answer stage will refuse to run until you rebuild.

WSL / NTFS HNSW quarantine:

  • Chroma occasionally quarantines the HNSW index on first query when the palace lives on NTFS. The answer stage runs three warmup search_memories calls before the parallel QA loop for this reason — keep them.

Where to look for full CLI: memqa/qa_agent_baselines/Mempalace/mempalace_baseline.py

Baseline README: memqa/qa_agent_baselines/Mempalace/README.md

Outputs (typical):

  • <output>/mempalace_answers.jsonl
  • <output>/retrieval_recall_details.json
  • <output>/retrieval_recall_comprehensive_summary.json
  • <output>/retrieval_recall_joint_accuracy_summary.json

NIAH (Needle-In-A-Haystack; Generation-only)

What it is: A generation-only protocol for the hard split where each question includes a fixed evidence pool (niah_evidence_ids) guaranteed to contain the ground truth.

What it measures: Answer synthesis under a fixed pool size k (not retrieval).

Packaged scripts:

  • bash scripts/QA_Agent/NIAH/run_niah_qwen3vl8b_SGM.sh
  • bash scripts/QA_Agent/NIAH/run_niah_gpt5_SGM.sh
  • bash scripts/QA_Agent/NIAH/run_niah_qwen3vl8b_raw.sh
  • bash scripts/QA_Agent/NIAH/run_niah_gpt5_raw.sh

How it is implemented here:

  • memqa/qa_agent_baselines/NIAH/niah_evaluate.py is a wrapper around the Oracle baseline: it swaps niah_evidence_ids -> evidence_ids and then calls oracle_baseline.py.
  • The *_SGM.sh scripts use --media-source batch_results with the paper's SGM fields: type,timestamp,location,short_caption,caption,ocr,tags.
  • The *_raw.sh scripts use --media-source raw and pass raw image/video inputs.

Common CLI flags (wrapper):

  • --qa-file / --niah-qa-file: NIAH pool QA file (must contain niah_evidence_ids)
  • --niah-field: field name for pools (default: niah_evidence_ids)
  • All other flags are passed through to oracle_baseline.py (provider/model/media flags, output path, etc.)

See docs/niah.md for schema and usage.