radiomicslab-keyword-extraction

An end-to-end pipeline for extracting clinical entities and relations from radiology reports using LLMs and the RadGraph-XL schema, designed to test LLM performance and produce a gold-standard dataset in Italian.

---

Table of Contents

• Overview
• Features
• Project Structure
• Requirements / Installation
• Usage
• Configuration
• Data
• License
• Author

---

Overview

This project is the Bachelor's Thesis work of Daniel Rabottini, developed at the University of Turin. It implements an end-to-end pipeline for extracting clinical entities and their relations from radiology reports, following the RadGraph-XL annotation schema. The pipeline leverages large language models via LiteLLM, supports optional EN → IT translation of reports with token-level alignment, and provides integration with the Doccano annotation platform for gold-standard curation. An interactive CLI built with Rich and Questionary drives the entire workflow.

The full thesis document is available in the docs/ directory.

---

Features

• LLM-based entity & relation extraction — Structured prompting (0-shot / 1-shot) against OpenAI and Google models via LiteLLM
• RadGraph-XL schema compliance — Atomic single-token entities, certainty labels, and typed relations
• F1-score evaluation — Micro/macro precision, recall, and F1 against gold-standard annotations with per-class breakdown
• EN → IT medical translation — LLM-driven translation with inflection-aware token mapping
• Token-level entity mapping — spaCy-based tokenization and alignment of extracted entities to character offsets
• Doccano integration — Bidirectional converters: gold → Doccano import, Doccano export → RadGraph-like format, RadGraph inference → Doccano
• Interactive CLI — Rich panels, progress bars, and Questionary menus for guided operation
• Centralized configuration — Single settings.yaml for models, paths, prompts, and output patterns

---

Project Structure

radiomicslab-keyword-extraction/
├── configs/
│   ├── prompts/             # System & user prompt templates (.txt)
│   └── settings.yaml        # Centralized configuration
├── data/                    # Input/output data directories (see Data section)
├── docs/                    # Thesis PDF
├── logs/                    # Runtime log files
├── src/
│   ├── cli/                 # Interactive CLI (Rich + Questionary)
│   ├── llm/                 # LLM gateway (LiteLLM) + prompt builder
│   ├── radgraph/            # RadGraph extraction models + F1 evaluation
│   ├── translate/           # EN → IT translation service
│   ├── map/                 # Token-level entity mapper
│   ├── preprocess/          # spaCy tokenizer wrapper
│   ├── doccano/             # Doccano import/export converters
│   ├── utils/               # Logger, I/O helpers
│   └── config.py            # YAML loader and path constants
├── main.py                  # Application entry point
├── requirements.txt
└── README.md

---

Requirements / Installation

Prerequisites

• Python ≥ 3.10
• API keys set as environment variables (used by LiteLLM):
  • OPENAI_API_KEY — for OpenAI models
  • GOOGLE_API_KEY — for Google Gemini models

Install dependencies

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

---

Usage

Launch the interactive CLI:

python main.py

The main menu presents four options:

Option	Description
Extract LLM	Run the entity extraction pipeline — select provider (OpenAI / Google), language (EN / IT), and number of reports to process
F1-Score	Evaluate extraction predictions against gold-standard annotations — select a prediction file and view micro/macro metrics with per-class detail
Doccano Management	Convert between formats: gold → Doccano import, Doccano export → RadGraph-like format, RadGraph inference → Doccano import, and more
Exit	Quit the application

Data Format Examples

Raw input (data/raw/*.jsonl) — one JSONL record per report:

{
  "dataset": "example-chest-xr", 
  "doc_key": 0,
  "text": "No pleural effusion. Small nodule in the right lung."
}

Gold standard (data/gold/*.jsonl) — tokenized sentences with NER spans and relations (RadGraph-XL schema). Each NER entry is [start_idx, end_idx, label]; each relation is [src_start, src_end, tgt_start, tgt_end, type]:

{
  "dataset": "example-chest-xr",
  "doc_key": 0,
  "sentences": [["No", "pleural", "effusion", ".", "Small", "nodule", "in", "the", "right", "lung", "."]],
  "ner": [
    [
      [1, 1, "Anatomy::definitely present"],
      [2, 2, "Observation::definitely absent"],
      [4, 4, "Observation::definitely present"],
      [5, 5, "Observation::definitely present"],
      [8, 8, "Anatomy::definitely present"],
      [9, 9, "Anatomy::definitely present"]
    ]
  ],
  "relations": [
    [
      [2, 2, 1, 1, "located_at"],
      [4, 4, 5, 5, "modify"],
      [5, 5, 9, 9, "located_at"],
      [8, 8, 9, 9, "modify"]
    ]
  ]
}

LLM extraction output (data/extract/*.json) — entity list produced by the pipeline. Each relation references a target entity by its array index:

[
  {
    "dataset": "example-chest-xr",
    "doc_key": 0,
    "text": "No pleural effusion. Small nodule in the right lung.",
    "entities": [
      {
        "tokens": "pleural",
        "label": "Anatomy::definitely present",
        "relations": [],
        "start_idx": 1, "end_idx": 1
      },
      {
        "tokens": "effusion",
        "label": "Observation::definitely absent",
        "relations": [["located_at", 0]],
        "start_idx": 2, "end_idx": 2
      },
      {
        "tokens": "Small",
        "label": "Observation::definitely present",
        "relations": [["modify", 3]],
        "start_idx": 4, "end_idx": 4
      },
      {
        "tokens": "nodule",
        "label": "Observation::definitely present",
        "relations": [["located_at", 5]],
        "start_idx": 5, "end_idx": 5
      },
      {
        "tokens": "right",
        "label": "Anatomy::definitely present",
        "relations": [["modify", 5]],
        "start_idx": 8, "end_idx": 8
      },
      {
        "tokens": "lung",
        "label": "Anatomy::definitely present",
        "relations": [],
        "start_idx": 9, "end_idx": 9
      }
    ],
    "sentences": ["No", "pleural", "effusion", ".", "Small", "nodule", "in", "the", "right", "lung", "."]
  }
]

---

Configuration

All tunable parameters are centralized in configs/settings.yaml:

llm:
  models:
    openai:    "gpt-5"
    google:    "gemini/gemini-2.5-pro"
    translate: "gemini/gemini-2.5-flash"
  default_model:       "gpt-4o"
  default_temperature: 1.0

spacy:
  model_it: "it_core_news_sm"
  model_en: "en_core_web_sm"

radgraph:
  model_type: "modern-radgraph-xl"

Key sections include LLM model identifiers, spaCy model names, RadGraph model type, data directory paths, prompt file paths, and output filename patterns.

Note: API keys must not be placed in settings.yaml. Set them as environment variables (OPENAI_API_KEY, GOOGLE_API_KEY) where LiteLLM will read them automatically.

---

Data

Input / Output Paths

All paths are defined in settings.yaml and are relative to the project root:

Path	Description	Format
data/raw/	Raw radiology reports	JSONL — fields: dataset, doc_key, text
data/gold/	Gold-standard annotations	JSONL — RadGraph-XL schema: dataset, doc_key, sentences, ner, relations
data/extract/	LLM extraction outputs	JSON
data/doccano/export/	Exports from Doccano	JSONL
data/doccano/import/	Files ready for Doccano import	JSONL
data/doccano/radgraph-like-format/	Gold annotations converted from Doccano	JSONL
data/doccano/radgraph-input-translate/	Input for RadGraph → Doccano flow	JSONL — fields: text_en, optionally text_it

RadGraph-XL Dataset

The project is designed to work with the RadGraph-XL annotated datasets:

• MIMIC reports (300 samples): Stanford AIMI — RadGraph-XL
• Stanford reports (2000 samples): PhysioNet — RadGraph-XL v1.0.0

---

Author

Daniel Rabottini Bachelor's Thesis — RadiomicsLab - University of Turin

The complete thesis document is available in docs/.