RunPod Serverless Worker for Mega-ASR

This repository provides a RunPod serverless worker wrapper around Mega-ASR, a full-scenario robust speech recognition foundation model. Built using the vLLM backend for fast Qwen3-ASR inference and materializing the Mega-ASR LoRA weights.

The interface and response payload structure match dembrane/runpod-whisper exactly, enabling plug-and-play replacement of Whisper workers with robust Mega-ASR workers.

Features

• vLLM-accelerated Inference: Runs on the fast vLLM engine with materialized Mega-ASR LoRA checkpoints.
• Pre-materialized Builds: LoRA weight materialization is performed during Docker build time, completely eliminating cold-start materialization delays.
• Audio Format Auto-normalization: Automatically converts any incoming audio (MP3, WAV, etc.) to a standardized 16kHz mono WAV using ffmpeg.
• Multilingual LLM-based Translation: Integrates with litellm to translate transcription parts that don't match the desired language.
• Heuristic-driven Hallucination Detection: Leverages LLMs to evaluate transcript coherence and rate hallucination scores on ASR outputs.

Repository Structure

dembrane/runpod-mega-asr/
├── Dockerfile                  # Builds the CUDA runtime with static FFmpeg and vLLM
├── requirements.txt            # Python dependencies (qwen-asr[vllm], runpod, litellm, etc.)
├── download_weights.py         # Warm-up script to preload and materialize weights during build
├── handler.py                  # RunPod serverless entrypoint and processing pipeline
├── test_input.json             # Test payload format
└── src/
    └── MegaASR/                # Sub-package copied from the Mega-ASR codebase
        ├── __init__.py
        └── model/
            ├── Qwen3_ASR.py
            ├── Qwen3_ASR_vllm.py
            ├── megaASR.py
            ├── router.py
            └── utils/          # Materialization and routing utilities

API Input Interface

The worker expects requests in the following format:

{
  "input": {
    "audio": "https://example.com/audio.mp3",
    "audio_base_64": "...", // Optional: base64 encoded audio instead of URL
    "language": "nl",       // Target translation language code (e.g., 'nl', 'es')
    "hotwords": "Word1,Word2", // Optional: hotwords to preserve during ASR and translation
    "enable_timestamps": true,  // Returns segment-level transcription timestamps
    "disable_hallucination_detection": false,
    "disable_translation": false,
    "conversation_id": "conv-123",
    "conversation_chunk_id": "chunk-456",
    "metadata_str": "custom metadata string"
  }
}

Response Payload

The response matches the dembrane/runpod-whisper format exactly:

{
  "conversation_id": "conv-123",
  "conversation_chunk_id": "chunk-456",
  "metadata_str": "custom metadata string",
  "enable_timestamps": true,
  "language": "nl",
  "detected_language": "en",
  "detected_language_confidence": 1.0,
  "joined_text": "Original English transcription text...",
  "translation_text": "Translated transcription text in Dutch...",
  "translation_error": null,
  "hallucination_score": 0.1,
  "hallucination_reason": "coherent text",
  "segments": [
    {
      "text": "Segment text...",
      "start": 0.0,
      "end": 2.5,
      "words": null
    }
  ]
}

Deployment & RunPod Configuration

Build the Docker image:

docker build -t dembrane/runpod-mega-asr:latest .

Push to your registry and configure the RunPod Serverless template using:

• Container Image: dembrane/runpod-mega-asr:latest
• Docker Command: python -u handler.py
• Environment Variables:
  • LITELLM_MODEL: (e.g., openai/gpt-4o-mini)
  • LITELLM_API_KEY: your-api-key
  • GPU_MEMORY_UTILIZATION: 0.85 (default)
  • MAX_MODEL_LEN: 8192 (default)
  • MAX_NUM_SEQS: 1 (default)
  • MAX_NUM_BATCHED_TOKENS: 2048 (default)