Add a note to the User Guide explaining that VoxHumana is designed for single-speaker recordings (one participant, one interviewer) and that this is the optimal input for MFA. Include a concrete example of what that looks like:
- One audio file per speaker or per interview session
- The target speaker should be the dominant voice
- Background noise, cross-talk, and multiple simultaneous speakers will degrade alignment quality
Also explain what "single speaker" means in MFA terms (--single_speaker mode) and why
VoxHumana doesn't expose it as a toggle: for typical fieldwork interviews, MFA's default
speaker-adaptive mode performs better and the distinction is unlikely to matter unless the
user is processing something unusual (e.g., a group conversation or a read-aloud wordlist
with no inter-speaker variation).
The Alignment section's acoustic model and dictionary dropdowns currently have only one
option each (english_us_arpa) and are disabled in the UI with a "coming soon" note.
Expanding them is blocked on the same multi-language MFA work described above.
When adding a new language:
- Install the MFA acoustic model and dictionary (
mfa model download acoustic <name>,mfa model download dictionary <name>). - Add the new
<option>to both dropdowns in the Alignment section. - Decide whether to auto-pair model and dictionary based on the Whisper language selection, or let the user choose them independently (independent choice is more flexible but requires more UI guidance to avoid mismatched pairs).
- Re-enable both dropdowns and remove the "coming soon" note once at least two options exist.
The include_overlaps parameter in fave_audio_textgrid excludes vowels that occur during
overlapping speech when set to False. It's not yet clear how new-fave determines what counts
as "overlapping" — specifically:
- Does it look at other tiers in the TextGrid? If so, which ones, and what label conventions does it expect?
- VoxHumana currently produces a single-speaker TextGrid from MFA (one Word tier, one Phone
tier). If new-fave's overlap detection requires a second speaker tier to be present, the
include_overlaps=Falseoption may have no effect for our use case. - If it does require a second tier: would it be worth adding an interviewer transcript tier to the TextGrid so that back-channels and interviewer overlaps are flagged? This would require either a separate transcription pass for the interviewer or manual annotation.
Check new-fave source (mark_overlaps in new_fave/utils/textgrid.py) to understand the
detection logic before advertising this option to users.
Power users can now skip any combination of pipeline steps and supply their own intermediate files instead of paying for the earlier steps to run again:
- Skip Transcription: upload a corrected/manual transcript as a TextGrid;
VxH routes it to
whisper_output/so MFA can pick it up from there. - Skip Alignment: upload an MFA-format TextGrid directly; VxH routes it to
mfa_output/so new-fave can use it without running MFA at all. - Skip Formant extraction: stop after alignment if you only need the TextGrid.
This covers the original motivating use case — a user runs the full pipeline once, notices Whisper made errors (e.g. OOV words, proper nouns, dialect forms) via the OOV words file or by inspecting the TextGrid, corrects the transcript, and resubmits from MFA onward without re-transcribing.
It's implemented end-to-end:
- The UI shows step toggles (Transcribe / Align / Extract formants) and a TextGrid upload zone that appears when Transcribe is unchecked.
- The backend (
_run_pipelineinweb/app.py) readsconfig["steps"]and runs only the requested steps, routes the uploaded TextGrid to the correct staging folder depending on which steps are skipped, and cleans up the now-unused staging folders afterward (e.g. removingwhisper_output/if the user supplied an MFA-ready TextGrid directly). - The processing log documents exactly which steps ran, which were skipped, and where the TextGrid came from (Whisper-generated vs. user-supplied).
- Orphaned-audio cleanup is Trolley-aware: it checks the in-memory
jobsdict rather than looking for specific output folders, so a Trolley job that legitimately has nowhisper_output/ornewfave_output/isn't mistaken for an orphaned/crashed job.
- User-facing help documentation explaining what the uploaded TextGrid must look like (tier names, format, common mistakes) — see the TextGrid upload help item below; this is the main remaining gap.
- Real-world testing with an actual corrected transcript containing a deliberate OOV word — see the OOV words file testing note below; this is now possible.
Two related features worth adding when there is demand:
MFA's OOV words are now extracted from its internal log and written to
mfa_output/oovs_found.txt, included in the download zip when present.
Testing note: Now that Trolley mode exists (see above), this can finally be
tested end-to-end: use the "skip Transcription" option to feed MFA a transcript
that contains a made-up or highly unusual word, and confirm it appears in
oovs_found.txt. Previously this was hard to test because Whisper tends to
recognize unfamiliar words as phonetically similar dictionary words, so true
OOVs rarely made it through to MFA in normal use.
Power users (e.g., researchers working with a specific dialect community) may want to
upload a custom pronunciation dictionary alongside their audio. MFA accepts a plain-text
dictionary file as the DICTIONARY_PATH argument instead of a model name.
To implement: add an optional file upload field in the Alignment section, validate that
it's a .txt or .dict file, and pass its path to MFA instead of the default model name.
Consider whether to allow this alongside or instead of the built-in dictionaries.
The upload zone, file-type filtering, and backend routing are all implemented (see
"Trolley" mode above — uploaded TextGrids are routed to whisper_output/ or
mfa_output/ depending on which steps are being skipped). What's still missing is
user-facing documentation explaining what the uploaded TextGrid must actually look
like. Before writing it, nail down the details:
- What tier name does MFA expect for an utterance-level TextGrid? (Currently
convert_whisper_to_textgridcreates anutterancestier — does MFA require that exact name, or does it accept any single-tier TextGrid?) - For the "skip Alignment" path, what tier structure does new-fave expect from a user-supplied MFA-format TextGrid (Word tier, Phone tier, naming conventions)?
- Does MFA/new-fave require non-empty intervals only, or do they handle empty/silence intervals gracefully?
- What happens if the TextGrid duration doesn't match the audio duration?
- Should the UI validate the uploaded file's internal structure beyond checking
the
.TextGridextension (e.g. parse it and check tier names before submitting)?
Then write a clear help drawer entry (a new help-textgrid.md, linked from the upload zone) that explains:
- The expected format for each scenario (Praat long TextGrid; a single utterance tier when skipping Transcription only, full Word/Phone tiers when also/instead skipping Alignment)
- How to prepare/export one from Praat
- Common mistakes (wrong tier name, mismatched duration, overlapping intervals)
Whisper, MFA, and the UI language selector are all now wired up for multiple languages.
Spanish (spanish_mfa) is the first non-English language supported: Transcription + Alignment
work end-to-end; Formant extraction is disabled for non-English for now (see below).
- Download MFA models on the server:
mfa model download acoustic <name>andmfa model download dictionary <name>. - Add to the two allowlists in
web/app.py:SUPPORTED_MFA_ACOUSTIC_MODELSandSUPPORTED_MFA_DICTIONARIES. - Add one
<option>to each of the three dropdowns inweb/static/index.html(Language, Acoustic model, Dictionary) and one entry to theLANG_TO_MFAJS map. - Test end-to-end on a real recording in that language.
French, Portuguese, German — to be added in a batch once each is tested individually. Longer-term: Italian, Dutch, Mandarin, Japanese, Korean, and others where both Whisper and MFA have solid models. See the language/MFA overlap table in session notes.
- The
taskparameter in Whisper: settingtask="translate"outputs an English transcript even for non-English audio. This could be a useful intermediate mode (transcribe → English → MFA with English models) before full multi-language MFA support is ready.
new-fave is NOT purely English-only — the underlying FastTrack analysis is language-independent.
The shipped fasttrack_config.yml already includes the full IPA vowel inventory in
target_labels, so it would detect and measure Spanish vowels (a, e, i, o, u) correctly.
The three parameters VxH passes to fave_audio_textgrid are English-specific:
| Parameter | Current value | Fix for non-English |
|---|---|---|
recode_rules |
"cmu2labov" |
"norecode" — identity pass-through; IPA labels preserved as-is |
labelset_parser |
"cmu_parser" |
None — skip CMU stress-digit parsing |
point_heuristic |
"fave" |
leave as "fave" — unknown labels fall back to 1/3 point, which is fine |
new-fave ships a built-in norecode scheme specifically for this purpose.
- Vowel labels in the CSV would be IPA symbols (e.g.
a,e,i,o,ufor Spanish) rather than Labov notation (ae,iy, etc.) - F1–F4 measurements would be acoustically valid
- The FAVE measurement-point heuristic defaults to
1/3through the vowel for any unrecognized label — reasonable but not language-tuned
The vowel_place.yml patterns drive the front/back optimization step. Some Spanish vowels
happen to match (e → front, i → front, o → back) but a and u don't match
anything. Whether this degrades the optimizer meaningfully needs a real-data test before
shipping.
- In
pipeline/extract_with_newfave.py(or the config layer), branch on the MFA model name: if non-English, passrecode_rules="norecode"andlabelset_parser=None. - Remove the formant lock-out from the UI (
syncLanguage()inindex.html) and the backend safety net (FORMANT_SUPPORTED_LANGUAGESinapp.py). - Test on a real Spanish recording; check that the CSV is populated and formant values are plausible.
- Decide whether to expose the
recode_ruleschoice to power users (probably not needed for most researchers).
The initial_prompt field is wired up and working. Possible future enhancements:
- Per-speaker prompts: if the recording has multiple speakers, allow separate hints per speaker (requires diarization, which is a larger feature).
- Saved prompts: let users save commonly used hints (e.g. a fieldwork community name and set of local vocabulary) and recall them from a dropdown.
- Auto-prompt from metadata: if the upload form eventually collects speaker/location metadata, pre-populate the hint field automatically.
The email field and notification section have been removed from the UI for now. The full intended workflow, when built:
- Close-tab mode — user submits, sees a confirmation that processing will continue server-side, and can safely close the browser. When the job finishes, VxH emails the results as a zip attachment (or a download link if the file is too large for email).
- Keep-open mode — current behaviour: user keeps the tab open, watches the progress pipeline track, and downloads results when done.
- Restore the Notification section (step 5) with email as a required field
- Add a radio/toggle to choose between the two modes
- In close-tab mode: after submit, show a "You can safely close this tab" screen instead of the progress view
- In keep-open mode: current progress view, unchanged
- On the progress note, mention both options
- Wire up an email library (e.g.
smtplibwith BYU SMTP, or SendGrid) - Store email in the job record for logging; do not persist it after the email is sent
- After job completion in close-tab mode: zip results and send/link via email
- Queue system ✓ (implemented) — the single-threaded executor already serialises
jobs;
GET /api/jobs/{job_id}now reportsqueue_position/queue_lengthand astep_namelike "Waiting in queue (position 2 of 3)" while a job is queued behind another. Tracked via a module-levelactive_jobslist inweb/app.py(FIFO, matches the executor's own processing order). Tested locally by submitting three jobs concurrently and confirming positions were reported and decremented correctly as earlier jobs finished. - Queue wait reporting ✓ (implemented) — every job now records
queue_position_at_submission(1 = started immediately) andwait_seconds(time between submission and actually starting) in the jobs dict. This shows up in three places: the user-facingprocessing_log.txt("Queue: started at position 2 in line — waited 0m 8s..."), the internal per-job server log indata/logs/YYYY-MM/<job_id>.txt("Queue pos.: 2 (wait: 0m 8s)"), and asqueue_position_at_submission/wait_secondsfields indata/logs/summary.jsonlfor aggregate analysis of typical wait times once real usage data accumulates.
"Your email is used to deliver your results and is recorded in our job log alongside your Job ID. It is not shared or used for any other purpose."
Currently, the processing continues after closing the tab. If the email thing doesn't happen soon, I should fix that so that canceled jobs don't clog the queue.
Discovered: while stress-testing the queue locally by submitting several large real files back-to-back, the laptop went to sleep partway through and killed the dev server. That's a separate problem (see the "computer sleep kills the local dev server" note in Alpha/Beta testing below) — but investigating it surfaced a real gap that matters independent of the email/close-tab feature above: if the browser tab is closed, reloaded, or crashes while a job is running, there is currently no way to get back to it.
currentJobId and currentDownloadToken (web/static/index.html — the two
module-level lets near the top of the polling script) exist only in that tab's
JS memory. Nothing is written to localStorage or anywhere else. So even though:
- job results sit server-side for
JOB_RETENTION_HOURS(72h, see_expire_old_jobsinweb/app.py), and - server-side processing is unaffected by the client disconnecting
(
_run_pipelineruns in the background executor regardless of the tab),
...losing that JS state means losing the only record of the download token, which
is never displayed to the user as text — only the job ID is shown on screen
(running-job-id / done-job-id elements). Without the token, GET /api/jobs/{job_id}/download returns 403.
This is exactly the scenario the user is likely to hit in real use: submitting a large file, walking away, and coming back later on the same machine (not necessarily wanting full email delivery, just wanting the tab to still work).
1. localStorage persistence (small, no backend changes)
- On successful submit, write
{job_id, download_token, filename, submitted_at}tolocalStorage(key e.g.vxh_jobs, structured as an array — see "handle multiple jobs per browser" below, since one user submitting several files in a row is the actual use case that surfaced this). - On page load, before showing the empty upload form: check
localStoragefor any job(s) not already known to be finished/downloaded. For each, callGET /api/jobs/{job_id}:- 404 (job expired past 72h, or server restarted and lost its in-memory
jobsdict — see the orphaned-job scenario from the sleep incident) → drop it fromlocalStorage, nothing to recover. - still queued/running → restore
currentJobId/currentDownloadTokenand resume the progress view / polling exactly where it would have been. - done/error → show the done/error screen directly, skip polling.
- 404 (job expired past 72h, or server restarted and lost its in-memory
- Clear a job's
localStorageentry once the user downloads results or dismisses an error, so the list doesn't grow unbounded.
2. Handle multiple jobs per browser (natural extension, not a separate feature)
Since the motivating case is someone submitting several files in sequence (the
whole point of the queue system above), a single currentJobId doesn't fit —
localStorage should hold a small list. Add a lightweight "your recent jobs"
view (job ID, filename, status, queue position if waiting, download link if
done) instead of just the current single-job progress screen. This also gives
users visibility into concurrent submissions without needing the email feature.
3. Cross-machine / cross-browser recovery (optional, security tradeoff)
localStorage only helps if it's the same browser profile on the same machine.
For "I want to check a job from my phone" or "I cleared my browser data,"
recovery requires a job-ID-only lookup form. Whether that should expose only
status (queued/running/done/error — safe, no data leak) or also the download
link is an open question tied to the existing "Job ID guessability" item under
Security audit below (job IDs are only ~1,190 combinations/day and are not
secret). Recommendation: status-only for a bare job ID; still require the
token for downloads. This keeps the download token as the actual bearer
credential and avoids weakening the existing security posture.
This is a much smaller, backend-change-free version of the same underlying problem ("I walked away, how do I get back to my results?"). It's worth doing on its own regardless of whether/when SMTP email delivery gets built — once email exists, this becomes a nice-to-have fallback rather than the primary recovery path.
Currently shows: "Processing time scales with recording length — a 1-hour interview may take up to an hour." Replace with a real estimate when enough data exists.
- Record two values for every completed job: audio duration (seconds) and total wall-clock processing time (seconds). Store these alongside the job log.
- Once ~20–30 jobs have completed, fit a simple linear regression: estimated_time = a × audio_duration + b (or separate models per Whisper model size, since turbo ≠ large-v3 speed)
- Show the estimate on the progress screen: "Estimated time remaining: ~12 min" Update it as steps complete and actual step times are known.
- Ask Claude to help build/tune the model once the data exists.
- Whisper is the dominant cost (~1× real-time without GPU, ~0.1× with GPU)
- MFA is fast (<1 min for most interviews)
- new-fave is fast (<1 min)
- Estimate should probably be shown before submit (on the form) so users can decide whether to use close-tab mode or keep the tab open
coupling the manuals, laying out the console, registering the swell box, drawing console, warming up pipes, tuning the reeds, lacing up organ shoes, Adjusting the wind pressure, Opening the expression box, voicing the flue pipes, Warming up the pipes...
The web UI already shows step-by-step progress and surfaces error messages with a job ID. Before building a separate debug tool, check whether the UI + error.log is sufficient for diagnosing user-reported failures.
If a dedicated tool turns out to be useful, it could accept a job ID or a path to an audio file and re-run individual steps with verbose output — useful for replaying a failed job locally without going through the web interface. Options to consider:
python debug.py --job <job_id>— re-run pipeline on an existing job directorypython debug.py --audio <file> --step whisper— run just one step with full logging- Flags for overriding config (model size, language, MFA env, etc.)
- Print full Whisper output, MFA stdout/stderr, new-fave warnings
Ask: does the existing tests/test_pipeline.py + the web error screen already cover
the debugging workflow well enough? If users can report a job ID and you can find
the error.log, a separate CLI debug tool may not be worth the maintenance cost.
Potentially make it so that the log files themselves can be read in by the debugging tool so that processing is completely replicable.
Add a "User Guide" tab to the UI (alongside the main upload form). Content to include:
Make clear that VoxHumana respects the sensitivity of sociolinguistic recordings:
- All processing happens entirely on BYU's server — your audio is never sent to OpenAI or any other external service. Whisper, MFA, and new-fave all run locally.
- Uploaded audio is deleted from the server as soon as processing finishes.
- Result files are available for download for 72 hours, then deleted.
- No audio or transcripts are retained, shared, or used for any other purpose.
Preferred: split the recording into segments.
- Recommended tool: Audacity (free) — File > Export > Export Multiple, split by time
- Command-line option:
ffmpeg -i interview.wav -f segment -segment_time 1800 -c copy part%03d.wav(splits into 30-minute chunks; adjust segment_time as needed) - Run each segment through VxH separately, then combine the output CSV files.
Also works: compress to a smaller format before uploading.
- MP3 (320 kbps): good quality, ~1/5 the size of WAV
- FLAC: lossless compression, ~1/2 the size of WAV
ffmpeg -i interview.wav -b:a 320k interview.mp3- VxH accepts .wav, .mp3, .flac, and other common formats.
Processing time scales with recording length. Without a GPU, Whisper transcription alone takes roughly 1× real-time (a 60-minute interview takes ~60 minutes).
Options:
- Split the recording (see above) and run segments in parallel on separate machines.
- Use a smaller Whisper model (e.g., "small" or "base") — faster but less accurate.
- Run VxH locally via the command line (
python main.py), which avoids upload time and lets you run on your own hardware with a GPU. - Contact the lab for access to a GPU-equipped server if you have many recordings.
VoxHumana handles sensitive sociolinguistic data — identifiable voices and personal conversations from research participants. A dedicated security review should be done before the tool is opened to broad public use. Key areas to audit:
- Data in transit: confirm all traffic runs over HTTPS (no HTTP fallback). Audio uploads and result downloads should never travel unencrypted.
- Job directory access: verify that
data/jobs/cannot be accessed directly via URL — only through the API endpoints. Check this holds after any nginx/proxy config changes. (Currently confirmed safe — onlyweb/static/is mounted as static.) - Audio deletion: confirm the audio file is always deleted after processing, including on pipeline failure. Orphan cleanup (end-of-job sweep) is now implemented but should be verified under crash conditions.
- On-server processing: all three tools (Whisper, MFA, new-fave) run entirely locally — audio never leaves the server. This should be stated explicitly in the User Guide and privacy notice.
- Job ID guessability: job IDs are now YYMMDD_Stop1_Stop2 (~1,190 combinations per day). A determined person could enumerate today's IDs. Consider whether result downloads need any additional authentication (e.g. a one-time token) if the tool is used for sensitive studies.
- Upload validation: confirm that only audio files can be uploaded (check MIME type and extension), and that the 1 GB size limit is enforced server-side.
- Large intermediates deleted after every job: uploaded audio,
mfa_corpus/,mfa_temp/. This recovers 1–3 GB per job immediately. - Orphaned audio cleanup: at the end of each completed job, a sweep deletes
audio files left behind by jobs the running server doesn't know about (e.g. ones
that were mid-run when the server last crashed/restarted). The check is based on
in-memory job-dict membership rather than which output folders exist, so it
correctly leaves Trolley jobs alone (a Trolley job can legitimately lack
whisper_output/ornewfave_output/and still be a normal, healthy job). - No world-readable job directories: confirmed — only
web/static/is served statically;data/jobs/is reachable only through the API endpoints. - Logs live separately: job logs (
data/logs/) are written outsidedata/jobs/and are never touched by the result-cleanup sweep — they remain the permanent audit trail even after a job's files have expired. - Result files expire: job directories are deleted 72 hours after completion
by
_expire_old_jobs(), called at the end of every job. The retention window is controlled by theJOB_RETENTION_HOURSconstant inweb/app.py.
Every job (success or failure) now writes a server-side log to:
data/logs/YYYY-MM/<job_id>.txt
Each file contains: job ID, filename, submitted/completed timestamps, total duration, per-step timings, all settings used, tool versions, final status, and full error traceback on failure. Logs are stored outside job directories and are never deleted by the result cleanup sweep.
Job IDs are now in the format YYMMDD_Stop1_Stop2, e.g. 260601_Bourdon_Flute. Two distinct organ stops are drawn at random; a collision check retries if the same pair was already used today (extremely rare at expected job volumes).
Organ stops pool (35), drawn from the Salt Lake Tabernacle organ: Bombarde, Bourdon, Celeste, Clarinet, Clarion, CorAnglais, Cornopean, Cymbelstern, Diaphone, Diapason, Doppelflote, Dulciana, Flugelhorn, Flute, FrenchHorn, Fugara, Gamba, Gemshorn, Harp, LieblichBourdon, Mixture, Nachthorn, Nazard, Oboe, Octave, Piccolo, Principal, Rauschquinte, Trombone, Trompette, Tremulant, Trumpet, Tuba, Tutti, Viole
35 × 34 = 1,190 combinations per day — sufficient for expected usage.
The job ID is used as the job directory name and shown to the user on both the error screen and the success/download screen so they can reference it when contacting the lab.
Alongside the human-readable per-job logs, every job also appends a structured
JSON record to data/logs/summary.jsonl — one line per job — covering things
like Whisper model/language used, step durations, audio duration, final status,
and error type on failure. This is meant to make it possible, after a few months
of real usage, to answer questions like which features people actually use, how
long jobs typically take, and where crashes tend to occur — without having to
parse the prose logs by hand.
- Valerie Freeman
- Peggy Renwick
- Dan Villarreal
- Computer sleep kills the local dev server. While stress-testing the queue
by submitting several large real files back-to-back locally, the laptop went
to sleep partway through and the
uvicornprocess died outright — not suspended/resumed, just gone (no error.log, no crash trace, nothing left running afterward). Everything queued behind the in-flight job at that point was silently orphaned (audio left on disk, never processed — see_cleanup_orphaned_audio, which only cleans up on the next server start, not proactively). Not a VoxHumana bug — a local-only concern from running the server and doing the submitting on the same machine. Once deployed on the real (always-on) server this won't apply, since the client sleeping doesn't touch the server process. For further local multi-job testing in the meantime: keep the machine awake (e.g.caffeinateon macOS) for the duration of the batch. This also motivated the "Client-side job persistence & status recovery" item above.
Batch file upload Add OOV words to MFA.