Skip to content

fix trace crashes from concurrent access#2616

Open
gauravyad86 wants to merge 1 commit intoconfident-ai:mainfrom
gauravyad86:fix/tracing-thread-safety-locks
Open

fix trace crashes from concurrent access#2616
gauravyad86 wants to merge 1 commit intoconfident-ai:mainfrom
gauravyad86:fix/tracing-thread-safety-locks

Conversation

@gauravyad86
Copy link
Copy Markdown

@gauravyad86 gauravyad86 commented Apr 16, 2026

What was the issue?

When many evaluations were running at the same time, DeepEval's tracing system was reading and updating shared in-memory data at once without proper protection.

In simple terms: two parts of the code could try to edit the same tracking data at the same moment, which could cause random crashes or incomplete trace results.

What was fixed?

A safety lock was added around shared tracing data so only one operation can change it at a time.

A few places that were directly reading shared data were updated to use safer helper methods.

One remaining method that was still reading trace data without lock protection was also fixed.

Result

Tracing is now much safer under concurrent workloads and less likely to fail with race-condition style errors.

Changes

  • Added threading.RLock (_state_lock) inside TraceManager.
  • Added lock protection for shared-state methods touching:
    • active_traces
    • active_spans
    • traces
  • Updated Observer call paths to avoid direct unsafe shared-state access:
    • replaced direct trace membership checks with is_trace_active(...)
    • replaced direct active span iteration with get_active_spans_for_trace(...)
  • Added thread-safe snapshot handling in get_all_traces_dict() before iterating trace data.

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 16, 2026

@gauravyad86 is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

@gauravyad86 gauravyad86 changed the title Fix trace crashes from concurrent access fix trace crashes from concurrent access Apr 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant