fix(providers): increase HTTP connection pool for embedding provider

[ronaldtangg]

Bug Report

App Version

v3.11.3

Description

When multiple concurrent requests (>5) trigger team_tasks(action="search"), all requests stuck indefinitely at the tool call. The system works normally with <5 concurrent requests but completely stalls under higher load.

Error Log

No explicit error log. Requests hang indefinitely at team_tasks tool call.
Trace shows: tool_call team_tasks - status: running

Steps to Reproduce

1. Deploy goclaw with embedding provider configured (OpenAI)
2. Send 10+ concurrent chat requests that trigger team_tasks(action="search")
3. Observe that requests 6+ hang indefinitely while requests 1-5 may complete

Expected Behavior

All concurrent requests should complete within reasonable time (~1-5 seconds for search).

Actual Behavior

• Requests 1-2: Complete normally
• Requests 3-5: Slow but complete
• Requests 6+: Stuck indefinitely, never complete

Root Cause

The OpenAIEmbeddingProvider in internal/providers/embedding_openai.go uses a default http.Client without custom transport configuration:

client: &http.Client{Timeout: 60 * time.Second}

Go's default http.Transport has MaxIdleConnsPerHost: 2, causing HTTP connection pool starvation when multiple requests need to call the embedding API concurrently. Reference: https://github.com/golang/go/blob/master/src/net/http/transport.go#L62

Fix

Configure custom http.Transport with higher connection limits:

  client: &http.Client{
      Timeout: 60 * time.Second,
      Transport: &http.Transport{
          MaxIdleConns:        100,                                                                                                                                                                                                                                                                                                                                                                                              
          MaxIdleConnsPerHost: 30,
          IdleConnTimeout:     90 * time.Second,                                                                                                                                                                                                                                                                                                                                                                                 
      },                                                              
  },                                                                       

Additional Context

• Affects any deployment using embedding provider for team task search
• Connection pool exhaustion causes cascading failures as requests queue up
• Each blocked request holds resources for up to 60 seconds (client timeout)

PR: #1123

[mrgoonie]

Summary: I re-checked this against current dev and the report is valid, with one implementation note for the linked PR.

Decision: confirmed / accepted; linked PR #1123 is the right fix path but needs a small alignment before merge.

Evidence:

• The screenshot shows 11 concurrent Response Orchestrator rows still in loading/running state, which supports the observed concurrent-search stall symptom, though the screenshot alone cannot prove the exact network root cause.
• The issue is specifically about team task semantic search using embeddings under concurrent load, and current dev still constructs the OpenAI-compatible embedding client in internal/providers/embedding_openai.go as &http.Client{Timeout: 60 * time.Second} with no custom transport.
• Go's default transport keeps MaxIdleConnsPerHost = 2, so this path is under-provisioned for bursty concurrent embedding calls compared with the repo's provider default transport in internal/providers/defaults.go, which already centralizes keep-alive and stage-timeout settings.
• Duplicate search only found linked PR fix(providers): increase HTTP connection pool for embedding provider #1123 and this issue for the exact embedding-provider pool starvation report.

Next step: proceed through #1123, but prefer using the repo-standard NewDefaultTransport()/shared transport defaults or a named embedding transport helper rather than adding another bespoke transport literal. Add a focused regression/unit check that the embedding provider client has increased per-host idle capacity so this does not regress.