[Bug] KnowledgePostProcess spawns orphaned downstream tasks for failed knowledge, causing massive queue backlog

[liangliwu]

Describe the bug

When a knowledge entry's parse_status is failed (e.g., due to application restart during parsing), the KnowledgePostProcessService.Handle handler still processes it and spawns downstream tasks (question:generation, summary:generation, wiki:ingest, graph-extract). These downstream tasks operate on incomplete or missing data, producing invalid results and clogging the task queues.

This contradicts the design intent documented in router/task.go:265-271:

// Question/Summary generation are NOT included: they run after parse_status
// has already become "completed" and have their own status fields.

To reproduce

1. Upload a large batch of documents (e.g., 8,000+ files) to a KB
2. Restart the WeKnora application while parsing is in progress
3. Documents will be marked parse_status=failed with error Task interrupted due to application restart
4. Observe asynq queues — a massive number of question:generation and image:multimodal tasks are spawned for the failed documents

Observed data (from a production instance):

KB 0e443c1a: 8,474 knowledge entries
  - 8,436 parse_status=failed (error: "Task interrupted due to application restart")
  -    30 parse_status=completed
  -     8 parse_status=finalizing

Queue backlog caused by failed knowledge:
  - question queue:  34,252 pending tasks (spawned for failed knowledge)
  - multimodal queue:  4,859 pending tasks (spawned for failed knowledge)
  - Total orphaned downstream tasks: ~39,000

These tasks clogged the asynq queues, causing user-facing operations (e.g., knowledge deletion) to be starved — deletion tasks in the low queue waited 30+ minutes without being processed.

Expected behavior

KnowledgePostProcessService.Handle should skip knowledge entries with parse_status=failed and not spawn downstream tasks. The status check at the handler entry should include failed in the skip list, alongside the existing cancelled and deleting checks.

Root cause

File: internal/application/service/knowledge_post_process.go:111-120

switch knowledge.ParseStatus {
case types.ParseStatusCancelled, types.ParseStatusDeleting:
    logger.Infof(ctx,
        "[KnowledgePostProcess] Knowledge %s aborted (%s), skipping post-processing.",
        payload.KnowledgeID, knowledge.ParseStatus,
    )
    s.tracker().SkipSpan(ctx, postSpan,
        "knowledge "+knowledge.ParseStatus+" before postprocess started")
    return nil
}
// ❌ parse_status=failed falls through to downstream task spawning logic at :159-308

The downstream spawning logic (:159-308):

willSpawnSummary := len(textChunks) > 0
willSpawnQuestion := willSpawnSummary && kb.NeedsEmbeddingModel() &&
    eff.QuestionGenerationConfig.Enabled
willSpawnWiki := kb.IndexingStrategy.WikiEnabled && len(textChunks) > 0

if willSpawnSummary {
    s.enqueueSummaryGenerationTask(ctx, payload, attempt)
    if willSpawnQuestion {
        s.enqueueQuestionGenerationTasks(...)  // Spawns N tasks per chunk batch
    }
}
if willSpawnWiki {
    EnqueueWikiIngest(...)
}

Since failed is not in the skip list, a failed knowledge entry with non-empty textChunks will spawn all downstream tasks — even though the knowledge itself failed to parse correctly.

Suggested fix

Add types.ParseStatusFailed to the switch case in knowledge_post_process.go:112:

switch knowledge.ParseStatus {
case types.ParseStatusCancelled, types.ParseStatusDeleting, types.ParseStatusFailed:
    logger.Infof(ctx,
        "[KnowledgePostProcess] Knowledge %s aborted (%s), skipping post-processing.",
        payload.KnowledgeID, knowledge.ParseStatus,
    )
    s.tracker().SkipSpan(ctx, postSpan,
        "knowledge "+knowledge.ParseStatus+" before postprocess started")
    return nil
}

This is a one-line fix that prevents orphaned downstream tasks from being spawned for failed knowledge entries.

Impact

In our production instance, this bug caused ~39,000 orphaned tasks to accumulate in asynq queues, which:

• Starved user-facing operations (deletion tasks waited 30+ minutes)
• Wasted LLM API calls on invalid question/multimodal generation
• Filled the task_dead_letters table (10,536 dead-letter entries for wiki:ingest alone)
• Required manual Redis cleanup to restore service

Environment

• WeKnora version: 0.6.3
• Deployment: Docker Compose
• KB with 8,474 knowledge entries, 8,436 failed due to app restart
• asynq concurrency: 48