CodeContext Sparse Vector Index Issue

[sorin-sds]

GitHub Issue: CodeContext Sparse Vector Index Issue

Issue Title: CodeContext fails to create sparse_vector index in Milvus, causing semantic search to fail

Labels: bug, needs-investigation, milvus

---

Environment Configuration

• Development Machine: Windows 11 (Goose AI with Qwen3-coder-next)
• Milvus Server: Linux (Ubuntu 22.04) - IP: 192.168.1.100
• Ollama Server: Linux (Ubuntu 22.04) - IP: 192.168.1.101
• Milvus Version: 2.6.12 (pymilvus 2.6.12)
• CodeContext MCP Server: Latest version (zilliztech/claude-context-mcp)

Summary

CodeContext/Claude Context MCP server fails to create required vector indexes for sparse_vector fields in Milvus, causing semantic search to fail with error:

"ErrorCode: UnexpectedError. Reason: there is no vector index on field: [sparse_vector], please create index firstly"

Expected Behavior

The CodeContext MCP server should:

1. Automatically create the sparse_vector index when indexing a codebase
2. Preserve indexes when re-indexing (no collection recreation that orphan indexes)
3. Enable semantic search without manual intervention

Actual Behavior

1. Indexing completes (files get chunked and embeddings generated)
2. Collection created but sparse_vector field has no index
3. Search fails with "no vector index" error
4. Collection hash changes between indexing attempts (hybrid_code_chunks_865434e6 → hybrid_code_chunks_f817ae2e)
5. Manual intervention required to create sparse_vector index

Reproduction Steps

1. Install CodeContext MCP server
2. Configure with Milvus server at 192.168.1.100:19530
3. Index a codebase with custom extensions (e.g., .pas, .dfm)
4. Attempt semantic search using Codecontext.searchCode()
5. Observe error: "no vector index on field: [sparse_vector]"

Error Messages

❌ Codebase 'D:\medfamserv' indexing failed.
🚨 Error: Failed to load collection 'hybrid_code_chunks_f817ae2e' after 5 attempts: 
Error: ErrorCode: UnexpectedError. Reason: there is no vector index on field: [sparse_vector], 
please create index firstly
📊 Failed at: 0.0% progress

Workaround

Manually create sparse_vector index using pymilvus:

from pymilvus import MilvusClient
from pymilvus.milvus_client.index import IndexParams, IndexParam

client = MilvusClient(uri="192.168.1.100:19530")
collection_name = "hybrid_code_chunks_f817ae2e"

# Create IndexParams object
index_params = IndexParams()

# Add sparse_vector index with correct type
index_params.add_index(
    field_name="sparse_vector",
    index_type="SPARSE_WAND",
    index_name="sparse_wand_index",
    metric_type="IP",
    params={"index_level": 1, "reorder_k": 100}
)

# Create index in Milvus
client.create_index(
    collection_name=collection_name,
    index_params=index_params
)

Technical Details

Collection Structure

Fields:
- id (VARCHAR, max_length=512, primary)
- content (VARCHAR, max_length=65535, analyzer enabled)
- vector (FLOAT_VECTOR, dim=768) ← Dense embedding
- sparse_vector (SPARSE_FLOAT_VECTOR) ← BM25 sparse vector
- relativePath (VARCHAR, max_length=1024)
- startLine (INT64)
- endLine (INT64)
- fileExtension (VARCHAR, max_length=32)
- metadata (VARCHAR, max_length=65535)

Required Indexes (Auto-Created vs Missing)

vector_index: HNSW on 'vector' field ✅
sparse_wand_index: SPARSE_WAND on 'sparse_vector' field ❌ (missing)

Investigation Results

Root Cause Analysis

1. Missing default index creation - MCP server doesn't create sparse_vector indexes automatically
2. Collection recreation - Hash changes between indexing attempts orphan old indexes
3. Index persistence - Indexes not transferred to new collections

API Requirements Discovered

• pymilvus 2.6.12 IndexParams is a list subclass, not dict-like
• Requires add_index() method for adding index configurations
• Sparse vector search requires SPARSE_WAND (not SPARSE_INVERTED)

Environment Architecture

┌─────────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  Windows Machine    │     │ Linux Server     │     │ Linux Server     │
│  (Goose + Qwen)     │────>│ (Milvus)         │────>│ (Ollama)         │
│  192.168.1.102      │     │ 192.168.1.100    │     │ 192.168.1.101    │
└─────────────────────┘     └──────────────────┘     └──────────────────┘

Files Created for Investigation

Diagnostic Scripts

1. fix_sparse_index_v2.py - Creates sparse_vector index for any CodeContext collection
2. diagnose_indexparams.py - Analyzes IndexParams class hierarchy
3. diagnose_indexparam.py - Analyzes IndexParam constructor
4. test_sparse_wand.py - Tests different index types
5. test_direct_search.py - Direct Milvus search testing

Documentation

1. CODECONTEXT_DIAGNOSTIC_REPORT.md - Full technical report
2. INDEX_FIX_SUMMARY.md - Quick reference for the fix

Suggestions for Fix

1. Add automatic index creation - MCP server should create indexes after collection creation
2. Fix collection naming - Use consistent collection names or migration strategy
3. Add config option - Allow users to specify index parameters for different vector fields
4. Add health check - Verify indexes exist before search attempts

Workaround Script (For Deployment)

#!/usr/bin/env python3
# auto_fix_codecontext_index.py
from pymilvus import MilvusClient
from pymilvus.milvus_client.index import IndexParams, IndexParam

client = MilvusClient(uri="192.168.1.100:19530")
collections = client.list_collections()
codecontext_collections = [c for c in collections if 'hybrid_code_chunks' in c.lower()]

for collection_name in codecontext_collections:
    try:
        collection_info = client.describe_collection(collection_name)
        field_names = [f['name'] for f in collection_info['fields']]
        
        if 'sparse_vector' not in field_names:
            continue
        
        current_indexes = client.list_indexes(collection_name)
        if any('sparse' in idx.lower() for idx in current_indexes):
            continue
        
        index_params = IndexParams()
        index_params.add_index(
            field_name="sparse_vector",
            index_type="SPARSE_WAND",
            index_name="sparse_wand_index",
            metric_type="IP",
            params={"index_level": 1, "reorder_k": 100}
        )
        
        client.create_index(collection_name=collection_name, index_params=index_params)
        print(f"✓ Index created for {collection_name}")
    except Exception as e:
        print(f"✗ Error for {collection_name}: {e}")

Questions for Maintainers

1. Is automatic sparse_vector index creation expected behavior?
2. Are there configuration options for customizing vector indexes?
3. Why does collection hash change between indexing attempts?
4. Is this a known issue with specific versions of pymilvus/Milvus?
5. Are there recommended versions for compatibility?

---

Report generated: May 2026
Maintainer: CodeContext tester
AI Tools Used: Goose & Qwen3-coder-next
Report Version: 1.0

[Showiix]

Thanks for the detailed report. I looked at the current Milvus implementation and it does attempt to create the sparse_vector index with SPARSE_INVERTED_INDEX + BM25.

One possible issue is that the Node SDK createIndex() call returns a ResStatus, and the current code does not appear to validate that status before continuing. If sparse index creation returns a non-success status instead of throwing, the collection can exist without a usable sparse_vector index, and the later load/search error would match what you are seeing.

Could you share the MCP/Milvus logs around the sparse index creation step? In particular, anything after Creating sparse vector index for field 'sparse_vector'... would help confirm whether the index creation request is failing silently or whether this is an index-parameter compatibility issue.

[PaddockEngineering]

Second independent reproduction, with a root-cause angle that may help: a dense-only embedder is a reliable trigger, and HYBRID_MODE=false is a clean user-side fix.

Environment

• Milvus 2.4.17 standalone (single-container; 2.6.x panics in standalone, so pinned)
• Embedder: Ollama nomic-embed-text — dense-only (cannot emit sparse vectors)
• @zilliz/claude-context-mcp tested on 0.1.7 and 0.1.14 — both reproduce identically

What happens
HYBRID_MODE defaults to true, so the collection is created with a sparse_vector field. With a dense-only embedder that field is never populated and never indexed. indexes/list on the resulting collection shows only ["vector_index"] (the dense field); sparse_vector has no index, so LoadCollection fails on a loop:

there is no vector index on field: [sparse_vector], please create index firstly

(~50 looped failures observed per index attempt.) Upgrading 0.1.7 → 0.1.14 did not change this — reproduced identically.

Fix that works here: set HYBRID_MODE=false. The sparse field is no longer declared, and indexing + search_code then work end-to-end on the same stack.

Note vs PR #364: that PR auto-falls-back to dense only on the field data type: 104 is not supported rejection (Milvus builds lacking SparseFloatVector). This path is different — Milvus 2.4.17 accepts the hybrid schema (no type-104 error), so #364's fallback never triggers, yet the sparse field is still left unindexed because the embedder is dense-only. Suggestion: when HYBRID_MODE is the implicit default, also fall back to a dense collection if the configured embedder doesn't emit sparse vectors (or validate the sparse_vector index status after creation, per @Showiix's createIndex ResStatus point above) — not only on the type-104 case.