Skip to content

feat(sqlitevec): add sqlite-vec vector store components#835

Open
Lonliet wants to merge 1 commit into
cloudwego:mainfrom
Lonliet:feat/sqlitevec-vector-store
Open

feat(sqlitevec): add sqlite-vec vector store components#835
Lonliet wants to merge 1 commit into
cloudwego:mainfrom
Lonliet:feat/sqlitevec-vector-store

Conversation

@Lonliet

@Lonliet Lonliet commented May 19, 2026

Copy link
Copy Markdown

What

Add pure-Go sqlite-vec Vector Store support for Eino:

  • components/indexer/sqlitevec
  • components/retriever/sqlitevec

Why

This provides an embedded vector store for local RAG, examples, tests, and small-scale applications.
It uses modernc.org/sqlite and modernc.org/sqlite/vec, so it does not require CGO or an external vector database service.

Scope

  • Store Eino documents and dense vectors into SQLite/sqlite-vec.
  • Retrieve TopK similar documents by query embedding.
  • Preserve document ID, content, metadata, Eino score, and raw sqlite-vec distance.
  • Include unit tests using in-memory SQLite.

Notes

  • modernc.org/sqlite/vec currently requires modernc.org/sqlite versions that require Go 1.25 or newer; the new component modules currently use Go 1.25.10.
  • sqlite-vec KNN queries require MATCH ? AND k = ?; parameterized LIMIT ? was not accepted by vec0 during testing.
  • Raw SQL filters are intentionally not supported in this first version.
  • I am willing to maintain these sqlitevec components and follow up on review feedback and future compatibility updates.
  • The contributing guide mentions develop, but the upstream repository currently has main as its default branch and no develop branch.

Tests

Docker Linux, golang:1.25-bookworm:

  • go test ./... under components/indexer/sqlitevec
  • go test -race ./... under components/indexer/sqlitevec
  • go vet ./... under components/indexer/sqlitevec
  • go test ./... under components/retriever/sqlitevec
  • go test -race ./... under components/retriever/sqlitevec
  • go vet ./... under components/retriever/sqlitevec
  • golangci-lint run under components/indexer/sqlitevec using golangci-lint v1.64.8
  • golangci-lint run under components/retriever/sqlitevec using golangci-lint v1.64.8

Additional compatibility check:

  • golang:1.22-bookworm with GOTOOLCHAIN=auto selects Go 1.25.10 for the sqlitevec modules and passes go test ./... for both modules.

Local Windows:

  • go test ./... passed for both modules.
  • go vet ./... passed for both modules.
  • go test -race ./... failed with Windows exit status 0xc0000139; Docker Linux race tests passed.

Pre-submit Checks

  • git diff --check origin/main..HEAD passed.
  • GitHub search found no existing cloudwego/eino-ext issues/PRs matching sqlitevec, sqlite-vec, or "sqlite vec".

Copilot AI review requested due to automatic review settings May 19, 2026 18:50
@CLAassistant

CLAassistant commented May 19, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a pure-Go embedded SQLite + sqlite-vec vector store implementation for Eino by introducing a paired SQLiteVec Indexer (store docs + embeddings) and SQLiteVec Retriever (TopK similarity search), along with docs, examples, and unit tests. This expands eino-ext with a local/embedded RAG option without CGO or external vector DB services.

Changes:

  • Introduce components/indexer/sqlitevec to upsert documents + vectors into SQLite/vec0 (optionally auto-creating schema).
  • Introduce components/retriever/sqlitevec to query vec0 KNN results and return Eino Document objects with score + raw distance metadata.
  • Add module scaffolding (go.mod/go.sum with Go 1.25), README/README_zh, examples, and in-memory unit tests for both modules.

Reviewed changes

Copilot reviewed 17 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
components/retriever/sqlitevec/retriever.go Implements SQLiteVec retriever (embed query, vec0 KNN query, score + metadata handling, callbacks).
components/retriever/sqlitevec/retriever_test.go In-memory SQLite tests covering validation, retrieval, options, and corrupt metadata.
components/retriever/sqlitevec/README.md English documentation for retriever usage/config/options.
components/retriever/sqlitevec/README_zh.md Chinese documentation for retriever usage/config/options.
components/retriever/sqlitevec/options.go Adds retriever impl-specific option WithMaxDistance.
components/retriever/sqlitevec/consts.go Defines component constants/defaults + metadata key name.
components/retriever/sqlitevec/codec.go Table-name validation + vector/metadata JSON helpers for retriever.
components/retriever/sqlitevec/examples/retriever/main.go Minimal runnable retriever example with in-memory DB seeding.
components/retriever/sqlitevec/go.mod New retriever module (Go 1.25) + dependencies.
components/retriever/sqlitevec/go.sum Dependency checksums for retriever module.
components/indexer/sqlitevec/indexer.go Implements SQLiteVec indexer (batch embedding, upsert docs, write vec0 rows, callbacks, optional auto-create).
components/indexer/sqlitevec/indexer_test.go In-memory SQLite tests covering validation, store/upsert behavior, and embedding error paths.
components/indexer/sqlitevec/README.md English documentation for indexer usage/config/schema/notes.
components/indexer/sqlitevec/README_zh.md Chinese documentation for indexer usage/config/schema/notes.
components/indexer/sqlitevec/consts.go Defines component constants/defaults.
components/indexer/sqlitevec/codec.go Table-name validation + metadata/vector JSON helpers for indexer.
components/indexer/sqlitevec/examples/indexer/main.go Minimal runnable indexer example.
components/indexer/sqlitevec/go.mod New indexer module (Go 1.25) + dependencies.
components/indexer/sqlitevec/go.sum Dependency checksums for indexer module.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread components/retriever/sqlitevec/retriever.go Outdated
Comment thread components/retriever/sqlitevec/README.md
Comment thread components/retriever/sqlitevec/README_zh.md
Comment thread components/indexer/sqlitevec/indexer.go Outdated
@Lonliet Lonliet force-pushed the feat/sqlitevec-vector-store branch from 5927961 to 601102f Compare May 20, 2026 02:59
@Lonliet Lonliet requested a review from Copilot May 20, 2026 03:32

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 19 changed files in this pull request and generated 3 comments.

Comment thread components/indexer/sqlitevec/indexer.go
Comment thread components/indexer/sqlitevec/indexer.go Outdated
Comment thread components/retriever/sqlitevec/options.go Outdated
@Lonliet Lonliet force-pushed the feat/sqlitevec-vector-store branch from 601102f to 4d0de07 Compare May 20, 2026 05:07

Lonliet commented May 20, 2026

Copy link
Copy Markdown
Author

Addressed the latest Copilot review comments in 4d0de07:

  • Wrapped tx.Commit() failures with [Store] commit failed for consistent Store error context.
  • Renamed WithMaxDistance's parameter to maxDistance and updated its GoDoc.

Verification after the change:

  • go test ./... and go vet ./... passed for both sqlitevec modules locally.
  • go test -race ./... passed for both sqlitevec modules in Docker Linux (golang:1.25-bookworm).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants