feat(sqlitevec): add sqlite-vec vector store components#835
Open
Lonliet wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a pure-Go embedded SQLite + sqlite-vec vector store implementation for Eino by introducing a paired SQLiteVec Indexer (store docs + embeddings) and SQLiteVec Retriever (TopK similarity search), along with docs, examples, and unit tests. This expands eino-ext with a local/embedded RAG option without CGO or external vector DB services.
Changes:
- Introduce
components/indexer/sqlitevecto upsert documents + vectors into SQLite/vec0 (optionally auto-creating schema). - Introduce
components/retriever/sqlitevecto query vec0 KNN results and return EinoDocumentobjects with score + raw distance metadata. - Add module scaffolding (
go.mod/go.sumwith Go 1.25), README/README_zh, examples, and in-memory unit tests for both modules.
Reviewed changes
Copilot reviewed 17 out of 19 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| components/retriever/sqlitevec/retriever.go | Implements SQLiteVec retriever (embed query, vec0 KNN query, score + metadata handling, callbacks). |
| components/retriever/sqlitevec/retriever_test.go | In-memory SQLite tests covering validation, retrieval, options, and corrupt metadata. |
| components/retriever/sqlitevec/README.md | English documentation for retriever usage/config/options. |
| components/retriever/sqlitevec/README_zh.md | Chinese documentation for retriever usage/config/options. |
| components/retriever/sqlitevec/options.go | Adds retriever impl-specific option WithMaxDistance. |
| components/retriever/sqlitevec/consts.go | Defines component constants/defaults + metadata key name. |
| components/retriever/sqlitevec/codec.go | Table-name validation + vector/metadata JSON helpers for retriever. |
| components/retriever/sqlitevec/examples/retriever/main.go | Minimal runnable retriever example with in-memory DB seeding. |
| components/retriever/sqlitevec/go.mod | New retriever module (Go 1.25) + dependencies. |
| components/retriever/sqlitevec/go.sum | Dependency checksums for retriever module. |
| components/indexer/sqlitevec/indexer.go | Implements SQLiteVec indexer (batch embedding, upsert docs, write vec0 rows, callbacks, optional auto-create). |
| components/indexer/sqlitevec/indexer_test.go | In-memory SQLite tests covering validation, store/upsert behavior, and embedding error paths. |
| components/indexer/sqlitevec/README.md | English documentation for indexer usage/config/schema/notes. |
| components/indexer/sqlitevec/README_zh.md | Chinese documentation for indexer usage/config/schema/notes. |
| components/indexer/sqlitevec/consts.go | Defines component constants/defaults. |
| components/indexer/sqlitevec/codec.go | Table-name validation + metadata/vector JSON helpers for indexer. |
| components/indexer/sqlitevec/examples/indexer/main.go | Minimal runnable indexer example. |
| components/indexer/sqlitevec/go.mod | New indexer module (Go 1.25) + dependencies. |
| components/indexer/sqlitevec/go.sum | Dependency checksums for indexer module. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
5927961 to
601102f
Compare
601102f to
4d0de07
Compare
Author
|
Addressed the latest Copilot review comments in
Verification after the change:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add pure-Go sqlite-vec Vector Store support for Eino:
components/indexer/sqliteveccomponents/retriever/sqlitevecWhy
This provides an embedded vector store for local RAG, examples, tests, and small-scale applications.
It uses
modernc.org/sqliteandmodernc.org/sqlite/vec, so it does not require CGO or an external vector database service.Scope
Notes
modernc.org/sqlite/veccurrently requiresmodernc.org/sqliteversions that require Go 1.25 or newer; the new component modules currently use Go 1.25.10.MATCH ? AND k = ?; parameterizedLIMIT ?was not accepted by vec0 during testing.develop, but the upstream repository currently hasmainas its default branch and nodevelopbranch.Tests
Docker Linux,
golang:1.25-bookworm:go test ./...undercomponents/indexer/sqlitevecgo test -race ./...undercomponents/indexer/sqlitevecgo vet ./...undercomponents/indexer/sqlitevecgo test ./...undercomponents/retriever/sqlitevecgo test -race ./...undercomponents/retriever/sqlitevecgo vet ./...undercomponents/retriever/sqlitevecgolangci-lint runundercomponents/indexer/sqlitevecusinggolangci-lint v1.64.8golangci-lint runundercomponents/retriever/sqlitevecusinggolangci-lint v1.64.8Additional compatibility check:
golang:1.22-bookwormwithGOTOOLCHAIN=autoselects Go 1.25.10 for the sqlitevec modules and passesgo test ./...for both modules.Local Windows:
go test ./...passed for both modules.go vet ./...passed for both modules.go test -race ./...failed with Windows exit status0xc0000139; Docker Linux race tests passed.Pre-submit Checks
git diff --check origin/main..HEADpassed.cloudwego/eino-extissues/PRs matchingsqlitevec,sqlite-vec, or"sqlite vec".