Turbovec: Fast Local Vector Search for RAG Pipelines

RAG systems often start as a small prototype and then turn into an infrastructure problem. Embeddings multiply, memory use climbs, access-control filters get bolted on late, and teams have to decide whether to run another managed vector service. Turbovec is interesting because it keeps the index local while attacking the two pain points SREs notice first: RAM and latency.

What Is Turbovec?

Turbovec is a vector index built in Rust with Python bindings. It implements TurboQuant, a quantization approach that compresses vectors without a separate training phase. The project says a 10 million document corpus that would take about 31 GB as float32 vectors can fit in about 4 GB with turbovec.

That matters for AI infrastructure teams that want semantic search inside a service, a private VPC, or an air-gapped environment. Instead of shipping embeddings to a managed database, you can keep the index close to the application and persist it to disk.

Key Features

• Online ingest: Add vectors as the corpus grows without a train step or full rebuild.
• Compressed storage: Use 2-bit or 4-bit quantization to reduce memory pressure for large embedding sets.
• Fast local search: Rust internals use SIMD kernels with NEON on ARM and AVX paths on x86.
• Stable IDs: IdMapIndex supports external uint64 IDs and O(1) removal by ID.
• Search-time filtering: Pass an allowlist so tenant, ACL, time-window, or SQL filters are enforced inside the search path.

Installation

For Python workloads:

pip install turbovec

For Rust services:

cargo add turbovec

Quick Usage Example

import numpy as np
from turbovec import IdMapIndex

index = IdMapIndex(dim=1536, bit_width=4)
index.add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64))

scores, ids = index.search(query, k=10)

index.write("docs.tvim")
loaded = IdMapIndex.load("docs.tvim")

The allowlist path is the more operationally useful part. A database, BM25 index, or policy service can first narrow the candidate IDs, then turbovec reranks only that set:

allowed = np.array(db.execute(
    "SELECT id FROM docs WHERE tenant_id = ?",
    (tenant_id,),
).fetchall(), dtype=np.uint64)

scores, ids = index.search(query, k=10, allowlist=allowed)

Operational Tips

Treat turbovec like an embedded index, not a magic replacement for your source of truth. Keep documents, metadata, and authorization rules in a database you already operate. Use turbovec for dense retrieval, persist index files during deploys, and add a rebuild path that can recreate the index from canonical data.

For SRE teams, the main checks are straightforward. Measure recall on your own queries before lowering bit width, track index size and reload time, and test allowlist behavior with small tenant sets. If your RAG system handles customer data, filtering inside the search path is worth validating carefully because over-fetching and post-filtering can leak quality or access-control bugs into production.

Conclusion

Turbovec is a practical option for teams that want fast local vector search without adding another service to the stack. It is especially worth a look for private RAG, agent memory, incident knowledge bases, and edge deployments where memory budget and data locality matter. Check out the GitHub repository for API details and benchmark notes.

Looking for an AI-powered platform to help your SRE team? Akmatori helps teams automate incident response and infrastructure management. Backed by Gcore, we're building the future of intelligent operations.