Vectron - Distributed Vector Database

From Single-Node to Sharded Similarity Search – A Major Project Deep Dive

Vectron is our semester major-project prototype: a distributed vector database built from scratch in Go to explore how modern AI retrieval infrastructure behaves under real distributed constraints.

It’s not a production-ready system — it’s an intentional learning artifact. We implemented core pieces (sharding, Raft consensus, HNSW indexing, query routing, auth, reranking) to understand tradeoffs like latency vs. relevance, freshness vs. consistency, throughput vs. durability, and availability vs. coordination overhead in practice.

The repo shows active development (127+ commits), extensive benchmarking, performance tracking docs, and even draft IEEE paper LaTeX files — reflecting its research-oriented focus.

---

Why We Chose This Project

Vector search underpins semantic search, RAG pipelines, recommendations, and multimodal AI apps.
Most people use Milvus/Qdrant/Pinecone — we wanted to understand them by building core subsystems ourselves.

Key learning goals:

• How sharding + replication actually works at scale
• Raft consensus in practice (via Dragonboat)
• HNSW indexing in a distributed setting
• Query fan-out, top-k merging, and reranking tradeoffs
• Failure modes: node crashes, partitions, recovery behavior
• Observability, metrics, and operator decision loops

It was less about a shiny product and more about building strong distributed systems intuition.

Project Overview & Current Status

Vectron is a microservices-based distributed vector DB prototype:

• Single-node baseline — validated local vector storage/search
• Distributed components — API Gateway, Placement Driver (Raft coordinator), Worker nodes (data shards), Reranker service, Auth Service
• Sharding — Placement Driver assigns shards to workers
• Replication — Raft consensus for metadata/control plane
• ANN Indexing — HNSW on PebbleDB at each worker for fast local similarity search
• Query Flow — Client → Gateway → Workers (shard-routed) → optional Reranker → results
• Ingestion — Vector insert + metadata updates
• Auth/Feedback — JWT/API keys (etcd + bcrypt), user feedback (SQLite, 1-5 ratings)
• Client SDKs — Go, Python, JavaScript
• Management Console — React frontend (currently mock-data, backend APIs pending)

Current limitations (explicitly tracked in TODO.md):

• No cross-shard search aggregation yet (queries local per shard)
• TLS/SSL pending
• JWT auth in progress
• Management console lacks full backend integration
• Some tests broken/needing fixes

Active development: frequent commits, benchmarks (scalability, dimensions, concurrency), perf optimization tracker.

Technical Highlights

• Placement Driver — Dragonboat Raft for shard assignment, node membership, cluster coordination
• Workers — PebbleDB + HNSW for vector storage/search
• API Gateway — gRPC/REST entry point, auth, feedback, reranker integration
• Reranker Service — Rule-based result reranking with Valkey/Redis caching (future LLM/RL potential)
• Benchmarking Suite — Scripts for dataset size, dimension impact (64D–1024D), concurrency (1–100 clients), distributed scaling
• Observability — Structured logging, node metrics, perf profiles
• Fault Injection — Manual tests for crashes/partitions to validate recovery

We separated control plane (metadata, Raft) from data plane (vector serving) to avoid coordination blocking throughput — a key lesson for reasoning about failures.

What This Project Taught Me

Vectron shifted how I think about distributed systems:

• Failures aren’t bugs — they’re the default state. Design for them.
• Defaults = implicit policy → make them explicit and measurable.
• Observability + rollback paths > perfect happy-path code.
• Architecture isn’t just components — it’s incident behavior under stress.
• Tradeoffs (latency vs quality, freshness vs cost) must be intentional and documented.

Building this gave me intuition no textbook or cloud tutorial could: real distributed behavior emerges from failure modes, not benchmarks alone.

Current Status & Next

Repo: github.com/pavandhadge/vectron

• Active (127+ commits)
• Benchmarks/docs/perf trackers in place
• IEEE paper drafts (LaTeX) indicate research intent
• TODOs: cross-shard search, TLS, full auth, console APIs, test fixes

If you’re exploring distributed vector DBs or Raft/HNSW in practice, clone it, run the benchmarks, break it — there’s a lot to learn.

“Vectron taught us that in distributed systems, the hardest part is not making things work once — it’s making behavior predictable and explainable under stress.”