feat: Separate VectorStore and the Embedder.

[espoirMur]

Feature description

Separate the Vector Store and the embedder.

Motivation

Embedding and Storage are two different concepts.

For a single responsibility purpose, we should separate the two entities, one should be responsible for computing the embedding of the document, and the one should be responsible for storing the embedding in the vector database.

Additional context

No response

[mhordynski]

Hey @espoirMur, thanks for raising this!

We decided to couple VectorStore and Embedders for a few reasons:

• If you mix different embedders with the same vector store, you can end up with mismatched vectors in your index.
• Some vector stores (pgvector atm) actually need to know the vector size right from the start, which is tied to whatever embedder you’re using.
• ragbits implementation of HybridSearch heavily benefits from the fact that we have information about Embedder in VectorStores.
• Pretty much every time users interact with a VectorStore (ingest/query), they want to hit the Embedder first anyway, so it just makes sense to keep them together.

Hope that clarifies why those interfaces are together.

[espoirMur]

Okay, @mhordynski!

Thanks for the clarification—that makes sense.

From a computational perspective, here's how I see it: When ingesting a large number of documents, you could:

Read the documents,

Cache them in object storage,

Compute the embeddings (also caching them in object storage),

Finally, load the embeddings into the database.

This could be structured as a pipeline:
Document → Embedding Computation → Database Storage

One advantage of this approach is scalability—in a distributed setup, you could allocate different hardware for each step. For example:

A GPU-powered machine for embedding computation,

A lighter machine for the remaining pipeline steps.

That said, I haven’t tested which method would be faster in practice.

[espoirMur]

As I am playing with the library, I am getting more thought on this.

Hey @espoirMur, thanks for raising this!

We decided to couple VectorStore and Embedders for a few reasons:

• If you mix different embedders with the same vector store, you can end up with mismatched vectors in your index.

There can be other ways to circumvent this.

And mathematically, embeddings are representation in a vector space. The same content will be represented the same way in the same embedding space, regardless of the embedding model they use. I may be wrong on this, I am just using Algebra to understand. I can dig more into the embedding representation.

• Some vector stores (pgvector atm) actually need to know the vector size right from the start, which is tied to whatever embedder you’re using.

Yes I understand this but you can instantiate the vectorstore with the embedding dimension from he embedder. I don't think you have to use the embedder in. the vector store for this.

• ragbits implementation of HybridSearch heavily benefits from the fact that we have information about Embedder in VectorStores.

I can't comment on this as I haven't tried the hybrid search yet to give my thoughs on this.

• Pretty much every time users interact with a VectorStore (ingest/query), they want to hit the Embedder first anyway, so it just makes sense to keep them together.

You can add all of them to the same pipeline. You hit the embedder to generate the embedding, and then you query or insert the embedding into the database.

Hope that clarifies why those interfaces are together.

[micpst]

And mathematically, embeddings are representation in a vector space. The same content will be represented the same way in the same embedding space, regardless of the embedding model they use.

If you mix up the models the embeddings will have different vector space.

Speaking of scalability, you can host the embedder model via vllm or llama.cpp and run it on dedicated infra and keep the index task fully io bound. The only drawback is caching, with the current implementation it can be cumbersome because we can't do it at the ingest strategy level.