.Net: [MEVD] Add Batch Search

[tjwald]

---

[MEVD] Add Batch Search

I am authoring a high-performance library for AI applications and would like to introduce MEVD as the baseline abstraction for integration with vector searches.

However, most AI applications perform batch operations, and some vector stores have batch capabilities or at least the ability to submit multiple requests in a single round trip.

The current MVDE search abstraction only enables a single query at a time, limiting my ability to harness when possible the batch capabilities of the underlying storage.
Can we add a new abstraction that the different vector stores can optionally implement for batch operations?

Then I can do a capability test in my code:

if (vectorStore is IBatchSearchableVectorStore)
{
     var groupedResult = await vectorStore.BatchQueryAsync(batch, ...);
     // use results
}

If we can also have adapters for a vector store that will perform serial / parallel queries that would be cool, but that is easy to add in our code once the interface exists.

All names (interface, methods, ...) are TBD.

[tjwald]

Another point to remember during this review and implementation - running the embedding model in batches is a lot more efficient that one by one on different hardware.

[roji]

@tjwald can you please reference some documentation for batch search APIs on vector databases? In general, like with any abstraction, we tend to only add APIs which are relatively widely-supported, and probably wouldn't want to do it if only a small minority of vector databases would support it.

Am noting that here's there's the possibility of a fallback, where the default implementation of a batch searching API would simply do the searches in serial, and implementations for databases with actual batching capabilities would override (a bit similar to filtered deleting. However, doing this kind of thing makes the inefficiency of the serial implementation invisible, which may not be a good idea.

[roji]

Another point to remember during this review and implementation - running the embedding model in batches is a lot more efficient that one by one on different hardware.

Right, and FWIW we indeed do batching when upserting records, doing a single call to the embedding generator for the entire batch. If batch querying would make sense (because widely supported across databases), I agree it would be great to do the same there.

[tjwald]

@roji I don't think it is widespread which was why I suggested implementing this as a separate interface with a capability check and serial / parallel adapter class.

Also, the performance of some of these solutions might be slower than parallel/serial queries in certain scenarios and data sizes, so being able to add this functionality with on/off/adapters would probably be the best approach.

can you please reference some documentation for batch search APIs on vector databases?

Postgres - pgvector

No out of the box solution, but you can reduce the network calls by using:

• UNION ALL between the different vectors.
• Complex CTEs like:

WITH QueryVectors AS (
    SELECT '[1,2,3]'::vector AS q_vec
    UNION ALL
    SELECT '[4,5,6]'::vector AS q_vec
)
select id, distance
  from (
SELECT
    t1.id,
    t1.embedding <-> qv.q_vec AS distance,
    RANK() OVER (PARTITION BY t1.id) as rank
FROM
    your_table t1,
    QueryVectors qv
ORDER BY
    distance)
where rank < 10

Quadrant

https://qdrant.tech/articles/batch-vector-search-with-qdrant/

Faiss

https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU#performance

Milvus

https://milvus.io/docs/single-vector-search.md#Bulk-Vector-Search

neo4j

(AI Generated answer)

UNWIND [
    [0.1, 0.2, 0.3, ...], // Query vector 1
    [0.4, 0.5, 0.6, ...], // Query vector 2
    // ... more query vectors
] AS queryVector
CALL db.index.vector.queryNodes('myVectorIndex', 10, queryVector) YIELD node, score
RETURN node, score

[roji]

Postgres - pgvector
No out of the box solution, but you can reduce the network calls by using:

UNION ALL between the different vectors.
Complex CTEs like:

I think you missed that PostgreSQL (and pretty much all other relational databases) do support batching; the older way is to simply concatenate multiple SQL statements into the same command (SELECT ...; SELECT ...), but PostgreSQL (and possibly some others) also supports a newer ADO.NET batching API (see docs). Both of these methods allow batching arbitrary SQL statements without resorting to UNION/CTE, which impact the query return type etc.

I actually suspect that some form of batching may also be possible on some other databases (Cosmos, Mongo). That might mean that this is wide-spread enough to warrant inclusion of the API in the abstraction, and possibly fall back to serial execution where not supported.

[github-actions]

This issue is stale because it has been open for 90 days with no activity.