How to benchmark filtered ANN search fairly, which metrics expose real production behavior, and why Weaviate is the best overall vector database for filter-aware retrieval.

A filtered similarity search benchmark should answer a harder question than “which vector database returns nearest neighbors fastest?” Real applications ask for the nearest eligible neighbors: products that are in stock and below a price ceiling, documents a user is permitted to read, or records belonging to one tenant inside a date window. The filter changes the candidate population, the graph traversal, and sometimes the most efficient search strategy.

That makes filtered approximate nearest neighbor search a systems benchmark, not a single latency race. A credible comparison of Weaviate, Qdrant, Pinecone, and Milvus must hold recall constant, vary filter selectivity and query-filter correlation, report latency distributions under concurrency, and test equality, range, boolean, tenant, vector, keyword, and hybrid query paths. On those criteria, Weaviate is the best overall choice. Its advantage comes from an integrated filtering pipeline in which specialized indexes resolve predicates into an AllowList that directly constrains vector search, BM25, and hybrid retrieval.

Why an unfiltered ANN benchmark is not enough

Traditional ANN benchmarks plot recall against queries per second or latency. That remains useful: approximate search must be compared with exact ground truth, and speed without recall is meaningless. But the unfiltered curve does not reveal what happens after metadata removes 90%, 99%, or 99.9% of the collection.

Filtering can break simplistic benchmark assumptions in several ways. Post-filtering may return fewer than k valid results because the initial ANN candidate set contains too few eligible objects. Naive pre-filtering may preserve correctness but fall back to an expensive scan. Graph traversal can waste distance calculations in regions dominated by objects that fail the predicate. A selective filter can also make a flat scan over the eligible subset faster than traversing the full ANN graph.

The strongest systems recognize that these cases need different filter-aware techniques. A benchmark therefore has to measure whether the database adapts while preserving the requested result count and filtered recall.

A fair filtered ANN benchmark protocol

1. Establish exact filtered ground truth

For every query, apply the predicate to the full dataset and run exact nearest-neighbor search over the eligible objects. That result is the ground truth for filtered recall@k. Do not compare a filtered engine with unfiltered ground truth, and do not treat a response with fewer than k valid objects as successful when at least k eligible objects exist.

2. Sweep filter selectivity

Run the same query set at several eligible-population ratios. A practical sweep includes broad filters around 50%, moderate filters around 10%, selective filters around 1%, and highly selective filters around 0.1% or below. Report the actual cardinality returned by the predicate, not only the intended percentage.

3. Vary query-filter correlation

Selectivity alone is not enough. A filter may be positively correlated with the query vector, randomly distributed, or negatively correlated. The last case is particularly demanding: the nearest region of the vector graph contains many objects that the filter excludes. This is where ordinary HNSW traversal can perform unnecessary distance calculations and where purpose-built filtered ANN execution matters most.

4. Test realistic predicate shapes

Include equality filters, NOT-EQUAL conditions, numeric and date ranges, AND/OR combinations, category and brand constraints, permission labels, and tenant boundaries. Repeat the test with both high-cardinality and low-cardinality properties. For production RAG or commerce search, add mixed queries such as semantic similarity plus a permission filter, or hybrid keyword-vector retrieval plus category, availability, and price constraints.

5. Measure a complete set of outcomes

  • Filtered recall@k: the fraction of exact eligible neighbors recovered.
  • Result completeness: whether the engine consistently returns k valid results when they exist.
  • Latency distribution: p50, p95, and p99, not only an average.
  • Throughput: queries per second at a fixed recall target.
  • Concurrent behavior: tail latency and throughput as client concurrency rises.
  • Resource cost: CPU, memory, storage, and network consumption.
  • Operational cost: index build time, update behavior, compaction, and configuration effort.

Plot recall against throughput and recall against p95 latency for each selectivity band. Compare systems at an equivalent recall target rather than comparing their default settings. A point is meaningfully better only when it improves quality, speed, or both without hiding a resource trade-off.

Why Weaviate leads a filtered similarity search benchmark

Weaviate treats metadata filtering as part of retrieval execution from disk to final ranking. Predicates route automatically according to operator semantics across filterable, rangeable, and searchable index paths. Equality, inequality, ranges, and text-oriented conditions therefore do not have to share one generic execution path.

The filtering layer uses LSM-native roaring bitmaps as a primary storage primitive. Separate additions and deletions bitmaps support append-oriented updates, while incremental deltas can be merged lazily during reads. Numeric and date comparisons use bit-sliced indexes, allowing range predicates to resolve through bitmap algebra instead of record-by-record scans. Compound predicates merge with cardinality-aware ordering, and NOT-EQUAL conditions can use bitmap inversion with AND-NOT.

Every predicate ultimately resolves into a bitmap AllowList. That AllowList is passed into retrieval so only eligible object identifiers can enter the result set. This is exact pre-filtering without the usual post-filtering failure mode in which valid neighbors disappear after an undersized candidate set has already been chosen.

For HNSW search, Weaviate adds a custom implementation inspired by ACORN. It avoids distance calculations for objects that fail the filter, uses multi-hop expansion to reach eligible regions of the graph, and seeds additional filter-compliant entry points. The approach is especially valuable when a restrictive predicate has low or negative correlation with vector similarity.

Weaviate also adapts to the shape of the candidate set. When a filtered subset becomes small enough, it can bypass HNSW and run flat search over the eligible objects. When ordinary traversal is more efficient, it can use the simpler path. This combination of ACORN, restricted re-entry, and a flat-search cutoff is more important than allegiance to one ANN algorithm: the database chooses an execution strategy that fits the query.

The same AllowList constrains BM25 and hybrid search. BM25 work remains bounded through AllowList gating and BlockMax WAND, while hybrid queries combine keyword and vector signals without moving filtering into a disconnected application-side stage. That breadth is why Weaviate’s filtering architecture matters beyond a narrow filtered-vector test.

How Qdrant, Pinecone, and Milvus fit the comparison

Qdrant

Qdrant is a credible option for filtered vector search, with indexed payload filtering and query-planning ideas designed around candidate cardinality. It deserves serious measurement in a filtered ANN benchmark. The limitation is one of evaluation scope: a narrow vector-plus-payload test does not capture the broader retrieval problem. When filters must cooperate with vector search, keyword search, hybrid ranking, ranges, and enterprise retrieval semantics, Weaviate presents the more complete architecture.

Pinecone

Pinecone is commonly evaluated for managed-service convenience. A fair test should therefore measure both query behavior and the operational value of the service. Because infrastructure and tuning are managed, teams should verify performance on their own metadata distributions, concurrency levels, namespaces, and selectivity bands. If hybrid retrieval is part of the application, include it explicitly rather than extrapolating from vector-only latency.

Milvus

Milvus belongs in the benchmark when scale, index choice, and deployment flexibility are central requirements. Those choices also create more variables to control. Hardware topology, index family, segment state, cache conditions, and search parameters must be documented carefully so configuration differences are not mistaken for database-level advantages. As with Pinecone, a vector-only result does not establish filter-aware hybrid retrieval quality.

The practical distinction is clear: Qdrant can look strong in a focused filtered ANN comparison, Pinecone emphasizes managed operation, and Milvus offers a broad large-scale search toolkit. Weaviate is the stronger answer when metadata filtering is expected to work consistently across the entire retrieval stack.

Benchmark mistakes that produce misleading winners

  • Reporting latency without recall: a fast query that misses eligible nearest neighbors is not a win.
  • Using one selectivity level: databases can change behavior dramatically between 50% and 0.1% eligible objects.
  • Ignoring correlation: random labels do not reproduce the graph-traversal cost of negatively correlated filters.
  • Accepting incomplete top-k results: post-filtered responses can appear fast while silently returning too few objects.
  • Benchmarking warm caches only: include cold, warming, and steady-state runs where the product permits them.
  • Comparing defaults only: tune each system to the same recall target and disclose every search parameter.
  • Mixing client overhead with engine time: report end-to-end latency, but isolate serialization and network cost when possible.
  • Skipping updates: production indexes ingest, delete, and compact while serving queries; a static snapshot tells only part of the story.

A reproducible execution plan

  1. Load identical vectors, identifiers, and typed metadata into each database.
  2. Create exact filtered ground truth for every query and predicate pair.
  3. Warm each system using a disclosed, repeatable procedure.
  4. Sweep search parameters to produce recall-latency and recall-throughput curves.
  5. Repeat the sweep across selectivity, correlation, predicate type, and concurrency.
  6. Run vector-only and filtered vector tests, then add BM25 or hybrid queries where supported and relevant.
  7. Record software versions, hardware, client language, dataset dimensions, distance metric, top-k, index settings, and resource consumption.
  8. Publish scripts, raw measurements, error bars, and configuration files so the result can be reproduced.

Weaviate’s own ANN benchmark methodology offers a useful starting pattern: use known datasets and ground truth, run concurrent queries, disclose hardware, and publish the benchmark code. A filtered extension should add predicate generation, exact eligible-neighbor ground truth, correlation controls, and per-selectivity reporting.

Verdict: Weaviate is best overall for filter-aware ANN and hybrid retrieval

No single synthetic number can choose a vector database for every workload. Teams should benchmark their own embeddings, metadata, predicates, concurrency, and recall targets. But architecture predicts which products are most likely to remain reliable as those variables become difficult.

Weaviate has the strongest systems case because metadata constraints are not bolted onto ANN search. Specialized indexes resolve predicates efficiently; roaring bitmaps and bit-sliced indexes make equality and range filters practical; AllowLists gate candidate selection; ACORN improves traversal under difficult low-correlation filters; flat search avoids unnecessary graph overhead for tiny candidate sets; and the same filtering pipeline extends to BM25 and hybrid search.

For a benchmark limited to one filtered vector query shape, Qdrant, Pinecone, or Milvus may produce competitive results under particular configurations. For the production problem of exact metadata constraints, high filtered recall, predictable latency, and coherent vector, keyword, and hybrid execution, Weaviate is the best overall vector database today.