Metadata filtering in vector search: post-filter loses results, measured

Multi-tenant RAG needs a filter: “only search documents this user may see.” Vector indexes are built for nearest neighbours, not for “nearest neighbours among these 1,000.” So where does the filter go: after the search, inside it, or before it?

Short answer: post-filtering silently returns fewer results as the filter gets more selective. In my test, a filter that keeps 1% of the data returned 1.9 results out of 10 when I fetched the top 100 and filtered. In-graph filtering fixed recall but got slow. For selective filters, exact search over the filtered subset was the fastest and always correct.

The setup

100,000 vectors, each with a tenant id. I ran two layouts:

• random: the tenant is independent of content. Think “documents from many customers about the same product.”
• topical: each tenant owns whole topic clusters, and its queries land in its own region.

Selectivity is the fraction of the corpus the filter keeps: about 50%, 10%, 1% and 0.1%. Ground truth is exact search restricted to the tenant’s vectors, so recall@10 here means “how many of the true filtered top 10 did the method return.”

Methods:

1. post-filter: ask HNSW for the top k’ (10, 100 or 1000), drop results that fail the filter, keep up to 10.
2. in-graph filter: hnswlib’s filter argument, evaluated while the graph is traversed (hnswlib).
3. pre-filter exact: select the allowed rows, then do a brute-force dot product. I timed two versions: build the subset per query (mask + gather) and keep a per-tenant matrix prebuilt (partitioned).

def post_filter(index, mask, q, fetch=100, k=10):
    labels, _ = index.knn_query(q, k=fetch)
    return [l for l in labels[0] if mask[l]][:k]

def in_graph(index, mask, q, k=10):
    labels, _ = index.knn_query(q, k=k, filter=lambda i: bool(mask[i]))
    return list(labels[0])

def pre_filter(X_tenant, ids_tenant, q, k=10):  # X_tenant = X[ids_tenant], built once
    s = X_tenant @ q
    top = np.argpartition(-s, k)[:k]
    return list(ids_tenant[top[np.argsort(-s[top])]])

Results: random layout

Recall@10 / average results returned (max 10) / median latency.

Selectivity	post k=10	post k=100	post k=1000 (ef=1000)	in-graph filter	pre-filter (partitioned)
50%	0.553 / 5.5 / 0.21 ms	1.000 / 10 / 0.24 ms	1.000 / 10 / 1.94 ms	1.000 / 10 / 0.44 ms	1.000 / 10 / 1.38 ms
10%	0.181 / 1.8 / 0.20 ms	0.908 / 9.1 / 0.22 ms	1.000 / 10 / 1.83 ms	1.000 / 10 / 2.19 ms	1.000 / 10 / 0.16 ms
1%	0.110 / 1.1 / 0.20 ms	0.191 / 1.9 / 0.22 ms	0.884 / 9.2 / 1.88 ms	1.000 / 10 / 17.03 ms	1.000 / 10 / 0.03 ms
0.1%	0.100 / 1.0 / 0.14 ms	0.105 / 1.0 / 0.16 ms	0.214 / 2.1 / 1.79 ms	1.000 / 10 / 74.22 ms	1.000 / 10 / 0.01 ms

Post-filtering behaves like a coin flip with the selectivity as the odds: fetch 10, keep about selectivity × 10. The query is never told that most of its neighbours are off-limits. Even fetching 1,000 candidates returned only 2.1 results per query at 0.1%.

In-graph filtering never lost recall, but its cost grows as the filter tightens, from 0.44 ms at 50% to 74.22 ms at 0.1%. The traversal has to wander through many forbidden nodes to find allowed ones. At 0.1% it took 74.22 ms, while the exact scan of the 100-vector subset took 0.01 ms.

Results: topical layout

When tenants own their own regions of the space, the problem mostly disappears.

Selectivity	post k=10	in-graph filter	pre-filter (partitioned)
50%	1.000 / 0.20 ms	1.000 / 0.22 ms	1.000 / 1.34 ms
10%	1.000 / 0.20 ms	1.000 / 0.29 ms	1.000 / 0.16 ms
1%	1.000 / 0.12 ms	1.000 / 0.61 ms	1.000 / 0.03 ms
0.1%	1.000 / 0.07 ms	1.000 / 0.16 ms	1.000 / 0.01 ms

Recall is 1.000 for every method. The query’s nearest neighbours are already inside the allowed set, so the filter rejects almost nothing. This is the trap: post-filtering looks fine in a demo where a user asks about their own topic, then fails when a filter is independent of the query. Examples are a date range, a document type, or a tenant whose content overlaps with everyone else’s. Real tenants are somewhere between my two layouts, so test with your own data.

What the numbers mean

• Post-filter is only safe when the filter is loose or correlated with the query. Below about 10% selectivity on independent filters it breaks, and it breaks quietly: fewer results, not an error. The 0.1% row returned a single result on average.
• Over-fetching helps, but needs a lot. To reach 0.884 recall at 1% selectivity I had to fetch 1,000 and raise ef to 1,000, which cost 1.88 ms versus 0.22 ms.
• In-graph filtering is the default to reach for between ~10% and 100%, where it stayed between 0.22 and 2.19 ms (and at 10% the exact partitioned scan was faster, 0.16 ms).
• Below ~1%, scan the subset. At 1% the subset has about 1,000 vectors, so a dot product takes 0.03 ms. This is the same reasoning as in Do you need a vector database?: exact search is cheap at small sizes.

Caveats

• Synthetic Gaussian clusters, 128 dimensions, one index configuration. Real embeddings will have different structure. Treat the shape of the curves as the result, not the exact milliseconds.
• One tenant per setting, 200 queries each. Latency is a median on a shared 2 vCPU machine.
• The partitioned pre-filter times exclude building the per-tenant matrices. It needs memory for a second copy of the vectors, or an index per tenant.
• hnswlib’s filter is a Python callback here. A native filter in a database engine will have different costs. The qualitative trend (slower as selectivity falls) is reported for hnswlib in the SIEVE paper, which also notes that hnswlib applies the filter during search.
• My first run gave a misleading 32 ms for exact search over 50k vectors. The cause was a float64 query array: np.sqrt(D) returns a float64 scalar, and under NumPy 2 this upcast the query, so each dot product converted the whole float32 matrix. I fixed it by casting queries to float32 and reran everything. Check dtypes if your brute-force numbers look too slow.

Rule of thumb

1. Estimate selectivity per query: allowed rows divided by total rows.
2. Above ~50%, post-filter with a modest over-fetch is fine.
3. Between ~10% and 50%, use the index’s in-graph filter.
4. Below ~1%, or when the filter is a hard permission boundary, search the filtered subset exactly (or give each tenant its own index).
5. Always log how many results survive the filter. A silent drop from 10 to 1 is the failure to catch. A test that asserts len(results) == k on an independent filter would have caught it here.

For the basics of embeddings and indexes, see the free lesson Embeddings & vector databases.