NEEDLE rebuilds search benchmarks before engines can memorize them

A search benchmark becomes less useful when the systems being tested can download its answer key. Keenable.ai’s open-source NEEDLE addresses that problem by generating new queries from live public data instead of publishing a permanent test set that search agents can memorize.

The benchmark evaluates search APIs against five workloads: news, finance, scholar, legal retrieval and agentic_rare, a stream of unusual queries drawn from real agent activity. News queries refresh hourly; the other workloads generate fresh material daily from sources including RSS feeds, Google Trends, SEC XBRL, arXiv, Europe PMC, CourtListener and public search-agent logs.

The project is available as a Python command-line harness in the official NEEDLE repository. It is not a search product. A typical run has two stages:

text needle <benchmark> generate … --out queries.jsonl needle <benchmark> run --queries queries.jsonl …

The repository commits none of the generated query or gold-answer data. That separation means a model can inspect the code and scoring rules, but it cannot simply retrieve a fixed answer file during evaluation.

What the five benchmarks actually test

NEEDLE does not use one generic relevance score for every search task. Each module targets a different failure mode in agentic retrieval.

News starts with the newest item from about 124 curated public feeds, usually within a one-hour freshness window, and can also ingest Google Trends RSS across all 52 US geographies. An LLM projects each item into a keyword query and rejects evergreen topics. Results are judged on a 0-to-4 relevance scale and scored with nDCG@5, including a penalty for duplicate URLs.

Finance uses Wikidata and GLEIF for company facts and SEC XBRL for single-quarter 10-Q figures. The registry suite can ask for a CEO, headquarters country, founding year, website, employees, legal entity identifier or ticker. The filings suite defaults to net income, operating income and diluted EPS, with revenue available as an additional field. It scores whether the answer appears in the top K results rather than requiring one particular URL, using deterministic matching with tolerances for names, money values and employee counts.

Scholar is a known-item test designed to separate metadata search from full-text retrieval. Each paper produces four query forms: a degraded title, a detail that appears only in the full text, a natural-language clue and a vague “tip-of-the-tongue” description. The system must return the correct arXiv ID, DOI or PMID. A paper remains in the cohort only when it passes the requested bucket checks, keeping comparisons between query types meaningful.

Legal retrieval uses recent CourtListener opinions from 14 federal courts and eCFR sections. It checks identifiers such as reporter citations, docket numbers, CourtListener or Justia URLs, and CFR citations. About half of finance and legal queries include operators such as site:, after: or before:; NEEDLE parses those operators and maps them to each engine’s native mechanism where possible instead of sending unsupported syntax as literal text.

The agentic_rare module is the closest match to the long-tail queries used by research agents. Its filter keeps English queries containing at least one rare word—defined as a term that BERT WordPiece splits into at least five subwords or maps to [UNK]. It removes URL-like strings, VINs, hashes, crypto addresses and near-duplicates, then samples queries by short, medium and long length buckets. Like news, it uses an LLM relevance judge and nDCG@5.

A pooled ceiling shows whether retrieval or ranking is broken

Every engine receives identical query text. The runner makes one request at a time, caps evidence at 2,000 characters per result and evaluates the title and snippet exactly as returned. It does not fetch pages or rerank the providers' output. Search and judging failures are reported separately and excluded from the mean rather than counted as zero.

NEEDLE’s ultimate engine pools the deduplicated results from all participating engines for each query and creates an oracle-ranked list: judge ratings determine the order for relevance-scored tasks, while the gold identifier determines the order for known-item retrieval. That gives each individual provider a ceiling based on what the field collectively retrieved.

A low score against a high ultimate value indicates a ranking or surfacing problem. A low ultimate value means the combined providers failed to retrieve strong evidence at all. This distinguishes a weak individual engine from a gap shared by the providers.

The published seven-day averages below cover the window ending August 28, 2026. Finance is comparatively strong, while rare-entity retrieval leaves a much larger gap to the pooled ceiling.

The agentic-rare figure shows the gap: the leading engine reached only 0.557 of ultimate, and Keenable reached 0.470. The benchmark’s own documentation describes this as a query-retrieval problem closer to research-agent traffic than title-based lookup. For scholar, the spread reflects the difference between queries answerable from paper metadata and queries whose identifying evidence exists only in the body.

Latency also matters because an agent may call search repeatedly during one task. In the same reporting window, Keenable-realtime recorded 193 ms p50 and 284 ms p95, compared with Exa at 1,876 ms and 2,955 ms, and Bing at 2,767 ms and 9,381 ms. Those figures come from serialized requests under the benchmark protocol, not concurrent load, so they are intended as comparable request latencies rather than a complete measure of system throughput.

How to reproduce the runs

NEEDLE runs with uv sync and can execute on a laptop or in CI. Most engines require their own API key, while KEENABLE_API_KEY is optional and falls back to a keyless, rate-limited endpoint. The default model for query projection and judging is openai/gpt-5.6-terra, configurable through NEEDLE_LLM_MODEL, NEEDLE_JUDGE_MODEL or command-line overrides.

Continuous runs are defined in bench.yaml: news runs hourly with a limit of 20 queries; finance runs at 00:17 UTC with a limit of 120; agentic_rare runs at 06:17 UTC with a limit of 100; scholar runs at 12:17 UTC; and legal runs at 18:17 UTC. Results append summary, URL-overlap and per-engine uniqueness data to GitHub Pages. Full artifacts are archived in the keenable-ai/needle-results Hugging Face dataset, while the repository’s code is released under the MIT license.

The benchmark also incorporates findings from the DeepResearchGym research paper, which analyzed 14.44 million search requests across 3.97 million sessions. More than 90% of multi-turn sessions had ten or fewer steps, 89% of inter-step gaps were under one minute, and 54% of newly introduced query terms could be traced to previously retrieved evidence. Those measurements explain why NEEDLE emphasizes fresh query generation, repeated calls and rare terms rather than treating search as a single question-and-answer event.

The design has one stated limitation: snippet-only evaluation is a lower bound. A provider may have found the right page but fail to expose the identifying text within its returned snippet, while a known-item score can miss pages that contain no inline identifier. NEEDLE makes those constraints uniform across providers, but it cannot measure information that the API never returns.