False alarmsOn 40 never-recalled products. Shorter is better.
SResults
What we measured, including what went against us.
Every number on this site comes from our own test runs in the repository. Where a simpler method does better on something, we say so.
Agent test set21/22cases pass, and 65 of 66 individual checks
Red team18/18attacks held, including 9 hard negatives
Catalog scan, 500 listings49/50planted problems found, with no wrong findings
Investigate, median6.6 sper live check; chat answers in 7.6 s
Agent tests ran against the live API on 16 Oct 2026The backtest on 13 Oct 2026
Do buyer reviews warn before a recall?
25 real US recalls (CPSC) and 40 products that were never recalled. Each method reads only the reviews written before the recall date. We count the recalls it would have flagged, and the safe products it would have wrongly flagged.
SafeShelfBenchRecalls flagged, how early, and false alarms
+| Method | Recalls flagged | Median warning | False alarms |
|---|---|---|---|
| Keyword rules | 11/25 | 170 weeks | 11/40 |
| Gemini alone | 6/25 | 104 weeks | 6/40 |
| SafeShelf | 6/25 | 93 weeks | 2/40 |
| SafeShelf, without pooling copies | 1/25 | 67 weeks | 2/40 |
"Median warning" is how long before the official recall the first warning would have appeared. Pooling the reviews of every copy of the same product is what moves SafeShelf from 1 recall to 6.
Where we fall short. Keyword rules flag more recalls (11 against our 6) and earlier, at the cost of 11 false alarms. On the half of the recalls we did not tune on, SafeShelf flagged 2 of 9 with no false alarms on 14 safe products. The sample is small, and we publish both numbers.
8,689 official recalls and notices.
Loaded from the regulators' own feeds and press releases. In India, raids and "don't buy without the ISI mark" notices are kept, but they never mark a product recalled; they only send it to review.
What else is on fileThe catalog behind the identity engine
Products27,800about, in the catalog
Real reviews29,379mostly from amazon.com
Energy registry8,470Australian records
Regions8US and 7 Asia-Pacific
Records by regionOfficial recalls and notices
| Region | Source | Records |
|---|---|---|
| Australia | ACCC | 2,962 |
| United States | CPSC | 2,537 |
| South Korea | KATS | 2,242 |
| Japan | CAA | 677 |
| ASEAN | ASEAN product alerts | 161 |
| Taiwan | BSMI | 45 |
| India | CCPA and BIS | 36 |
| Singapore | CPSO | 29 |
| Total | 8,689 |
Early work: recalled in one country, then another.
We found 19 products recalled in one country and later in another, mostly the US first and Australia later. The second recall came a median of 22 days after the first, and up to 293 days.
Not yet checked by a personThese 19 pairs, and the 34 recalled products we found possibly still on sale, are leads from Firecrawl and Gemini. None has been hand-checked yet, so we do not count them as findings.
Median gap between the first recall and the second, in another country.
Run the numbers yourself. It's all in the repo.
The bench code, the test sets and every result file are in the repository.
