Humanity’s List

AIarXiv

Ranked by formulaVerdict: Notable

CulturalRiddles: A Multicultural Benchmark of Riddles

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important.

By Nguyen, Hu, Anbalagan +155

Score████░░░░░░4.4

Key numbers

  • 22% accuracy in some communities

Caveats

  • Preprint; not yet peer reviewed.

VerdictWorth a reader's time today.

Read the originalPDFCode

Abstract

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.

My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, et al.

The editor's rubric

Heuristic review

Leverage4 / 5
A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
24% of this desk’s rubric
Magnitude2 / 5
Solid incremental gain on a meaningful problem.
18% of this desk’s rubric
Evidence4 / 5
Strong: large scale, preregistered, independently replicated, or a well-powered randomized trial.
14% of this desk’s rubric
Novelty2 / 5
A new combination of known ideas.
16% of this desk’s rubric
Trajectory2 / 5
Some room to improve with obvious engineering.
18% of this desk’s rubric
Stakes2 / 5
Benefits a professional community (practitioners, clinicians, engineers).
10% of this desk’s rubric

Editor’s rationale

Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (many tasks); design (multicenter); verification (multiple benchmarks, human evaluation, code released); scale (scalable); stakes (general AI). Red flags: derivative (comparative study).

How the score was computed

rank-2026-10-18

Score████░░░░░░4.4

Score = 10 × (65% × adjusted merit / 10 + 25% × attention + 10% × freshness)

Desks place stronger verdicts first, then use this score to order work within the same verdict.

Merit
5.5 / 10
Weighted rubric, evidence-gated.
Adjusted merit
4.7 / 10
Shrunk toward the desk prior by editor confidence (44%).
Attention
25%
Citations, upvotes, points, mentions.
Freshness
78%
Half-life decay since publication.
  • Citations0 (reference 15, via semantic-scholar, Oct 22, 2026, 13:43 UTC)
  • Influential citations0 (reference 3, via semantic-scholar, Oct 22, 2026, 13:43 UTC)
  • Hugging Face upvotes8 (reference 25, via hf-daily, Oct 22, 2026, 13:43 UTC)
  • GitHub stars1 (reference 250, via hf-daily, Oct 22, 2026, 13:43 UTC)

The record

  • Reviewed by heuristic-v7 on Oct 22, 2026, 13:43 UTC. Paper type: method.
  • Categories: cs.CL, cs.AI
  • BRIEF, No.2 in the Artificial Intelligence edition of October 22, 2026.