Topic: distributional evaluation

Chen, Zhao, and Cohan's study evaluates LLM-generated research ideas but faces criticism for survivorship bias in its human baseline, which includes only published papers, while the LLM baseline considers one-shot responses. This discrepancy highlights a potential flaw in how the LLM's performance is being compared to real-world scenarios.

arxiv.org↗