Topic: evaluation

A new study evaluates large language models (LLMs) based on their coding behaviors rather than just performance metrics like pass@k, highlighting that as models improve, traditional evaluation methods become less effective in distinguishing between them.

arxiv.org↗

OpenAI proposes establishing global standards for AI to enhance safety through coordinated evaluation and governance. This initiative aims to address AI risks by fostering international cooperation in the tech sector.

A new study proposes using large language models (LLMs) in a persona-grounded, open-ended simulation as discharge educators to better adapt to patients' literacy, recall, and personality needs, addressing limitations of current LLM evaluations that focus on static or artifact-generation tasks. This approach aims to improve patient understanding and adherence to discharge plans.

arxiv.org↗

Large language models were evaluated against physicians in diagnosing and treating cases within traditional Chinese medicine, with a clinical case library of 349 de-identified patients used to assess their effectiveness. This study highlights the potential of LLMs in TCM but also underscores the need for further validation in real-world applications.

arxiv.org↗

Chen, Zhao, and Cohan's study evaluates LLM-generated research ideas but faces criticism for survivorship bias in its human baseline, which includes only published papers, while the LLM baseline considers one-shot responses. This discrepancy highlights a potential flaw in how the LLM's performance is being compared to real-world scenarios.

arxiv.org↗

The study argues that while independent evaluation is crucial to rejecting harmful updates, it should not hinder the continual learning process for embodied agents. It suggests assessing update admission based on both error control and preserving learning opportunities within a set interaction limit.

arxiv.org↗