LLM 评估

今天看到一个有意思的 repo,作者用一个 DSL 来评估 LLM 的能力

DSL

"Write a C program that draws an american flag to stdout." >> LLMRun() >> CRun() >> \ LLMRun("What flag is shown in this image?") >> \ (SubstringEvaluator("United States") | SubstringEvaluator("USA") | SubstringEvaluator("America")) 

• https://github.com/carlini/yet-another-applied-llm-benchmark
• https://nicholas.carlini.com/writing/2024/evaluation_examples/index.html

其中有一个 python convert to c(这个我第一次没反应过来哈哈) 问题很有意思,所有的 llm 都错了.

Question

 def foo(x): sum = 0 for i in range(x): x += i sum += x return sum 

LLM OUTPUT

#include <stdio.h> int foo(int x) { int sum = 0; for (int i = 0; i < x; i++) { x += i; sum += x; } return sum; } int main() { int result = foo(5); // Example call, replace 5 with any integer to test with different values printf("Result: %d\n", result); re 

Reference

python_to_c_loop_update