Evaluation

Understanding Large Language Model Performance in Software Engineering: A Large-scale Question Answering Benchmark
In this work, we introduce CodeRepoQA, a large-scale benchmark specifically designed for evaluating repository-level question-answering …
Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, Cuiyun Gao
PDF Cite DOI
Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study
Recent advancements in large language models (LLMs) have significantly enhanced their coding capabilities. However, existing benchmarks …
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen
PDF Cite Project