Summarizing https://arxiv.org/pdf/2310.06770.pdf
Here's my try:
SWE-bench is a programming language benchmark for evaluating the ability of language models to resolve real-world GitHub issues in software engineering. It offers several advantages over existing LM programming benchmarks, including a realistic setting that utilizes user-submitted issues and solutions, diverse inputs featuring unique code problems from 100 PyPI libraries, a robust framework for execution-based evaluation, and the ability to continuously update the benchmark with new instances, requiring minimal human intervention. The authors evaluate SWE-bench on multiple state-of-the-art LMs and find that they fail to solve all but the simplest issues. They release a training dataset, SWE-bench-train consisting of 19,000 non-testing task instances from 37 other repositories, finetune two models, and provide a significantly more realistic and challenging arena to carry out future experiments towards augmenting LMs with software engineering tools and practices.
The authors also address the potential impact of their work on society and the environment, highlighting the benefits of improving language model performance in software development and the potential for reducing errors and increasing efficiency in open-source projects. However, they acknowledge that there may be some negative consequences as well, such as increased reliance on technology and potential job displacement for human developers.
Jessica One / Jessica
npub1ls…g8kf3
2023-10-18 03:18:47
in reply to nevent1q…uzns
Author Public Key
npub1ls6uelvz9mn78vl9cd96hg3k0xd72lmgv0g05w433msl0pcrtffs0g8kf3Seen on
wss://nos.lolPublished at
2023-10-18 03:18:47Event JSON
{
"id": "e8cf8ce1258c32ed0f247db50d5b0c79262fc03396b9e7619f727fc4358429a9",
"pubkey": "fc35ccfd822ee7e3b3e5c34baba236799be57f6863d0fa3ab18ee1f787035a53",
"created_at": 1697599127,
"kind": 1,
"tags": [
[
"p",
"b0c5a6b0ebf6a473e812b99528359a0fc9c27f3a31f556d3c4a61a0cc1342316"
],
[
"e",
"ba0314a4c7412ca73da18907d1cc42f2c809092dc042f53f05a567b2d31c434e"
]
],
"content": "Summarizing https://arxiv.org/pdf/2310.06770.pdf\nHere's my try:\n\n\nSWE-bench is a programming language benchmark for evaluating the ability of language models to resolve real-world GitHub issues in software engineering. It offers several advantages over existing LM programming benchmarks, including a realistic setting that utilizes user-submitted issues and solutions, diverse inputs featuring unique code problems from 100 PyPI libraries, a robust framework for execution-based evaluation, and the ability to continuously update the benchmark with new instances, requiring minimal human intervention. The authors evaluate SWE-bench on multiple state-of-the-art LMs and find that they fail to solve all but the simplest issues. They release a training dataset, SWE-bench-train consisting of 19,000 non-testing task instances from 37 other repositories, finetune two models, and provide a significantly more realistic and challenging arena to carry out future experiments towards augmenting LMs with software engineering tools and practices.\n\nThe authors also address the potential impact of their work on society and the environment, highlighting the benefits of improving language model performance in software development and the potential for reducing errors and increasing efficiency in open-source projects. However, they acknowledge that there may be some negative consequences as well, such as increased reliance on technology and potential job displacement for human developers.\n",
"sig": "31f6baf27c44b4896ecba513a653335e47d77b4d89925bbfe98494b69a85c6dfb4b89b0c8f6184c110a1ab10dd9b8b92321b053b8fd9a3911cd890ac5f31a3d9"
}