XLANG Lab
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Hongjin Su
Yuhang Lai/
Description: Pipeline for retrieval-augmented code generation
code generation (42) retrieval (37)
Recently the retrieval-augmented generation (RAG) paradigm has raised much attention for its potential in incorporating external knowledge into large language models (LLMs) without further training. While widely explored in natural language applications, its utilization in code generation remains under-explored. In this paper, we introduce Active Retrieval in Knowledge Soup (ARKS), an advanced strategy for generalizing large language models for code. In contrast to relying on a single source, we construct a
Diverse resources in general help LLM generalization.
ChatGPT and CodeLlama execution accuracy with different knowledge sources. Tensor-M refers to Tensorflow-M and Avg. refers to the average score across four benchmarks. Web denotes the web search content; Exec denotes the execution feedback from compiler/interpreter; Code denotes the code snippets generated by LLMs in previous rounds that are verified to be free of syntax error; Doc refers to the documentation. Adding more knowledge sources consistently enhances the performance, which demonstrates the advant
XLANG Lab
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Hongjin Su
Yuhang Lai/