WTU-Eval: A New Standard Benchmark Tool for Evaluating Large Language Models LLMs Usage Capabilities

Large Language Models (LLMs) excel in various tasks, including text generation, translation, and summarization. However, a growing challenge within NLP is how these models can effectively interact with external tools to perform tasks beyond their inherent capabilities. This challenge is particularly relevant in real-world applications where LLMs must fetch real-time data, perform complex calculations, or interact with APIs to complete tasks accurately.

One major issue is LLMs’ decision-making process regarding when to use external tools. In real-world scenarios, it is often unclear whether a tool is necessary. Incorrect or unnecessary tool usage can lead to significant errors and inefficiencies. Therefore, the core problem recent research addresses is enhancing LLMs’ ability to discern their capability boundaries and make accurate decisions about tool usage. This improvement is crucial for maintaining LLMs’ performance and reliability in practical applications.

Traditionally, methods to improve LLMs’ tool usage have focused on fine-tuning models for specific tasks where tool use is mandatory. Techniques such as reinforcement learning & decision trees have shown promise, particularly in mathematical reasoning and web searches. Benchmarks like APIBench and ToolBench have been developed to evaluate LLMs’ proficiency with APIs and real-world tools. However, these benchmarks typically assume that tool usage is always required, which does not reflect the uncertainty and variability encountered in real-world scenarios.

Researchers from Beijing Jiaotong University, Fuzhou University, and the Institute of Automation CAS introduced the Whether-or-not tool usage Evaluation benchmark (WTU-Eval) to address this gap. This benchmark is designed to assess the decision-making flexibility of LLMs regarding tool usage. WTU-Eval comprises eleven datasets, six of which explicitly require tool usage, while the remaining five are general datasets that can be solved without tools. This structure allows for a comprehensive evaluation of whether LLMs can discern when tool usage is necessary. The benchmark includes tasks such as machine translation, math reasoning, and real-time web searches, providing a robust framework for assessment.

The research team also developed a fine-tuning dataset of 4000 instances derived from WTU-Eval’s training sets. This dataset is designed to improve the decision-making capabilities of LLMs regarding tool usage. By fine-tuning the models with this dataset, the researchers aimed to enhance the accuracy and efficiency of LLMs in recognizing when to use tools and effectively integrating tool outputs into their responses.

The evaluation of eight prominent LLMs using WTU-Eval revealed several key findings. Firstly, most models need help determining tool use in general datasets. For example, the performance of Llama2-13B dropped to 0% on some tool questions in zero-shot settings, highlighting the difficulty LLMs face in these scenarios. However, the models improved performance in tool-usage datasets when their abilities aligned more closely with models like ChatGPT. Fine-tuning the Llama2-7B model led to a 14% average performance improvement and a 16.8% decrease in incorrect tool usage. This enhancement was particularly notable in datasets requiring real-time information retrieval and mathematical calculations.

Further analysis showed that different tools had varying impacts on LLM performance. For instance, simpler tools like translators were managed more efficiently by LLMs, while complex tools like calculators and search engines presented greater challenges. In zero-shot settings, the proficiency of LLMs decreased significantly with the complexity of the tools. For example, Llama2-7B’s performance dropped to 0% when using complex tools in certain datasets, while ChatGPT showed significant improvements of up to 25% in tasks like GSM8K when tools were used appropriately.

The WTU-Eval benchmark’s rigorous evaluation process provides valuable insights into LLMs’ tool usage limitations and potential improvements. The benchmark’s design, which includes a mix of tool usage and general datasets, allows for a detailed assessment of models’ decision-making capabilities. The fine-tuning dataset’s success in improving performance underscores the importance of targeted training to enhance LLMs’ tool usage decisions.

In conclusion, the research highlights the critical need for LLMs to develop better decision-making capabilities regarding tool usage. The WTU-Eval benchmark offers a comprehensive framework for assessing these capabilities, revealing that while fine-tuning can significantly improve performance, many models still struggle to determine their capability boundaries accurately. Future work should focus on expanding the benchmark with more datasets and tools and exploring different LLM types further to enhance their practical applications in diverse real-world scenarios. 

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter..

Don’t Forget to join our 46k+ ML SubReddit

Find Upcoming AI Webinars here

Asjad is an intern consultant at Marktechpost. He is persuing B.Tech in mechanical engineering at the Indian Institute of Technology, Kharagpur. Asjad is a Machine learning and deep learning enthusiast who is always researching the applications of machine learning in healthcare.