(T1330-05-32) DrugChat: An Intelligent AI Agent for Herb-Drug Interaction Queries

Purpose: In recent years, the rapid advancement of large language models (LLMs) has allowed to achieve expert-level performance in semantic understanding and text generation as demonstrated by ChatGPT and DeepSeek ^[1]. However, due to high training costs and limited data sources, LLMs are restricted to training data available at their cutoff date. To enable LLMs to dynamically acquire additional knowledge during inference, Model Content Protocol (MCP) has recently been developed to facilitate seamless integration between LLMs and various external data sources. Such integration empowers LLMs to generate more accurate and relevant responses. Our recently developed Probot is the most comprehensive herb-drug interaction (HDI) database available, covering over 300 commonly used medicinal herbs and being updated in real-time. The current study was proposed aiming to develop an AI agent that utilizes MCP to connect LLMs with external sources, including the Probot and web content from the internet, to enable natural language search and the summarization of results.

Methods: The AI agent comprised two key components including LLM and MCP tools. The LLM is responsible for reasoning, decision-making, and generating text, while the MCP tools serve as bridges that allow the LLM to access external databases, such as Probot and the internet. The Probot MCP tool were developed to enable searches within Probot and the Tavily MCP tool was employed for web search. Both MCP tools support bilingual searching in Chinese and English. In addition to these core components, we adopted a step-by-step approach in designing the System Prompt of LLM, including 1) identification of the user's intent; 2) strategy for invoking MCP tools; 3) optimization of LLM output formatting. To improve the robustness of invoking the Probot MCP tool, we configured the System Prompt to generate additional parameters if no information is returned on the first attempt. This was achieved by either i) removing the dosage form from the drug name and searching using only the active ingredient; or ii) searching using an alternative language or synonym for the drug name. To select the most suitable LLM, we tested five popular LLMs including gpt-4o, Qwen-Plus, QwQ-Plus, DeepSeek-V3, and DeepSeek-r1. Their performance were evaluated and compared based on their responses to the following three questions with different levels of complexity: Q1: Can I take Gegen with Danshen? Q2: Please research and summarize the interactions of Danshen with two drugs: aspirin and fluorouracil. Q3: Please search the internet to identify three commonly used cardiovascular drugs. Then, use Probot to investigate their interactions with Danshen to determine if these drugs can be safely used together.

Results: As illustrated in the DrugChat diagram (upper of Figure 1), our developed agent could process both text and voice inquiries regarding HDI. Our Probot MCP tool, developed in TypeScript, has been open-sourced and is available for download at @kinginsun/mcp-probot-hdi. Based on performance evaluations in answering Questions Q1 to Q3 shown in Table 1, gpt-4o was selected as the final LLM for the DrugChat Agent, whereas DeepSeek-v3 and DeepSeek-r1 were excluded from evaluation as they did not support tool calls necessary for these questions. Although Qwen-Plus performed well on the straightforward Q1 and moderately complex Q2 questions, it failed on the complex Q3 due to its need for additional tool integration. Similarly, QwQ-Plus showed comparable overall performance to Qwen-Plus but struggled with adhering to the System Prompt instructions regarding output formatting. The final DrugChat Agent has been implemented using the open-source LibreChat platform (bottom of Figure 1) and deployed on an Ubuntu 22.04 server.

Conclusion: We have successfully developed an advanced AI agent named DrugChat, which leverages the capabilities of the gpt-4o LLM as its core intelligence to enable dynamically access and retrieve HDI information. Our current work has demonstrated how LLMs can be effectively integrated with professional databases to enhance the functionality and accuracy of AI-driven solutions.

References: [1] Wang, X., Liu, Y., Lin, Z., Xu, Z., Fu, Y., Zhang, Z., Liu, J., & Xu, C. (2023). The Rise and Potential of Large Language Model Based Agents: A Survey. arXiv preprint arXiv:2309.07864.

Acknowledgements: This project has received financial support from Hong Kong SAR's Innovation and Technology Fund (ITP/005/23LP) and the Chinese Medicine Development Fund (23B2/014A).


Figure 1. Diagram of Processing User Inquiries (Upper) and User Interface (Bottom) of DrugChat


Table 1. Evaluation and Comparison of Responses to Questions Q1 to Q3 Across the Selected LLMs