MindSearch: An open source Web search engine framework

By /

MindSearch: An open source Web search engine framework that leverages large language models (LLM) and multi-agent architecture to provide advanced search capabilities

Name: MindSearch
Introduction: An open source Web search engine framework that leverages large language models (LLM) and multi-agent architecture to provide advanced search capabilities. The project supports multiple search engines such as DuckDuckGo, Bing, Brave Google, and you can easily deploy it to build your own search engine:

MindSearch is InternLM Team open source AI search engine framework, its goal is Simulate the human thought processImprove the intelligence level of search. it uses Multi-step reasoning and hierarchical retrieval method to obtain depth information from multiple perspectives, similar to Perplexity.ai Pro, but supports open source and customizable deployment.

1. main features

🔍 Complex query decomposition

  • WebPlanner assembly can Break down complex problems into multiple sub-problems, and then search in parallel, thereby increasing the depth and breadth of the search.

📂 hierarchical search

  • WebSearcher component Responsible for performing hierarchical searches, including:
    • Basic information acquisition
    • Advanced in-depth analysis
  • This approach makes search results more accurate and avoids one-sidedness of information brought about by a single search.

🤖 Multiple LLM (Large Language Model) support

  • support GPT (e.g. GPT-4), Claude
  • InternLM2.5 Series (optimized open source model)
  • Users can freely choose and integrate the LLM model that suits them.

🧐 of interpretability

  • MindSearch Will show search keywords and reasoning processLet users understand how AI reaches conclusions and enhance transparency.

🌐 Multiple interaction methods

  • provide Web front-end interface, can be based on React, Gradio and Streamlit interact.
  • can also APIs are integrated directly into your own applications medium.

2. technical architecture

The overall architecture of MindSearch is as follows:

  • ˇWebPlanner (planner)：
    • Responsible for parsing user issues, breaking them into multiple sub-tasks, and generating a search plan.
  • 🔍WebSearcher (search engine)：
    • Find the most relevant information through a multi-level search method.
  • LLM(Big Language Model)：
    • Responsible for integrating the searched information, summarizing, reasoning, and generating final answers.

3. usage scenarios

  • 🔎Intelligent search engine(Similar to Perplexity.ai)
  • AI Research Assistant(Help researchers obtain high-quality relevant information)
  • 💬AI Question and Answer System(Can be integrated into customer service, education and other fields)
  • Knowledge management(Used for internal knowledge retrieval)

4. Code and installation

installation

The installation method of MindSearch is very simple, just:

git clone https://github.com/InternLM/MindSearch
cd MindSearch
pip install -r requirements.txt

Github：https://github.com/InternLM/MindSearch

5. future development

The InternLM team plans to further optimize:

  • 🎯Improve search accuracy and allow AI to better understand users ‘true intentions
  • Add plug-in mechanism to support more search sources, such as Google, Bing
  • Optimize large model reasoning and reduce computing resource consumption

🔎Summary

MindSearch is an efficient and extensible AI search engine framework suitable for various intelligent question and answer and information retrieval scenarios.
It combines WebPlanner, WebSearcher, and LLM to provide a Perplexity.ai It is fully open source and supports custom deployments.

📌GitHub address:https://github.com/InternLM/MindSearch

Oil tubing:

Scroll to Top