An intelligent agent framework that performs a wide range of complex computer tasks.
It is capable of self-learning and improvement, handling various operating system-level tasks.
Including but not limited to file management, data processing, environment settings, multimedia operations, web browsing, code writing, third-party application interaction, automated testing, etc.
The project was jointly developed by researchers from the Shanghai AI Laboratory, East China Normal University, Princeton University and the University of Hong Kong.
OS-Copilot provides a common interface for Linux and MacOS, integrating common operating system manipulation practices such as Python code interpreters, bash terminals, mouse/keyboard controls, and API calls.
An instance or concrete agent created by researchers based on the OS-Copilot framework: FRIDAY agent
FRIDAY can learn from pictures, videos or text, and can perform a range of computer tasks, such as drawing in Excel or creating a website. Most importantly, FRIDAY is able to learn new skills by doing tasks, just as humans become better through constant trial and practice.
The main functions of FRIDAY agent:
1. Self-learning and improvement: FRIDAY can continuously accumulate new skills and knowledge through self-guided learning, enabling it to handle more unknown tasks and applications. This includes learning how to use software applications more effectively, best practices for performing specific tasks, etc.
2. Extensive task execution: FRIDAY is designed as a universal agent capable of performing a variety of computer tasks, including data analysis, file management, automated programming tasks, multimedia editing, web browsing and information retrieval.
3. Operating system-level interaction: FRIDAY can interact with various elements within the operating system, such as file systems, terminals, third-party applications, etc., to control and manage these elements.
4. Integration and control of third-party applications: Using the OS-Copilot framework, FRIDAY can integrate and control with a variety of third-party applications, such as office software (Microsoft Office, Google Docs), programming environments (IDEs), social media platforms, etc.
5. Automate complex workflows: FRIDAY can automate complex workflows, such as the entire process from performing data collection and processing to report generation, greatly improving work efficiency.
6. Creation of custom tasks and workflows: Users can customize FRIDAY agents to perform specific tasks and workflows, and agents can learn these custom operations and improve execution efficiency over time.
7. A high degree of adaptability and flexibility: Through its self-learning ability, FRIDAY can adapt to various new scenarios and needs, and flexibly handle various computer tasks.
8. Improve productivity and efficiency: By automating daily tasks and complex workflows, FRIDAY aims to significantly improve user productivity and work efficiency.
9. Through these functions, FRIDAY agent demonstrates the powerful potential of the OS-Copilot framework and provides users with a highly flexible, self-learning and improvement universal computer agent, aiming to handle various computer-related tasks, thereby liberating users ‘hands and improving work efficiency.
The FRIDAY agent can work with the operating system to:
- Automate daily tasks: such as setting up the working environment, automatically organizing files, backing up data, etc.
- Data processing and analysis: Automate calculations, generate charts, etc. in applications like Excel.
- Multimedia content creation: For example, automatically creating presentations or editing videos in PowerPoint.
- Web browsing and information retrieval: Automate web searches to quickly summarize and collect specific information.
- Programming and script execution: Automatically generate code snippets or execute specific scripts to complete tasks.
- Third-party application interaction: Interaction with third-party applications such as mail clients, calendars, and social media to perform tasks such as sending emails and updating events.
- Self-learning and improvement: Learn new skills and applications through self-guidance to continuously improve the efficiency and accuracy of tasks.
Projects and demonstrations:https://os-copilot.github.io
Thesis:https://arxiv.org/abs/2402.07456
GitHub：https://github.com/OS-Copilot/FR
Video: