The development of Large Language Models (LLMs) has led to the creation of agentic systems that can perform sophisticated tasks independently through function calls. However, most research has focused on cloud-based infrastructure, leaving a gap in using these models locally on devices like laptops and smartphones.
To fill this gap, researchers at UC Berkeley and ICSI introduced the TinyAgent framework, which trains and deploys task-specific little language model agents. These agents can operate independently on local devices without relying on cloud- infrastructure.
The TinyAgent framework starts by modifying open-source models to execute function calls using the LLMCompiler framework. This involves fine-tuning the models to ensure consistent command execution and curating a high-quality dataset for function-calling tasks.
TinyAgent generates two variants, TinyAgent-1.1B and TinyAgent-7B, which are much smaller than larger equivalents but highly precise at handling specific jobs.
A key contribution of the TinyAgent framework is its unique tool retrieval technique, which helps shorten input prompts during inference. This allows the model to choose the right tool or function quickly and effectively.
To further improve performance, TinyAgent uses quantization to shrink the size and complexity of the model. These optimizations are crucial for ensuring compact models can function properly on local devices with constrained computational resources.
The TinyAgent framework has been deployed as a local Siri-like system for MacBooks, showcasing its real-world applications. This localized deployment allows users to comprehend orders from text or voice input and perform actions like starting apps, creating reminders, and doing information searches without requiring cloud access.
In conclusion, the TinyAgent framework offers an effective method for enabling edge devices to harness the potential of LLM-driven agentic systems while retaining strong performance in real-time applications.
Source: https://www.marktechpost.com/2024/09/19/tinyagent-an-end-to-end-ai-framework-for-training-and-deploying-task-specific-small-language-model-agents/