Researchers have trained a new large language model (LLM) called Collective-1, which uses GPUs and public data across the globe, potentially disrupting the dominant method of building artificial intelligence.
Flower AI and Vana, two startups, collaborated to create Collective-1. Flower’s technology allows training to be spread across hundreds of computers connected over the internet, while Vana provided private messages from platforms like X, Reddit, and Telegram.
Collective-1 is smaller than modern models, with 7 billion parameters compared to hundreds of billions in other LLMs. However, its creators believe this approach can scale far beyond Collective-1’s size. Flower AI plans to train larger models, including one with 100 billion parameters, later this year.
This new method of building AI could upset the current power dynamics, as companies rely on vast amounts of training data and compute resources concentrated in data centers. A distributed approach enables smaller companies and universities to build advanced AI by pooling disparate resources together or networking together multiple data centers.
Experts say this approach is interesting and potentially relevant to AI competition and governance. The distributed method “allows you to scale compute much more elegantly than the data center model,” according to computer scientist Nic Lane.
The new tool, Photon, developed in collaboration with researchers in China, improves upon Google’s approach by making distributed training more efficient. This process is slower but more flexible, allowing for new hardware to be added during training.
Vana’s software allows users to contribute private data to AI training and specify end-use permissions or financial benefits. Experts believe this unlocks new kinds of data and reduces the risks associated with data centralization in industries like healthcare and finance.
Source: https://www.wired.com/story/these-startups-are-building-advanced-ai-models-over-the-internet-with-untapped-data