Petals

Petals is an open source system for running large language models across a community run network of volunteer GPUs, in the manner of BitTorrent. Instead of loading an entire model on one machine, a user's computer holds a small part of it and the remaining layers are served by other participants, so models far too large for a desktop can still generate text and be fine tuned. The project provides a Python library built on the Hugging Face Transformers interface. A developer loads a model with a distributed model class, then calls the usual generate method as if the model were local. The repository lists support for Llama 3.1 up to 405B parameters, Mixtral 8x22B, Falcon 40B and above, and BLOOM 176B. Fine tuning for custom tasks is supported, and the project states that fine tuning and inference run up to ten times faster than offloading. A Colab notebook, a hosted chatbot app and a health monitor showing available models are linked from the repository. People can also contribute by hosting model layers on their own GPU, using Linux with Anaconda, Windows through WSL, or Docker. Because data is processed with help from other participants in the public swarm, the project points users with sensitive data to a private swarm that can be launched among trusted parties. Some models, such as Llama, require requesting access to the weights first. Petals is released under the MIT license and is free to use. The repository publishes no paid plans.