Subnet 09
Pre Training
Macrocosmos
Macrocosmos’s subnet focuses on pre-training large language models using extensive datasets and continuous validation
SN9 : Pre-Training
| Subnet | Description | Category | Company |
|---|---|---|---|
| SN9 : Pre-Training | Training competition | Decentralized Training | Macrocosmos |
Developed by Macrocosmos and with support from Tensorplex Labs, it focuses on pretraining large language models. Miners train models on the Falcon Refined Web dataset and improve model performance through continuous benchmarking and validation mechanisms.
The pre-training subnet (SN9) lies at the core of Bittensor’s vision. Pre-training marks the pivotal initial phase in developing modern AI models, consuming vast amounts of data and computational resources to instill a foundational understanding of the world. It often stands as the most resource-intensive stage, with costs reaching millions, if not tens of millions, of dollars for large-scale models. Moreover, the quality of model pre-training significantly influences both the performance and security of subsequent applications.
In the realm of neural networks, pre-training typically entails training a model on a vast dataset before refining it on a smaller, task-specific dataset. This methodology harnesses transfer learning to enhance model efficacy and decrease training duration. A Bittensor subnet specialized in pre-training acts as a hub for training models on expansive generic datasets before further refining them in other subnets tailored to specific tasks.
Macrocosmos aims to elevate the creation of subnets, emphasizing a focus on crafting incentives and mechanisms for the Bittensor network. The pre-training subnet is distributed among highly skilled teams competing to enhance models continuously while receiving compensation for their efforts. Consequently, SN9 produces world-class large language models (LLMs) that embrace openness at every level: open source, open data, open models, open competition—open everything. Its track record speaks volumes; it has surpassed major industry players by creating models that outperform on a pound-for-pound basis.
In this subnet, miners and validators join forces to furnish the essential computational resources and authenticate the pre-training procedure, guaranteeing that the models undergo thorough training and are primed for subsequent fine-tuning.
Bittensor subnet 9 offers rewards to miners (including engineers) who develop pretrained Foundation-Models using the Falcon Refined Web dataset. It functions as an ongoing benchmark where miners receive rewards for achieving optimal losses on randomly selected pages of Falcon while maintaining a consistent model architecture. The reward system operates as follows:
The chain consolidates weights from all active validators and employs Yuma Consensus to determine the distribution of TAO emission rewards to miners and validators.
Miners train and periodically host trained model weights associated with their miner key, as demonstrated by the code in neurons/miner.py.
Validators continuously evaluate the hosted models, executing the validation system outlined in neurons/validator.py, and assign weights to the chain based on each miner’s performance on the Falcon dataset.
Will Squires – CEO and Co-Founder
Will has dedicated his career to navigating complexity, spanning from designing and constructing significant infrastructure to spearheading the establishment of an AI accelerator. With a background in engineering, he made notable contributions to transport projects such as Crossrail and HS2. Will’s expertise led to an invitation to serve on the Mayor of London’s infrastructure advisory panel and to lecture at UCL’s Centre for Advanced Spatial Analysis (CASA). He was appointed by AtkinsRéalis to develop an AI accelerator, which expanded to encompass over 60 staff members globally. At XYZ Reality, a company specializing in augmented reality headsets, Will played a pivotal role in product and software development, focusing on holographic technology. Since 2023, Will has provided advisory services for the Opentensor Foundation, contributing to the launch of Revolution.
Steffen Cruz – CTO and Co-Founder
Steffen earned his PhD in subatomic physics from the University of British Columbia, Canada, focusing on developing software to enhance the detection of extremely rare events (10^-7). His groundbreaking research contributed to the identification of novel exotic states of nuclear matter and has been published in prestigious scientific journals. As the founding engineer of SolidState AI, he pioneered innovative techniques for physics-informed machine learning (PIML). Steffen was subsequently appointed as the Chief Technology Officer of the Opentensor Foundation, where he played a pivotal role as a core developer of Subnet 1, the foundation’s flagship subnet. In this capacity, he enhanced the adoption and accessibility of Bittensor by authoring technical documentation, tutorials, and collaborating on the development of the subnet template.
Pedro Ferreira – Machine Learning Engineer
Kalei Brady – Data Scientist
Sergio Champoux – Data Scientist
Brian McCrindle – Machine Learning Researcher
Elena Nesterova – Lead Technical Program Manager
Richard Hudson – Communications Lead
Alex Williams – Recruitment Lead