Bittensor Subnet 15: De_Val

De-Val is a groundbreaking decentralized evaluation subnet for large language models (LLMs). Leveraging the robust Bittensor network, De-Val sets a new standard in evaluating LLM outputs with unparalleled objectivity and reliability. They address key challenges such as hallucinations, misattributions, relevancy, and summary completeness.

Thier innovative approach begins with using synthetic data for initial training. Looking ahead, they plan to offer comprehensive evaluation and monitoring capabilities for businesses aiming to refine their LLM-based solutions. This includes tools for A/B testing and offline validation. Specifically, they focus on evaluation within Retrieval-Augmented Generation (RAG) scenarios to help businesses answer critical questions such as:

• How accurate is my LLM given the provided context?
• How comprehensive are our abstractive summaries? Are we missing crucial details?
• Can we accurately identify key users and align their names with relevant action items?

In thier initial rollout, the subnet will feature evaluators dedicated to:

• Hallucination Detection: Spotting false or nonsensical outputs within LLM summaries in the context of RAG.
• Misattribution Correction: Assessing the accuracy of participant attribution in LLM summaries from sales calls, meetings, and general chat messages.
• Summary Completeness: Analyzing the thoroughness and reliability of generated summaries.
• Relevancy: Providing binary predictions on whether a response is pertinent to a user’s query.