Relevancy

​

Overview

The Relevancy metric assesses how well the answer generated by a model addresses a posed question. This metric focuses on the contextual appropriateness of the response, measuring whether the answer is pertinent and adequately responds to the question’s intent.

RelevancyMetric utilizes the evaluateRelevancy function to determine the relevance of the answer to the question.

​

Methods

​

evaluateRelevancy Function

This function evaluates the relevancy of the generated answer in response to a specific question.

• input: The question posed to the model.
• output: The generated answer by the model.

The function checks the contextual alignment between the question and the answer, returning a promise that resolves to a boolean value indicating whether the answer is relevant. The relevancy is quantified on a scale from 0 to 1, where scores close to 1 indicate high relevance.

​

RelevancyMetric Class

RelevancyMetric evaluates how well an answer addresses the posed question, focusing on its relevance.

• input: The question posed to the model.
• output: The generated answer by the model.

The evaluateSteps method calls evaluateRelevancy and provides a detailed result, including a score and reasons. The evaluation concludes with a score indicating high or low relevance based on the contextual appropriateness of the answer.

​

Example

import { evaluate, RelevancyMetric } from '@evalkit/core';

evaluate({
    // The question posed to the model
    input: "What are the benefits of renewable energy?",
    // The answer generated by the model
    output: "Renewable energy sources like solar and wind power reduce greenhouse gas emissions and help combat climate change.",
}, [RelevancyMetric])

// outputs
{
  passed: true,
  score: 1,
  reasons: ['The answer is relevant to the question.']
}