Coherence

​

Overview

The Coherence metric assesses the logical flow and consistency of paragraphs in text generated by a model. It evaluates the coherence by analyzing the transitions between consecutive statements, determining if each subsequent statement logically follows from the one before it. A high coherence score indicates a well-structured text with clear, logical progressions.

CoherenceMetric utilizes the evaluateCoherence function to calculate this metric.

​

Methods

​

evaluateCoherence Function

This function evaluates the coherence of the generated text by assessing the logical flow between consecutive statements.

• output: The text generated by the model.

It examines each pair of consecutive statements to determine if the transition between them is logical. The function returns a promise that resolves to an object containing the coherence score and a brief reason explaining the score.

​

CoherenceMetric Class

CoherenceMetric leverages the evaluateCoherence function to determine the logical flow and consistency of the paragraph in the generated text.

• output: The text generated by the model.

The evaluateSteps method calls evaluateCoherence and returns a result that includes the coherence score and a reason for the evaluation. The method focuses on the clarity of transitions and the logical structuring of content.

​

Example

import { evaluate, CoherenceMetric } from '@evalkit/core';

evaluate({
    // The generated text from an LLM
    output: "First, ensure all data is encrypted. Subsequently, verify the integrity of the encryption algorithms.",
}, [CoherenceMetric])

// outputs
{
  passed: true,
  // The coherence score is 0.9, which is calculated based on the percentage of coherent statements from all statements in the text.
  score: 0.5,
  reasons: ['The text demonstrates a coherent logical flow.']
}