Get started with the EvalKit evaluations framework.
EvalKit exports multiple NPM packages.
We’ll concentrate on the evaluations framework package, which is available at:
Currently, EvalKit is using OpenAI API in order to perform evaluations.
Therefore, you’ll need to set up an OpenAI account and get an API key. Or configure it otherwise
Simply set your OpenAI API key in your environment:
Or if you prefer to set it programmatically:
For custom OpenAI endpoints (like Azure OpenAI), use the config:
The configuration is environment-agnostic - use any environment variables or configuration source you prefer. Required fields for custom endpoints:
apiKey
baseURL
deploymentName
(for Azure OpenAI)You can also configure EvalKit using an evalkit.config.ts
(or .js
) file in your project root:
The config file supports:
Same as the programmatic configuration above.
outputFormats
: Array of report formats to generate
[]
: Console output only (default)['json']
: JSON reports['html']
: HTML reports['json', 'html']
: Both formatsoutputDir
: Directory where reports will be saved (default: ’./eval-reports’)The EvalKit evaluations framework provides a simple API to evaluate text using pre-trained transformer models. Here’s a quick example to get you started:
Get started with the EvalKit evaluations framework.
EvalKit exports multiple NPM packages.
We’ll concentrate on the evaluations framework package, which is available at:
Currently, EvalKit is using OpenAI API in order to perform evaluations.
Therefore, you’ll need to set up an OpenAI account and get an API key. Or configure it otherwise
Simply set your OpenAI API key in your environment:
Or if you prefer to set it programmatically:
For custom OpenAI endpoints (like Azure OpenAI), use the config:
The configuration is environment-agnostic - use any environment variables or configuration source you prefer. Required fields for custom endpoints:
apiKey
baseURL
deploymentName
(for Azure OpenAI)You can also configure EvalKit using an evalkit.config.ts
(or .js
) file in your project root:
The config file supports:
Same as the programmatic configuration above.
outputFormats
: Array of report formats to generate
[]
: Console output only (default)['json']
: JSON reports['html']
: HTML reports['json', 'html']
: Both formatsoutputDir
: Directory where reports will be saved (default: ’./eval-reports’)The EvalKit evaluations framework provides a simple API to evaluate text using pre-trained transformer models. Here’s a quick example to get you started: