237  lm-eval-harness(1) - Kailash Tool Reference

237.1 lm-eval-harness

lm-eval-harness - EleutherAI Language Model Evaluation Harness — standard few-shot benchmark runner (lm-eval).

Layer DEFENCE
Category d-03-assurance-evals-benchmarks
Safety class safe
Licence MIT
Upstream github.com/EleutherAI/lm-evaluation-harness

237.1.1 NAME

lm-eval-harness - EleutherAI Language Model Evaluation Harness — standard few-shot benchmark runner (lm-eval).

237.1.2 SYNOPSIS & usage

Derived from the upstream README at generation time; verify against --help where the tool is installed.

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
pip install "lm_eval[hf]"

237.1.3 DESCRIPTION (upstream)

  • [2026/09] Plugins: model backends, filters, metrics and aggregations can now be registered from your own package instead of forking — declare an lm_eval.* entry point for zero-config discovery, or point --plugins at a local module. See the Plugin Guide. - [2025/12] CLI refactored with subcommands (run, ls, validate) and YAML config file support via --config. See the CLI Reference and Configuration Guide. - [2025/12] **Lighte

237.1.4 CONFIGURATION (detected)

Config-surface markers detected in the upstream README (5 matches, e.g. --config, environment variable). A full config reference page for this tool is queued under the config-page policy — this line is the detection record, not the page.

237.1.5 SEE ALSO

ref: pin - verified 2026-10-14