A standard evaluation bench for NLG evaluations

Table of Contents

1. Downstream Datasets

Card Name selected SeqLen Comments
wikisql 512  
spider 512  
allenai/commongen 256  
e2enlg 512  
UCL-DARK/openai-tldr-filtered 2048 filter
cnndailymail 4096 filter
samsum 2048 filter
piqa 256  
truthfulqa 256  
allenai/ai2arc 256  

1.1. Downstream Tasks: Status

1.1.1. Translation

  1. REVIEW WMT16: cs-en
  2. REVIEW WMT16: de-en
  3. REVIEW WMT16: fi-en

1.1.2. Summarization

  1. TODO TLDR
  2. TODO cnndailymail ? too long
  3. TODO samsum

1.1.3. Structure Text generation

  1. TODO WikiSQL
  2. TODO Spider

1.1.4. Data-to-Text

  1. TODO e2enlg
  2. TODO CommonGen

1.1.5. Question Answering

  1. TODO piqa
  2. TODO truthfulqa
  3. TODO allenai/ai2arc

1.2. Generalized Ability: Status

1.2.1. Safety

  1. TODO HELM: bias, toxicity, fairness
  2. TODO …

1.2.2. Reasoning (math)

  1. TODO MMLU (59 subsets)
Author: Zi Liang (zi1415926.liang@connect.polyu.hk) Create Date: Tue Mar 26 11:08:33 2024 Last modified: 2026-05-12 Tue 20:20 Creator: Emacs 30.2 (Org mode 9.7.11)