Data connectivity & integrationPipeline Builder ExpressionsROUGE score

ROUGE score

Supported in: Batch

Computes ROUGE precision, recall, and F-measure for predicted and target text. Text is lowercased; only ASCII letters and digits are compared. All other characters are treated as word separators. Returns null if either input is null or the text cannot be scored, including when it exceeds the scorer's size limits.

Expression categories: Distance measurement, String

Declared arguments

Prediction: The machine-generated text to evaluate.
Expression<String>
ROUGE type: The ROUGE variant used to compare the texts.
Enum<ROUGE-1, ROUGE-2, ROUGE-3, ROUGE-L, ROUGE-Lsum, ROUGE-Lsum (detected sentences)>
Target: The reference text to compare against.
Expression<String>
optional Use stemmer: Stem words before comparing them.
Literal<Boolean>

Output type: Struct<precision, recall, fmeasure>

Examples

Example 1: Base case

Description: Only ASCII letters and digits are compared after lowercasing; other characters separate words. Text without any tokens scores zero, even when the inputs are identical.

Argument values:

Prediction: prediction
ROUGE type: rouge1
Target: target
Use stemmer: null

prediction	target	Output
你好	你好	{ fmeasure: 0.0, precision: 0.0, recall: 0.0, }
cat你好dog	cat dog	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }
CAT 42	cat 42	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }
empty string	empty string	{ fmeasure: 0.0, precision: 0.0, recall: 0.0, }

Example 2: Base case

Description: ROUGE-2 measures overlap between adjacent pairs of words.

Argument values:

Prediction: prediction
ROUGE type: rouge2
Target: target
Use stemmer: null

prediction	target	Output
the cat sat	the cat slept	{ fmeasure: 0.5, precision: 0.5, recall: 0.5, }

Example 3: Base case

Argument values:

Prediction: prediction
ROUGE type: rouge1
Target: target
Use stemmer: null

prediction	target	Output
the cat sat	the cat sat	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }

Example 4: Base case

Description: ROUGE-L measures the longest sequence of words appearing in the same order.

Argument values:

Prediction: prediction
ROUGE type: rouge_l
Target: target
Use stemmer: null

prediction	target	Output
cat sat	sat cat	{ fmeasure: 0.5, precision: 0.5, recall: 0.5, }

Example 5: Base case

Description: ROUGE-Lsum uses newlines to separate sentences; punctuation alone does not split them.

Argument values:

Prediction: prediction
ROUGE type: rouge_lsum
Target: target
Use stemmer: null

prediction	target	Output
cat sat. dog ran.	dog ran. cat sat.	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }
cat sat. dog ran.	dog ran. cat sat.	{ fmeasure: 0.5, precision: 0.5, recall: 0.5, }

Example 6: Base case

Description: ROUGE-1 measures overlap between individual words.

Argument values:

Prediction: prediction
ROUGE type: rouge1
Target: target
Use stemmer: null

prediction	target	Output
the cat sat	the cat slept	{ fmeasure: 0.6666666666666666, precision: 0.6666666666666666, recall: 0.6666666666666666, }

Example 7: Base case

Description: Stemming allows related word forms to match.

Argument values:

Prediction: prediction
ROUGE type: rouge1
Target: target
Use stemmer: true

prediction	target	Output
cars	car	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }

Example 8: Base case

Description: ROUGE-Lsum sentences automatically detects sentence boundaries and compares sentence-level matches without penalizing sentence reordering.

Argument values:

Prediction: prediction
ROUGE type: rouge_lsum_sentences
Target: target
Use stemmer: null

prediction	target	Output
The dog ran. The cat sat.	The cat sat. The dog ran.	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }
Cat sat. Dog ran.	Dog ran. Cat sat.	{ fmeasure: 1.0, precision: 1.0, recall: 1.0, }

Example 9: Base case

Description: ROUGE-3 measures overlap between adjacent groups of three words.

Argument values:

Prediction: prediction
ROUGE type: rouge3
Target: target
Use stemmer: null

prediction	target	Output
the cat sat down	the cat sat still	{ fmeasure: 0.5, precision: 0.5, recall: 0.5, }
the cat	the cat	{ fmeasure: 0.0, precision: 0.0, recall: 0.0, }

Example 10: Null case

Description: If either input is null, the entire ROUGE score output is null.

Argument values:

Prediction: prediction
ROUGE type: rouge1
Target: target
Use stemmer: null

prediction	target	Output
null	target	null
prediction	null	null
null	null	null