Netisopakul, Ponrudee and Taoto, Usanisa (2023) Comparison of Evaluation Metrics for Short Story Generation IEEE Access.. (In Press)
Official URL: https://ieeexplore.ieee.org/document/10329351
This study aimed to analyze the correlation among different automatic evaluation metrics for short story generation. In the study, texts were generated from short stories using different language models: the N-gram model, the Continuous Bag-of-Word (CBOW) model, the Gated recurrent unit (GRU) model and the Generative Pre-trained Transformer 2 (GPT-2) model. All models were trained on short Aesop’s fables. The quality of the generated text was measured with various metrics: Perplexity, BLEU score, the number of grammatical errors, Self-BLEU score, ROUGE score, BERTScore, and Word Mover’s Distance (WMD). The resulting correlation analysis of the evaluation metrics revealed four groups of correlated metrics. Firstly, perplexity and grammatical errors were moderately correlated. Secondly, BLEU, ROUGE and BERTScore were highly correlated. Next, WMD was negatively correlated with BLEU, ROUGE and BERTScore. On the other hand, Self-BLEU, which measures text diversity within the model, did not correlate with the other metrics. In conclusion, to evaluate text generation, a combination of various metrics should be used to measure different aspects of the generated text.
Item Type:
Article
Identification Number (DOI):
Subjects:
Subjects > Computer Science > Artificial Intelligence
Subjects > Computer Science > Computation and Language (Computational Linguistics and Natural Language and Speech Processing)
Subjects > Computer Science > Computer Science and Game Theory
Subjects > Mathematics > Representation Theory
Subjects > Statistics > Applications
Subjects > Statistics > Computation
Subjects > Statistics > Methodology
Subjects > Statistics > Machine Learning
Deposited by:
Ponrudee Netisopakul
Date Deposited:
2023-11-28 18:43:08
Last Modified:
2023-12-18 22:56:30