Sonia Cromp (BS in Computer Science, BA in Linguistics, Class of 2021) was awarded the best paper award at the Workshop for Undergraduates in Educational Data Mining and Learning Engineering at the 2021 Educational Data Mining Conference for her paper Essay Revision and Corresponding Grade Chang as Captured by Text Similarity and Revision Purposes. The paper was written in conjunction with Dr. Diane Litman (Professor, CS and LRDC.)
Writing and revision are abstract skills that can be challenging to teach to students. Automatic essay revision assistants offer to help in this area because they compare two drafts of a student’s essay and analyze the revisions performed. For these assistants to be useful, they need to provide useful information such as whether the revisions are likely to lead to an improvement in the student’s grade. It is necessary to better understand the connection between revisions and grade change so that this information could be displayed in an assistant. So, this work explores the relationship between the tf-idf cosine similarity of two essay drafts and the resulting essay grade change. Prior work has demonstrated that identifying the revisions between drafts, then labeling each revision with the purpose behind why the revision was performed is useful to predicting grade change. However, this process is expensive because this sort of annotation is time-consuming for humans. Moreover, classifiers achieve lower accuracy than humans when predicting purposes. Using similarity measures instead of or as a supplement to revision purposes may correct these issues, as similarity can be computed automatically and without the issue of classification accuracy. As such, the correlations between grade change and the similarity measure are compared to the correlations between grade change and revision purposes with the potential use-case of an automatic writing assistant in mind. Findings suggest tf-idf cosine similarity captures the overall essay and overall grade change while revision purposes capture lighter changes that fix errors or cause the essay to read better.