| I. IDENTIFYING INFORMATION |
|
| Title* |
DaLAJ v1.0
|
| Subtitle |
A proof-of-concept version of a DAtaset for Linguistic Acceptability Judgments, based on the correction-annotated version of Swedish learner essays (SweLL corpus)
|
| Created by* |
Elena Volodina & Yousuf Ali Mohammed (elena.volodina@svenska.gu.se)
|
| Publisher(s)* |
Språkbanken Text (sb-info@svenska.gu.se)
|
| Link(s) / permanent identifier(s)* |
https://spraakbanken.gu.se/en/resources/dalaj; https://spraakbanken.gu.se/en/resources/superlim
|
| License(s)* |
CC BY 4.0
|
| Abstract* |
DaLAJ 1.0 is a Dataset for Linguistic Acceptability Judgments for Swedish, comprising 9 596 sentences in its first version; The baseline for the dataset is 58% accuracy on a binary classification task (i.e. sentence either is correct or not) using BERT embeddings.
DaLAJ is based on the SweLL second language learner data (Volodina et al., 2019), consisting of essays at different levels of proficiency. To make sure the dataset can be freely available despite the GDPR regulations, we have sentence-scrambled learner essays and removed part of the metadata about learners, keeping for each sentence only information about the mother tongue and the level of the course where the essay has been written. We use the normalized version of learner language as the basis for the DaLAJ sentences, and keep only one error per sentence. We repeat the same sentence for each individual correction tag used in the sentence. For DaLAJ 1.0 we have used four error categories (out of 35 available in SweLL), all connected to lexical or word-building choices, namely errors with compounding, word choices, use of foreign words, use of wrong derivation pattern. Our baseline results for the binary classification show an accuracy of 58% for DaLAJ 1.0. The dataset is included in the SwedishGlue (Swe. SuperLim) benchmark. Description of the DaLAJ format, first experiments, our insights and the motivation for the chosen approach to data sharing are provided in https://arxiv.org/pdf/2105.06681.pdf.
|
| Funded by* |
This work has been supported by Nationella Språkbanken – jointly funded by its 10 partner institutions and the Swedish Research Council (dnr 2017-00626), as well as partly supported by a grant from the Swedish Riksbankens Jubileumsfond (SweLL - research infrastructure for Swedish as a second language, dnr IN16-0464:1).
|
| Cite as |
Consider citing [1,2]
|
| Related datasets |
Coming: full DaLAJ (v.2.0 or later)
|
|
|
| II. USAGE |
|
| Key applications |
Machine Learning, Neural networks; Linguistic Acceptability Judgments, Error detection; Error classification; First language identification; Classification by proficiency level; etc.
|
| Intended task(s)/usage(s) |
(1) Given a sentence, decide whether the sentence is correct or not; (2) Given a sentence (containing an error), find a string that is incorrect; (3) Given a sentence (containing an error), classify the error type; (4) Given a (group of?) sentence(s) (both correct and incorrect), identify the first language of the writer; (5) Given a (group of?) sentence(s) (both correct and incorrect), classify the level of proficiency of the writer (beginner, intermediate, advanced);
|
| Recommended evaluation measures |
(1) Accuracy (2) F0.5-score (3) F0.5-score
|
| Dataset function(s) |
Training, evaluation, testing
|
| Recommended split(s) |
80-10-10 (training-validation-texting) Recommended split of the sentences is provided in the data. Note that there are duplicates in the correct sentences.
|
|
|
| III. DATA |
|
| Primary data* |
Sentences
|
| Language* |
Swedish
|
| Dataset in numbers* |
4730 incorrect sentences; 4730 correct sentences (including duplicates); more statistics in [1,2]
|
| Nature of the content* |
see [1,2]
|
| Format* |
csv format; 10 columns:
(1) running number
(2) original sentence (that contains an error)
(3) correction sentence
(4) error indices, i.e. start character index - end character index
(5) correction indices, i.e. start character index - end character index
(6) error-correction pair (two strings)
(7) correction tag
(8) mother tongue of the writer
(9) approximate level of the writer - information derived from the course the learner has been taking when writing the essay
(10) split used for the experiment: train, valid, test
|
| Data source(s)* |
SweLL learner essays, see [3]
|
| Data collection method(s)* |
SweLL learner written essays are collected in the SweLL project, from the test situations (2017-2020) provided learners signed consents, and filled in demographic metadata. Teacheers filled in Task metadata sheets, and where appropriate, grades for the essays. All metadata is registered in the SweLL portal.
|
| Data selection and filtering* |
Essays were manually pseudonymized, normalized and correct-annotated, see [3]. The selection of essays for manual annotation was subject to balancing the essays, where possible, after mother tongues (10 most frequent in Sweden), gender balance, course level balance, prior education level balance (which in the end was impossible to achieve)
|
| Data preprocessing* |
Manual transcription, pseudonymizarion, normalization, correction annotation, see [3]
|
| Data labeling* |
see [3]
|
| Annotator characteristics |
Krippendorff’s alpha on correction annotation task is 0,85% on the basis of 10% of essays that have been double-annotated.
|
|
|
| IV. ETHICS AND CAVEATS |
|
| Ethical considerations |
SweLL dataset is subject to GDPR and Ethical Review Authority restrictions. The DaLAJ format allows to use the dataset openly, since (1) the demographic metadata is practically removed (apart from the information on mother tongue and level of the course), (2) the essays sentences are randomly scrambled and (3) all personal information in the sentences is pseudonymized (e.g. city names replaced with other city names)
|
| Things to watch out for |
|
|
|
| V. ABOUT DOCUMENTATION |
|
| Data last updated* |
2021-05-26, v1.0
|
| Which changes have been made, compared to the previous version* |
This is the first official version
|
| Access to previous versions |
|
| This document created* |
2021-05-27, Elena Volodina
|
| This document last updated* |
2021-06-19, Elena Volodina
|
| Where to look for further details |
[1], [2]
|
| Documentation template version* |
v1.0
|
|
|
| VI. OTHER |
|
| Related projects |
SweLL - research infrastructure for Swedish as a Second Language. https://spraakbanken.gu.se/en/projects/swell
|
| References |
[1] Volodina, Elena, Yousuf Ali Mohammed, and Julia Klezl. (2021). DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing. arXiv preprint arXiv:2105.06681.
https://arxiv.org/pdf/2105.06681.pdf
[2] Volodina, Elena, Yousuf Ali Mohammed, and Julia Klezl. (2021). DaLAJ - a dataset for linguistic acceptability judgments for Swedish. In Proceedings of the 10th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2021). Linköping Electronic Conference Proceedings 177:3, s. 28-37.
https://ep.liu.se/en/conference-issue.aspx?series=ecp&issue=177
https://ep.liu.se/ecp/177/003/ecp2021177003.pdf
[3] Elena Volodina, Lena Granstedt, Arild Matsson, Beáta Megyesi, Ildikó Pilán, Julia Prentice, Dan Rosén, Lisa Rudebeck, Carl-Johan Schenström, Gunlög Sundberg and Mats Wirén (2019). The SweLL Language Learner Corpus: From Design to Annotation. Northern European Journal of Language Technology, Special Issue.
https://nejlt.ep.liu.se/article/view/1374/1010
|