Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis

Dateibereich 42141

221,3 KB in einer Datei, zuletzt geändert am 09.11.2016

Dateiliste / Details

DateiDateien geändert amGröße
reliability_hate_speech_annotations.pdf09.11.2016 07:30:06221,3 KB
Some users of social media are spreading racist, sexist, and otherwise hateful content. For the purpose of training a hate speech detection system, the reliability of the annotations is crucial, but there is no universally agreed-upon definition. We collected potentially hateful messages and asked two groups of internet users to determine whether they were hate speech or not, whether they should be banned or not and to rate their degree of offensiveness. One of the groups was shown a definition prior to completing the survey. We aimed to assess whether hate speech can be annotated reliably, and the extent to which existing definitions are in accordance with subjective ratings. Our results indicate that showing users a definition caused them to partially align their own opinion with the definition but did not improve reliability, which was very low overall. We conclude that the presence of hate speech should perhaps not be considered a binary yes-or-no decision, and raters need more detailed instructions for the annotation.
Veranstaltungen, Tagungen, Ereignisse » Tagungsbericht, Konferenzschrift, Ausstellung
Fakultät / Institut:
Fakultät für Ingenieurwissenschaften » Informatik und Angewandte Kognitionswissenschaft
Dewey Dezimal-Klassifikation:
000 Informatik, Informationswissenschaft, allgemeine Werke
hate speech, hate speech detection, twitter, social media, social media analytics, natural language processing
Kollektion / Status:
E-Publikationen / Dokument veröffentlicht
Dokument erstellt am:
Dateien geändert am:
Creative Commons Namensnennung - Keine Bearbeitungen 4.0 InternationalDieses Werk kann unter einer Creative Commons Namensnennung - Keine Bearbeitungen 4.0 International Lizenz genutzt werden.
Originally published in Bochumer Linguistische Arbeitsberichte 17, NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, by Michael BeiƟwenger, Michael Wojatzki and Torsten Zesch (Eds.), 22 September 2016 (ISSN 2190-0949).