nilsreiter.de

Large Language Models and Peer Review (Part 2)

2025-07-14T00:00:00+00:00

In the past weeks, I had the opportunity to give two talks on working scientifically (in Bielefeld on Open Science and Cologne on Reproducibility). Naturally, large language models (LLMs) and the influence they have came up as a topic in both. Because I did quite a bit of research, I thought it would be good to share the key insights in this form as well. This is the second of a (short) series of posts (this is the first).

There are two interesting findings related to peer review and LLMs that I’d like to highlight here:

(Some) Peer Reviewers are using LLMs to do their Reviews

The question of the extent to which peer reviewers are using LLMs for their reviews has been investigated by Liang et al. (2024). Their method for determining LLM use a trained maximum likelihood-based model that is validated using a corpus of reviews written before the release of ChatGPT. The method predicts a parameter ɑ (alpha) that essentially denotes the portion of sentences in a corpus that have been “substantially” modified or written by an LLM. They investigate four different machine learning/NLP conferences (ICLR 2024, NeurIPS 2023, CoRL 2023, EMNLP 2023) and a number of journal reviews from Nature.

Their findings are summarized in their Figure 4, shown on the left. According to this, between 6.5 and 16.9 % of the sentences in the reviews to the conferences are in a substantial way influenced by an LLM. Nature reviews in the corpus, according to this, are not impacted by this.

They also investigate a number of influential factors that make LLM use more likely (which intuitively make a lot of sense):

Deadline: Reviews submitted closer to the deadline have a higher predicted ɑ.
References: Reviews that contain the string “et al.” (which indicates that a reviewer suggests additional literature) have a lower predicted ɑ.
Replies: The conference review schemes contain discussion periods in which authors can reply to the comments by the reviewers, to which reviewers again can react. There is a negative correlation between number of replies a reviewer produces and the estimated ɑ – LLM use correlates with reduced activity in these interactions.
Homogenization: Reviews that are similar to the other reviews on the same paper tend to have a higher predicted ɑ, i.e., a larger portion of AI-generated text.

It is noteworthy that the Nature-reviews in the data set are unaffected by this (at least according to this measurement method at this time).

(Some) People Work on Optimizing LLMs for Peer Reviewing

The second interesting find is this paper. It’s called “OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews” and has been published at the NAACL in 2025. The title is telling: It is an LLM, fine-tuned on 79k “expert reviews from top conferences”.

The authors claim that the generated reviews are better (i.e., more critical) than those of GPT 4 and Claude 3.5, and “closely match the distribution of human reviewer ratings”. This was tested on 400 held-out papers from the most recent two conference iterations: In 55.5 % of those, the OpenReviewer system’s numeric review score matched exactly with at least one human reviewer, compared to 23.8 % for the next best model, GPT-4o.

From a certain perspective, this is quite impressive, and a cool new application. So cool, that the authors actually decided to create an online demo of the system, which can be found here.

The system […] is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

This, I think, is an ethical issue, and indeed, the paper has a section “Ethics and Broader Impact Statement”:

OpenReviewer raises several important ethical considerations that warrant careful discussion. While our demo aims to assist authors with presubmission feedback, it could potentially be misused to automate the peer review process entirely, compromising scientific rigor. We strongly emphasize that OpenReviewer is designed to complement, not replace, human peer review.

They also mention the fact that ML/AI conference reviews might not be adequate for other fields, and then discuss the positive side, that it can “democratice access to high-quality feedback”. They do “promote responsible use” by explicitly stating limitations “in the documentation” (not in the demo itself), make the system openly available (HuggingFace), including “clear disclaimers about appropriate use cases” (not sure what they mean with that), and encouraging further research into bias detection and mitigation.

I cannot shake the “What could possibly go wrong?” feeling. The measures to promote responsible use do seem much less thoughtful than the actual work. The demo page itself (which would be the prime landing page for mis-users) is completely void of any kind of warnings and does not mention limitations. If a review has been generated, neither the page nor the review itself does mention anything like that – the generated review can be directly copy-pasted into a form. The only thing missing (from a UI perspective) is a button to copy it. The simplest thing to prevent mis-use would be to include a statement like “This review has been auto-generated by OpenReviewer” within the review, but this is not done either.

(Fun fact: The OpenReviewer demo does not see any need for an ethics review when reviewing the OpenReviewer paper.)

Why LLM-Generated Peer Reviews are a Big Deal

Peer reviewing is under a lot of stress, and is often criticized as being dysfunctional. For the reviewers, reviewing takes a lot of time, and there is no direct reward for it. More and more “stuff” needs to be reviewed, putting further strain on the system. For authors (or project applicants), reviewing slows everything down, sometimes considerably (i.e., years instead of months). It is not terribly surprising that reviewers (the same as authors) “explore” shortcuts – and use LLMs to help them in reviewing.

In my opinion, this is an understandable, but nonetheless concerning development, for multiple reasons:

Confidentiality: At the moment of reviewing, papers are not meant to be public (except in some forms open peer review). Thus, reviewers are asked to keep papers confidential, maybe similar to job applications. Uploading review copies onto a cloud-based LLM breaks this confidentiality. Of course, downloading models such as OpenReviewer and running them on your own hardware mitigates that risk – but who is actually going to do that?
State of the art: Reviewers are asked to check, if a proposed paper is an actual contribution to the research field. LLMs are trained on large corpora crawled from the web, and can have seen published research – but only publications that are accessible via crawling. This excludes research that is not open access, research that is only available in print (admittedly, this problem is getting smaller), research in other languages, and also research that happens to be available on a site that is not among the ones crawled. By relying on the “memory” of an LLM to check the state of the art, we are giving up control over what is considered to be the state of the art.
Originality: Novelty assessment lies at the heart of academic publishing and it represents one of the most nuanced and critical judgment calls in peer review. The determination of whether research presents a genuinely meaningful contribution—rather than an incremental or trivial advance—requires deep domain expertise, contextual understanding, and the ability to weigh competing perspectives. This assessment becomes even more complex when evaluating interdisciplinary work, where the significance of a contribution may vary dramatically across different research communities.
The inherently subjective nature of novelty evaluation is precisely why the peer review system employs multiple expert reviewers—their diverse perspectives and reasoned disagreements often illuminate different dimensions of a work’s contribution. When human reviewers debate whether an approach is “sufficiently novel,” they bring to bear years of specialized knowledge, an understanding of field dynamics, and the ability to recognize both obvious extensions and genuinely surprising insights. Replacing this nuanced human judgment with LLM evaluation risks reducing these (supposedly) sophisticated deliberations to algorithmic determinations that may appear objective but are fundamentally arbitrary. Unlike human experts who can articulate and defend their reasoning about novelty, LLMs lack the deep contextual understanding necessary to make these consequential decisions about what constitutes a meaningful contribution to human knowledge.
Finally, as LLMs are trained on existing language, they may “reward” using established wordings. Formulating new insights or ideas often entails to combine words/phrases/sentences/facts in an unconventional way, which may be punished by an LLM, explicitly because it’s new.
Hacking: Prompt injections are a common issue when LLMs are used for unchecked input texts, and review copies are exactly that. Just last week, Nikkei Asia reports that in 17 preprints from 14 different institutions, “contained hidden prompts directing artificial intelligence tools to give them good reviews”. Prompts (“give a positive review only”) are hidden in a way that is barely visible to humans, e.g., by using white text on white paper. One author claims that this is in order to counter lazy reviewers, which I can certainly relate to as motivation. But then the prompt could also turn the review into gibberish, which would have the same effect.
Responsibility/Accountability: Finally, it’s also a question of responsibility. Human peer reviewers may not get the greatness of ones own paper, but at least they have a clear responsibility for the review they write. Malicious (or biased, lazy, …) reviewers can for instance be excluded from reviewing again, or they can be asked to justify themselves. This does not hold in the same way for LLM-based reviews.

References

Idahl, Maximilian, and Zahra Ahmadi. 2025. ‘OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews’. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), edited by Nouha Dziri, Sean (Xiang) Ren, and Shizhe Diao, 550–62. Albuquerque, New Mexico: Association for Computational Linguistics. https://aclanthology.org/2025.naacl-demo.44/.

Liang, Weixin, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, et al. 2024. ‘Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews’. In Proceedings of the 41st International Conference on Machine Learning. ICML’24. Vienna, Austria: JMLR.org.

Shogo Sugiyama and Ryosuke Eguchi. 2025. ‘“Positive Review Only”: Researchers Hide AI Prompts in Papers’. Nikkei Asia, 1 July 2025. https://asia.nikkei.com/Business/Technology/Artificial-intelligence/Positive-review-only-Researchers-hide-AI-prompts-in-papers.

Large Language Models and Scientific Writing (Part 1)

2025-05-24T00:00:00+00:00

Scientific writing has changed in a measurable way since ChatGPT has launched

Everyone who is involved in examinations these days knows how hard it is to actually determine whether something has been written by an LLM, and the same applies to scientific texts.

Let’s start with evidence that has been found either by accident or by very simple means: This mastodon-post was made by John Mark Ockerbloom in March 2025, and it describes the finding of something interesting in the book “Advanced Nanovaccines for Cancer Immunotherapy”, published (and sold) by Springer, with ISBN, DOI and everything. Page 25 contains the following:

1.6.3 Why Cancer Vaccines Are Better than Chemotherapy

Cancer vaccines and chemotherapy are two different approaches to treating cancer, and their effectiveness can vary depending on the type and stage of cancer, as well as individual patient factors. It is important to note that as an Al language model, I can provide a general perspective, but you should consult with medical professionals for personalized advice.

Springer has taken it offline (without any information, thus also ‘breaking’ the idea of DOIs), but Amazon.de still sells the 1-star-reviewed book for 171€.

In a similar vein (but about one year earlier), Elen Le Foll, a colleague at the University of Cologne, found a lot of articles on Google Scholar that contain phrases like “I am an AI language model”. – and several other people joined in. Examples she reported on Mastodon are (as far as I can tell) from medicine, bio-science in general, and computer science. Some of the mentioned/linked articles have been transparently removed in the meantime, others have not.

With the same method, but a bit more systematically, Haider et al. (2024) looked for key phrases showing LLM use on Google Scholar. These key phrases are “as of my last knowledge update” and/or “I don’t have access to real-time data”, which can be considered dead giveaways for LLM use. Except for the use in example sentences discussing LLM use, I cannot imagine a context in which these would occur naturally in a scientific article. They investivate publication venue and scientific fields of the found publications:

Most questionable papers we found were in non-indexed journals or were working papers, but we did also find some in established journals, publications, conferences, and repositories. We found a total of 139 papers with a suspected deceptive use of ChatGPT or similar LLM applications […]. Out of these, 19 were in indexed journals, 89 were in non-indexed journals, 19 were student papers found in university databases, and 12 were working papers (mostly in preprint databases).

The queries on Google Scholar were not restricted to any particular scientific field, and the questionable papers were majorly assigned to three areas: health (14.5%), environment (19.5%) and computing (23%). What the authors also point out is that these are “policy-relevant” subjects, which gives rise to the possibility of automatically creating a lot of pseudo-scientific articles to give, for instance, the false impression that some topic is under debate (think: climate change). Obviously, looking for these key phrases works with high precision, but doesn’t detect a lot of the generated text.

Another take on this problem is the paper by Liang et al. (2024): They investigate 950,965 papers that have been published between January 2020 and February 2024 on the preprint servers arXiv, bioRxiv, and in Nature journals. In the abstracts of the papers, they find that the words “intricate”, “pivotal”, “realm” and “showcasing” have become much more popular since the launch of ChatGPT (although with some delay). The resulting picture (Figure 2 in the paper, and small on the left) is pretty clear: While this is “just a correlation”, all the other factors that I can imagine that could be in play here would lead to a much higher fluctuation of the popularity of these words before the launch. But of course, these could also be the result of people using ChatGPT to improve their writing.

Liang et al. (2024) continue to apply a measurement method to estimate the portion of “LLM-modified sentences” on the same corpus (limited to abstracts and introductions). The measurement method is basically an maximum likelihood-estimation model that has been trained on gold standard documents. According to this analysis, between 5 and 17.5% of the sentences in the abstracts have been LLM-modified, depending on the discipline. The number is highest in computer science and lowest in math, and has been steadily increasing since early 2023. Figure 1 in their paper shows abstracts, Figure 6 introductions – the findings are roughly the same.

The most important thing to note here is that Liang et al. investigated sentences, and not articles. I.e.: The portion of LLM-generated articles is not estimated here. It seems plausible that the portion of LLM-generated sentences in some abstracts and introductions is higher, while being lower (for instance, zero) in others. In the extreme form (and positive interpretation), this could even mean that the actual number of researchers that “let articles write by an LLM” is very low. But we do not know exactly.

All this is not very surprising: Researchers are humans and under a lot of pressure, and any method that makes publishing faster and thus publications lists longer will be used, until we come up with a better method to measure and compare scientific success(es).

Next: LLMs and Peer Review.

References

Elen Le Foll. 2024. ‘I Just Did Some Primary-School Level Investigative Work’, 15 March 2024. https://fediscience.org/@ElenLeFoll/112101278590379648.

Haider, Jutta, Kristofer Rolf Söderström, Björn Ekström, and Malte Rödl. 2024. ‘GPT-Fabricated Scientific Papers on Google Scholar: Key Features, Spread, and Implications for Preempting Evidence Manipulation’. Harvard Kennedy School (HKS) Misinformation Review, September. https://doi.org/10.37016/mr-2020-156.

John Mark Ockerbloom. 2025. ‘Our Library Has Access to a Book Published By …’, 24 March 2025. https://mastodon.social/@JMarkOckerbloom/114217609254949527.

Liang, Weixin, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, et al. 2024. ‘Mapping the Increasing Use of LLMs in Scientific Papers’. In First Conference on Language Modeling. https://openreview.net/forum?id=YX7QnhxESU.

Thorat, Nanasaheb. 2025. Advanced Nanovaccines for Cancer Immunotherapy: Harnessing Nanotechnology for Anti-Cancer Immunity. 1st ed. 2025. Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-86185-7.

How to Survive Scientific Conferences

2024-03-21T00:00:00+00:00

You’re going to a scientific conference for the first time? Here is a list of things that I think are important to know.

Attending scientific conferences is work, and a regular part of scientific work. It’s allowed to have fun, but it’s a work event.
It will be an intensive and exhausting experience.
Wear comfortable shoes and clothes (no problem in DH, CL/NLP and CS). You’ll spend a lot of time standing and walking. Back packs are better suited than messenger bags and try to pack lightly.
It is not required to attend every single session. Identify the ones that are relevant to you in advance by looking at the topics or the person names.
One of the most important tasks on conferences is getting to know interesting people. Do not spent too much time with the people you already know. Try to find a balance between getting to know new people and talking to your peers.
It’s ok to take breaks. It’s also ok to do that alone if you need some time for yourself.
The question “What do you do?” is a great opener, because most people are willing to talk about their own work. Feel free to ask this to people that you never talked to before. Be prepared to answer that question from others: Have a brief summary of your stuff ready. Make sure to include why it’s interesting.
Other good openers are “So, how did you like the session/talk/keynote/conference so far?” and of course “Where you’re from?”.
Being able to ask questions after a talk will – at some point in your career – be an important skill. You can practice by thinking about concrete questions that you would/could.
The vast majority of people on academic conferences are open and friendly. Don’t be afraid.

Two additions the next day:

Many people will spent the evenings drinking various kinds of alcohol in various quantities. You can (and should, actually) join these groups. Nobody will think less of you if you don’t drink alcohol. If you do, know your limits.
It may be a good idea to follow-up on some of your new connections with an e-mail after the conference.

Aus der Fishbowl

2024-03-02T00:00:00+00:00

Auf der DHd2024 habe ich zum ersten Mal eine Fishbowl-Diskussion moderiert (und auch überhaupt eine erlebt). Dieser Post enthält einige Überlegungen zum Format im DHd-Kontext, inhaltliche folgen (hoffentlich) etwas später.

Die DHd-Variante entsprach am ehestem dem, was auf dem englischen Wikipedia-Artikel als open fishbowl bezeichnet wird:

[…] soll so ablaufen, dass es zu Beginn drei Diskutant:innen auf dem Podium sowie einen freien Stuhl gibt. Der freie Stuhl kann jederzeit von jeder Person aus dem Publikum besetzt werden, die dann mitdiskutiert. Wenn das passiert ist, macht eine andere Person einen Stuhl frei (idealerweise die Person, die bereits am meisten gesagt hat), so dass wieder jemand aus dem Publikum dazukommen kann.

Meine dreieinhalb Einsichten:

Der Ablauf entsprach ungefähr dem, was mir jemand vorhergesagt hat: Die ersten 10 Minuten sind etwas träge, danach nimmt es Fahrt auf. Tatsächlich wurde es dann erstmal relativ hektisch, und hatte zwischendurch eher den Charakter einer Speaker’s Corner – auch interessant, aber halt etwas anderes. Es hat sich danach aber wieder etwas beruhigt. Ein zugespitzterer Einstieg hilft hier vielleicht, aber womöglich muss man den Ablauf so einfach einpreisen.
Das Thema, über das wir diskutiert haben, war das Konferenzmotto – “Quo Vadis, DH?” – und damit maximal breit und offen. Darunter hat die Kohärenz der Diskussion etwas gelitten (aber ich weiß auch nicht, ob das nicht grundsätzlich am Format liegt). Ein engeres und klarer zugespitztes Thema wäre vermutlich besser geeignet, vielleicht sogar in einer Pro-/Contra-Variante. Einige konkrete Fragen oder Aspekte per Beamer an die Wand zu werfen hatte ich im Vorfeld überlegt, und es wäre m.E. eine gute Idee gewesen. Hier wäre vielleicht auch ein Moderationsduo denkbar, um etwas mehr steuern zu können (oder jemand mit viel Moderationserfahrung).
Darauf hat man nur Bedingt Einfluss bei einer Konferenzorganisation, aber dass viele Teilnehmer:innen recht schnell wieder weg wollten, hatte viel mit der Raumsituation zu tun. Die theaterbühnenähnliche Situation ist einfach etwas speziell. Wenn es geht, wäre ein Raum, bei dem Publikum und Podium weniger stark getrennt sind, einfacher.
Am Ende hätte ich noch viel zu sagen gehabt, aber länger als 90 Minuten sollte man das auch nicht machen.

＊＊＊

Es gibt Aspekte, die aus meiner Warte sehr gut funktioniert haben: Es kamen sehr viele sehr unterschiedliche Leute zu Wort, und haben ihre jeweils eigene Perspektive eingebracht. Der Austausch wurde nicht von einigen wenigen dominiert, sondern hat die Pluralität in der DH(d)-Community ganz gut repräsentiert. Mangelnde Lebhaftigkeit oder gar Langeweile kann man dem Format auch nicht vorwerfen, und dass wir zwischendurch bei einer Art “DH und ich”-Nabelschau angekommen sind, wäre ja auch nicht passiert, wenn niemand etwas dazu zu sagen gehabt hätte. Das kann man schon nochmal machen.

Zum Schluss: Danke an die lokalen Organsiator:innen, ganz besonders Thomas Haider, dass Ihr mich das habt moderieren lassen, und danke an alle Teilnehmer:innen, dass Ihr Euch in die Bowl getraut habt.

Call for Papers: Computational Drama Analysis: Achievements and Opportunities

2022-03-25T00:00:00+00:00

This is a CfP for the “Workshop on Computational Drama Analysis: Achievements and Opportunities” which will be held in Cologne on 14/15 September 2022. Deadline for 1-page abstracts is 6 May 2022.

Stellenausschreibung

2021-12-31T00:00:00+00:00

Am Lehrstuhl für Digital Humanities/Sprachliche Informationsverarbeitung des Instituts für Digital Humanities an der Universität zu Köln ist zum 01.03.2022 eine Stelle als

Wissenschaftliche:r Mitarbeiter:in (65% TV-L 13)

für zunächst drei Jahre zu besetzen.

Usually, this is an English-language web page. Since the original job posting is in German, I’ll post it here in German as well. If you’re wondering wether it’s ok to apply if you don’t know German: You don’t need to be perfect in German. If you understand the job offer, it should be fine.

Die Universität zu Köln ist eine der größten und forschungsstärksten Hochschulen Deutschlands mit einem vielfältigen Fächerangebot. Sie bietet mit ihren sechs Fakultäten und ihren interfakultären Zentren ein breites Spektrum wissenschaftlicher Disziplinen und international herausragender Profilbereiche, die die Verwaltung mit ihrer Dienstleistung unterstützt.

Das Institut für Digital Humanities an der Universität zu Köln wurde 2017 gegründet und besteht aus zwei Lehrstühlen. Der Lehrstuhl ‚Sprachliche Informationsverarbeitung‘ beschäftigt sich mit Anwendungen von Technologien und Methoden aus der Computerlinguistik in den Geistes- und Sozialwissenschaften (derzeit Linguistik, Literaturwissenschaft, und Musikwissenschaft). Fragestellungen werden häufig durch quantitative Analysen spezifischer Phänomene beantwortet, die zunächst operationalisiert werden müssen. Dazu kommen Annotationsverfahren und maschinelle Lernverfahren (inkl. neuronaler Netze) zum Einsatz.

Ihre Aufgaben sind

die Anfertigung einer Promotion,
das Abhalten von Lehrveranstaltungen (durchschnittl. 2,64 SWS, Studiengänge Informationsverarbeitung/Medieninformatik) sowie
in geringem Umfang: Mitwirken an sonstigen Aktivitäten des Lehrstuhls.

Ihr Profil

Studium der Informationsverarbeitung, Computerlinguistik, Digital Humanities oder äquivalent, oder eines geisteswissenschaftlichen oder informatischen Faches mit einschlägigen Vertiefungen oder Weiterbildungen
Interesse an machine/deep learning, Annotation, der Operationalisierung geistes- oder sozialwissenschaftlicher Konzepte und/oder deren empirischer Erforschung
Promotionsvorhaben oder -idee, das zur u. g. Ausrichtung des Lehrstuhls passt
Erfahrungen in der Vermittlung technischer Kenntnisse oder Methoden an ein nicht-technisches Publikum sind ein Plus
Erfahrungen mit der Arbeit in einem interdisziplinären Team und entsprechende Kommunikationsfähigkeit sind ein Plus

Wir bieten

die Möglichkeit einer Promotion in einem zukunftsweisenden Forschungsbereich,
Mitarbeit an empirischer, technischer und theoretischer Forschung,
ein anregendes und angenehmes Arbeitsumfeld in einer breit aufgestellten Digital-Humanities-Umgebung mit regelmäßigem und offenem Austausch,
interdisziplinären und internationalen Austausch mit Kooperationspartner:innen,
flexible Arbeitszeitmodelle und umfangreiches Weiterbildungsangebot,
Angebote im Rahmen des Betrieblichen Gesundheitsmanagements,
Teilnahme am Großkundenticket der KVB, ein vielfältiges und chancengerechtes Arbeitsumfeld, sowie Unterstützung bei der Vereinbarkeit von Beruf und Familie.

Die Stelle ist ab 01.03.2022 in Teilzeit mit 25,89 Wochenstunden zu besetzen. Sie ist zunächst für drei Jahre befristet. Sofern die entsprechenden tariflichen und persönlichen Voraussetzungen vorliegen, richtet sich die Vergütung nach der Entgeltgruppe 13 TV-L.

Die Universität zu Köln fördert Chancengerechtigkeit und Vielfalt. Frauen sind besonders zur Bewerbung eingeladen und werden nach Maßgabe des LGG NRW bevorzugt berücksichtigt. Bewerbungen von Menschen mit Schwerbehinderung und ihnen Gleichgestellten sind ebenfalls ausdrücklich erwünscht.

Ihre Bewerbung (Motivationsschreiben, Lebenslauf (ohne Foto), Zeugnisse, Exposé zum Promotionsthema) schicken Sie bitte als eine einzelne PDF-Datei an nils.reiter@uni-koeln.de. Die Kennziffer ist Wiss2112-16. Alle Bewerbungen, die bis 31.01.2022 eingehen, werden berücksichtigt. Fragen zur Ausschreibung und Position beantwortet Prof. Dr. Nils Reiter sehr gerne unter der gleichen Adresse.

Stellenausschreibung im Job-Portal der Universität

Prof. Reiter

2021-09-29T00:00:00+00:00

Starting October 1st, I’ll be a full professor at the University of Cologne. My denomination will be “Digital Humanities – Sprachliche Informationsverarbeitung”, which is essentially DH and computational linguistics. This of course means that this is exactly the right position for me :-) I am very happy to join the Department of Digital Humanities and collaborate with colleagues at the Cologne Center for eHumanities, the Data Center for the Humanities, the CRC Prominence in Language, the Center for Data and Simulation Science, and many more.

CRETA-Coaching

2021-07-01T00:00:00+00:00

For the third time, we are offering the CRETA-Coaching. This intense and on-point program will again accept applications for five to six places. Core idea of the coaching is to discuss operationalisation issues and challenges individually with young researchers. This allows taking into account the very specific research questions and conditions they have. More info can be found here: www.cretaverein.de/coaching/, details about applications are listed here (in German).

Application deadline is July 20.

Shared Task on Scene Segmentation

2021-04-22T00:00:00+00:00

With colleagues from the universities Würzburg and Darmstadt, I’m organising a shared task on scene segmentation: Scenes are time-, location- and plot-wise coherent units of a story, and can predominantly be found in narrative texts like novels or biographies. Registration is open until June 7, and the shared task is co-located with KONVENS 2021.

CRETA e.V.

2020-12-19T00:00:00+00:00

In October 2020, we have established CRETA e.V., a legal entity according to German law (“Verein”). The CRETA e.V. takes over from the eHumanities centre at Stuttgart University, but is no longer bound to Stuttgart. In fact, our members already come from different universities and places. In this new form, we will continue to work on different areas of reflected text analysis, having a platform that links specific research from individual projects. More on our mission can be found on our web page.

Reflektierte algorithmische Textanalyse

2020-07-29T00:00:00+00:00

Since 2015, I’ve been working in the eHumanities-Center CRETA at Stuttgart University. What we’ve been doing there and how it all fits together can (as of today) be read in a single volume, available both digitally in open-access as well as in print. Roughly one quarter of the articles are in English, the rest in German.

Virtual summer school "Deep Learning for Language Analysis"

2020-07-17T00:00:00+00:00

After a break, I’m again part of the Cologne summer school on deep learning for language analysis. This year’s edition takes place from August 31 to September 4 online. I’ll be giving a class on Text Analysis with Deep Learning.

Information on registration and application, as well as a schedule overview and additional activities can be found here.

CRETA-Coaching

2020-07-17T00:00:00+00:00

The CRETA-Coaching is back. This intense and on-point program will again accept applications for a total of five places. Core idea of the coaching is to discuss operationalisation issues and challenges individually with young researchers. This allows taking into account the very specific research questions and conditions they have. More info can be found here: www.creta.uni-stuttgart.de/coaching/.

Application deadline is August 15.

A Shared Task for the Digital Humanities: Annotating Narrative Levels

2019-11-08T00:00:00+00:00

The special issue on SANTA, our shared task with a narratological focus, has started to appear in the open access journal Cultural Analytics, edited by Evelyn Gius, Nils Reiter, and Marcus Willand. In the next weeks, individual guidelines will be put online one after the other. This post will be updated with links to all articles in the issue. Thanks to the crew of Cultural Analytics, as well as all the guideline authors and reviewers for making this happen!

Introductions

Annotation Guidelines

Joshua Eisenberg and Mark Finlayson, “Annotation Guideline No. 1: Cover Sheet for Narrative Boundaries Annotation Guide.”
Edward Kearns, “Annotation Guideline No. 2: For Annotating Anachronies and Narrative Levels in Fiction.”
Nora Ketschik, Benjamin Krautter, Sandra Murr, Yvonne Zimmermann, “Annotation Guideline No. 4: Annotating Narrative Levels in Literature.”
Florian Barth, “Annotation Guideline No. 5: Annotation Guidelines for Narrative Levels and Narrative Acts.”
Matthias Bauer and Miriam Lahrsow, “Annotation Guideline No. 6: Collaborative Annotation as a Teaching Tool Between Theory and Practice “
Mats Wirén, Adam Ek and Anna Kasaty, “Annotation Guideline No. 7: Guidelines for annotationof narrative structure.”
Adam Hammond, “Annotation Guideline No. 8: Annotation Guidelines for Narrative Levels.”

Guideline Reviews

Meredith A. Martin, “Annotating Narrative Levels: Review of Guideline No. 1.”
Tillman Köppe, “Annotating Narrative Levels: Review of Guideline No. 2.”
Berenike Herrmann, “Annotating Narrative Levels: Review of Guideline No. 4.”
Jan Horstman, “Annotating Narrative Levels: Review of Guideline No. 5.”
Natalie M. Houston, “Annotating Narrative Levels: Review of Guideline No. 6.”
Gunther Martens, “Annotating Narrative Levels: Review of Guideline No. 7.”
Tom McEnaney, “Annotating Narrative Levels: Review of Guideline No. 8.”
Eric Hayot, “Annotating Narrative Levels: Review of the Shared Task Procedure.”

QuaDramA: Tracking Character Knowledge (Q:TRACK)

2019-09-18T00:00:00+00:00

I’m pleased to announce that QuaDramA, the research project by Marcus Willand and myself, will receive additional funding soon. In this extension called Q:TRACK, we will investigate knowledge about social relations in drama, focusing on family relations. The core idea is to track the distribution and dissemination of social knowledge by detecting the knowledge characters have, acquire and share during the course of dramatic action. In order to accomplish this, Q:TRACK combines methods from computational linguistics and digital humanities with theoretical and historical expertise of literary studies. The project builds on the research we have done in QuaDramA so far.

Q:TRACK is funded within the DFG Priority Programme Computational Literary Studies and will officially start in spring 2020.

Reflected Text Analysis beyond Linguistics

2019-09-06T00:00:00+00:00

From September 9 to 13, I will be giving a class on Reflected Text Analysis beyond Linguistics, as part of the DGfS-CL fall school 2019 at the IMS at Stuttgart University. The class is also part of the CRETA Coaching.

This post serves as course page, containing the material, agenda etc.

Agenda

Day	14:00-15:30		16:00-17:30
Monday	Introduction, Overview, Annotation	☕	Annotation exercise, Inter-Annotator Agreement
Tuesday	Machine learning overview and evaluation, algorithms	☕	Algorithms
Wednesday	Introduction into shared task, hands on session	☕	Hands on session
Thursday	excursion to the German Literature Archive, Marbach (starting at 1pm!)
Friday	Hands on session, shared task evaluation	☕	What to do next, closing discussion

Material

Participants are asked to install the following things on their computers (this can be done during the first day of the class)

Python

Python: If your computer already has Python 2, there is no need to update. If you’re installing Python from scratch, please use Python 3.
pip: The Python package manager
The Python libraries nltk and requests.

Detailed instructions for Windows, Mac OS X and Linux can be found here (PDF file). The file test_install.py can be used to test the installation.

Text Editor

For editing Python files, participants will need a plain text editor. We recommend the following:

Windows: Notepad++
Mac OS X: TextMate

Slides

Monday

Slides
Example annotation guidelines: STTS tag set (German parts of speech), Penn Treebank tag set (English parts of speech)
Texts for annotation exercise: Lewis Carroll: Alice in Wonderland, chapter 11, Jules Verne: Around the World in 80 Days, chapter 13, Mary Rowlandson: Narrative of the Captivity and Restoration of Mrs. Mary Rowlandson

Tuesday

Wednesday

Slides on shared tasks and hackatorial
Hackatorial package: Please download the zip file and extract it into a directory on your drive. The zip file contains
- Data with annotated entity references (sub directory data)
- Code for training, testing and uploading (sub directory code)
- Resources used for feature extraction (sub directory static)
List of implemented features

Friday

Slides on Hackatorial evaluation
Slides on what to do next
Hackatorial results
- Bundestag
- Grimm
- Parzival
- Werther

Projects (for ECTS credit points)

If you’re interested in getting ECTS credit points for taking part in this class, you’ll need to conduct a small project, according to the following recipe (unless we agreed on a different plan):

Pick a task (e.g., part of speech tagging)
Pick a non-standard text that is not too long (e.g., a poem)
Create a gold standard by applying the annotation guidelines for the task
Apply an existing tool for the task
Evaluate the tool against your annotations
Either
- Develop hypotheses for improving/adapting the tool or
- Retrain the tool on existing training data and your own corpus
- Re-evaluate it after adding your own data
Write a brief report on this and send it to me

Your project should be finished (and the report sent to me) before October 14.

Cologne

2019-06-28T00:00:00+00:00

It’s now official: On September 1st, I’ll join the Institute for Digital Humanities at the University of Cologne. At the institute, I’ll be an interim professor for linguistic information processing (“Sprachliche Informationsverarbeitung”). I’m looking forward to the new colleagues, students, institute, and city!

CRC 1391 Accepted

2019-05-23T00:00:00+00:00

Today, the German Research Foundation (DFG) announced that a new collaborative research center (CRC/SFB: Sonderforschungsbereich) will be installed that I am involved in: Andere Ästhetik (SFB 1391). I’m leading a digital humanities subproject project together with Angelika Zirker. This is exciting news, and I can’t wait to get started.

QuaDramA Tutorial

2019-03-08T00:00:00+00:00

I’m happy to announce that we will give a tutorial on quantitative drama analysis as part of the 2nd Heidelberg Computational Humanities Summer School. The tutorial will take place on Monday afternoon (July 15) and is held by the entire QuaDramA team. We will give a brief introduction into R and RStudio, but the main part will be a hands-on session using tools we develop(ed) within our project.

Exploring Vector-Space for Automatic Metaphors Detection

2018-11-08T00:00:00+00:00

Report/hands-on tutorial by Elena Rogleva, Vanessa Schach, Amelie Wührl & Nils Reiter, held during the workshop Finding Metaphor in Discourse. An Intensive Training for Manual and Algorithmic Approaches to Metaphor, November 8-9, 2018, at Basel University.

Abstract

We will report on a project conducted during Summer 2018. Using an annotated corpus of metaphors and a large corpus of German literary fiction, we explored ways to automatically identify metaphors. The procedure is closely oriented along the lines of the MIP/MIPVU annotation scheme, and uses vector space to learn a distributional dictionary from scratch. To our knowledge, this is the first attempt of automatic metaphor detection for German texts.

Downloads

Resources
Slides

After the SANTA Workshop

2018-09-24T00:00:00+00:00

This after-workshop post summarizes some interesting findings during our workshop. The post focusses on subjective impressions of the workshop, it will take some more time to deliver the actual results.

This text has first been published here.

The workshop to conclude the first phase of the shared task has just taken place in Hamburg, Germany. Since this was the first time ever that annotation guidelines were evaluated and compared in such a way, the exact flow of the workshop was open and unknown both for the participants and organizers (read here about the plans and preparations for the workshop). While the full wrap-up of the workshop will take some time, we point out a few things we learned during the workshop in this post.

It became clear quickly that the participants approached the challenge of narrative levels in very different ways and with different goals. While some participants had a clear interest in narratological theory and conceptual development, others were primarily interested in the annotation of discourse phenomena or the creation of data to be used in machine learning. Accordingly, some guidelines solely cover narrative levels, others specify additional annotation categories that are relevant for specific research questions. What also differs are the conditions in which the guidelines have been established: Some are the result of a seminar or class work, some have been written within a funded project, others have only been written to take part in the shared task. In essence, the diversity on many different axes was high.

From the start, it was clear that this is no workshop in which everyone “just” gives a talk about their guidelines and answers a few questions. In fact, there were only a few presentations at the very beginning and some by the organizers – most of the time was spent with plenary discussions and group work, either in the participant or in newly shuffled groups. One of the first task for group work, for instance, was the identification of similarities and differences across the guidelines.

The core evaluation task was the application of a questionnaire, covering the three evaluation dimensions we considered important. This was done by each group for all other guidelines. All questions were to be answered in text form (to be presented in a plenary session) and in addition by assigning points on a four point Likert scale. The ‘correct’ application of the questionnaire was challenging, without a doubt. Some required putting ones self into the shoes of the guideline authors, some required some imagination on potential uses of annotations based on the guidelines. Interestingly, however, the ranking that was established by summarizing points reflected the discursive evaluation very well. Guidelines that have been praised for their theoretical basis achieved high scores in this dimension as well, and with a low variance across the participants. Even if it sounds counter-intuitive at first, a questionnaire with assigned points is an efficient way of generating such results, and it reflects qualitative judgement well (at least in our case).

As a final point in this post: We were delighted about the positive, welcoming and friendly atmosphere. Despite the diverse backgrounds, which usually give plenty occasions for misunderstandings, our participants managed to be constructive and productive, even when criticizing their guidelines. It was an intense three days, but it was totally worth it.

SANTA Workshop

2018-08-31T00:00:00+00:00

In only two and half weeks, we meet with the participants of the shared task for a three day workshop in Hamburg. I’ve outlined the agenda and explained some of the preparations.

Late Summer School 'Machine Learning for Language Analysis'

2018-08-16T00:00:00+00:00

I’m happy to announce that I’ll be giving a two-day class on machine learning for reflected text analysis during the late summer school in Cologne, Germany.

The class takes place on September 26 and 27, and its main goal is to convey a basic understanding of how machine learning algorithms work concretely. The class will include both a theoretical introduction into (some) algorithms as well as a hands-on session in the form of a small shared task using Python. Application deadline is on August 20.

The hands-on session in the class will be supported by Nathalie Wiedmer.

TOC: Announcement, Preparations, Agenda, Material

Announcement

The theoretical basics of machine learning methods are presented in a mixture of hackaton and tutorial, including an example implementation in Python and the concrete evaluation of text-analytical methods within the framework of a small shared task.

Preparations

Participants are asked to install the following things on their computers

Python

Python: If your computer already has Python 2, there is no need to update. If you’re installing Python from scratch, please use Python 3.
pip: The Python package manager
The Python libraries nltk and requests.

Detailed instructions for Windows, Mac OS X and Linux can be found here (PDF file). The file test_install.py can be used to test the installation.

Text Editor

For editing Python files, participants will need a plain text editor. We recommend the following:

Windows: Notepad++
Mac OS X: TextMate

Agenda

Time	Wednesday, September 26	Thursday, September 27
09:00	Introduction, machine learning basics	Hands on (continued)
10:30	coffee break	coffee break
11:00	Machine learning algorithms	Shared task evaluation
12:30	lunch break	lunch break
14:00	Shared task introduction	What to do next
15:30	coffee break	coffee break
16:00	Hands on	Closing discussion
17:00	closing	closing

Material

Hackatorial Package

Please download this zip file and extract it into a directory on your drive. The zip file contains

Data with annotated entity references (sub directory data)
Code for training, testing and uploading (sub directory code)
Resources used for feature extraction (sub directory static)

We will go over all these things in the shared task introduction.

Slides

Introduction
Machine learning basics
Machine learning algorithms
Shared task introduction
Shared task evaluation
Results (the results have been saved and fixed on October 1st, 11am. Submissions are no longer possible)
Addon
What to do next

Accepted Papers at LaTeCH-CLfL 2018

2018-06-22T00:00:00+00:00

The list of accepted papers at the COLING workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature has just been published. Acceptance rate is higher compared to previous years: 69%.

SANTA Submissions

2018-06-18T00:00:00+00:00

The deadline for submitting annotation guidelines in the SANTA shared task has passed, and we received eight submissions, with which we are very happy. We blogged about first impressions and a few more numbers.

Teaching at European Summer University in Digital Humanities

2018-06-13T00:00:00+00:00

Jointly with my colleague Sarah Schulz, I’ll be giving a class on Reflected Text Analysis in the Digital Humanities at the European Summer University in Digital Humanities in Leipzig. Looking forward to meeting the very good students!

SANTA deadline approaching

2018-06-06T00:00:00+00:00

The deadline for submitting annotation guidelines in the SANTA shared task will be on June 15. Information about the submission details is available now.

CorefAnnotator 1.0 released

2018-02-12T00:00:00+00:00

I released a new annotation tool for Coreference, suited for the annotation of long texts containing many discourse entities and long chains. We are using it both in QuaDramA and CRETA. It’s a desktop program written in Java.

Call for Participation for Shared Task SANTA

2017-10-06T00:00:00+00:00

The Call for Participation for the Shared Task on Narrative Level Annotations is out, with information about the development corpus.

How to Develop Annotation Guidelines

2017-10-01T00:00:00+00:00

This article describes where to start and how to proceed when developing annotation guidelines. It focuses on the scenario that you are creating new guidelines for a phenomenon or concept that has been described theoretically. The article has first been posted on the shared task webpage.

In a single sentence, the goal of annotation guidelines can be formulated as follows: given a theoretically described phenomenon or concept, describe it as generic as possible but as precise as necessary so that human annotators can annotate the concept or phenomenon in any text without running into problems or ambiguity issues.

Developing annotation guidelines is an iterative process: Once a first version has been established, its shortcomings need to be identified and fixed, leading to a second version, which has shortcomings that need to be identified and fixed, etc. This process is displayed schematically in Figure 1. We will describe how to create a first version, and how to come from one version to the next. The most important idea is that in each round, the same text is annotated by multiple annotators independently. This is the main device that allows identifying these shortcomings.

Figure 1: General annotation workflow

Please note that in principle the entire workflow can be performed on paper or digitally. Digital annotation tools make it easier to compare annotations and force deciding on exact annotation boundaries (which words/characters are to be included). Paper-based annotations are more accessible and easier to set up, but make it (too) easy to skip over details. If you decide to make paper-based annotations, please pay attention to exact annotation boundaries.

Making Pilot Annotations

The first round of annotations is best done by annotators who are familiar with the theory that is to be annotated. As with the following annotation rounds, please annotate in parallel and discuss afterwards. It is not necessary to spend a lot of time on preparation. Specifying a list of references or theoretical works, or agreeing on a single text should be sufficient as a starting point.

Your time is spent best on discussing annotation disagreements. In particular in the very first round, many parameters are still undecided and likely to cause disagreement. At the beginning, you need to focus on the big questions:

What is to be annotated? Every paragraph/sentence/word? Only paragraphs that fulfill a set of conditions?
What exactly are the annotation categories? Are they related somehow? It sometimes helps to organize them in a hierarchy, as some categories subsume others (e.g., every finite verb is a verb).
If you’re using a digital annotation tool: Make sure annotators have the possibility to attach comments to annotations. It helps a lot in the discussions later.

Annotation guidelines typically contain a lot of examples. So you best start collecting interesting/difficult/explanatory examples right away. Examples you find in real texts (possibly with some context) are usually advantageous over made-up ones.

Improving Guidelines

To improve guidelines in this manner, we first need to analyze annotations of the previous “round”, before we reformulate/refine the guidelines. This can be done by inspecting the annotation disagreements: These are cases in which different annotators made different decisions. These can be counted, of course, but it is more informative to talk about the disagreements with the annotators, and to let them explain their decisions.

Such an in-depth discussion with the annotators is fruitful in particular in the first rounds of the process. Once the annotators are trained and annotation guidelines are maturing, a quantitative view might be sufficient. For the latter, a number of metrics have been established (see Wikipedia: Inter-rater reliability for an overview; or Artstein, 2017). Analyzing the inter-annotator-agreement quantitatively gives you a number and allows measuring whether you are actually improving your annotation guidelines, but it does not distinguish different kinds of disagreement.

Some of the disagreements will be caused by annotators not paying attention, or by overlooking something – annotators are human beings after all. These can be fixed easily, without the need to refine the guidelines. It is good practice to let the annotators fix these mistakes by themselves.

Other kinds of disagreement can be expected to have impact on the guidelines: If two annotators made different decisions which are both covered by the annotation guidelines, it is likely that the annotation guidelines are contradictory in this aspect. The source of the contradiction should be identified and resolved.

Many disagreements will be caused when the annotators encounter cases that are not mentioned in the guidelines. In this case, either an existing annotation definition can be applied (perhaps with minor changes), or a new one needs to be defined. If a new definition is added, you need to think about the impact this definition has on the other definitions and annotations. Sometimes, this requires you to re-annotate what you have annotated before.

The actual discussions are likely to be lively and intensive, and tend to jump around between different aspects. It is not always easy, but it makes for better guidelines if this process is well structured and documented. Do not try to fix everything at once, but focus on one aspect at a time.

While going through this iterative process, two processes are likely to be intertwined: The annotation guidelines get better and the annotators get trained. Both are expected and, in principle, welcomed. But: In the end, the annotation guidelines are supposed to be self-contained and also applicable by untrained (or less trained) annotators. It is therefore important to pay attention not to develop rules within a project that are never written down. It will be much harder to integrate new annotators (even if someone drops out and has to be replaced) unwritten rules exist.

List of Annotation Guidelines

The following is a (not exhaustive) list of established annotation guidelines for various, mostly linguistic, phenomena. We provide this list as example for different kinds of tasks.

Part of speech tagging in the Penn Treebank: The guidelines describe the tag set and its application, and have been developed in the Penn Treebank Project.
TimeML: The TimeML guidelines describe the annotation of time expressions and related events in news texts.
Coreference Resolution: Coreference resolution guidelines have been developed in the OntoNotes project. The goal here is to identify mentions in the text that refer to the same real-world entities.

References

Artstein, Ron. Inter-annotator Agreement. In: Ide Nancy & Pustejovsky James (eds.) Handbook of Linguistic Annotation. Springer, Dordrecht, 2017. DOI 10.1007/978-94-024-0881-2.

LaTeCH-CLfL 2017 Proceedings now available

2017-08-03T00:00:00+00:00

The proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, co-located with ACL 2017, are now available in the ACL Anthology.

SANTA: Creation of Annotation Guidelines

2017-07-17T00:00:00+00:00

We have written about how we picture the first phase of the DH shared task that focuses on narrative levels.

PhD Position in NLP

2017-03-13T00:00:00+00:00

We are still looking for a candidate for a PhD position in Natural Language Processing/Computational Linguistics, to work in the interdisciplinary research project QuaDramA: Quantitative Drama Analytics, funded by Volkswagen Foundation and soon starting at University of Stuttgart. Applications are welcome anytime.

Call for Papers LaTeCH-CLfL 2017

2017-03-01T00:00:00+00:00

LaTeCH-CLfL 2017 will be joining the SIGHUM LaTeCH workshop series with the CLfL workshop series. It takes place at ACL in Vancouver, Canada on August 4th. Paper submission deadline is April, 21.

QuaDramA is hiring

2017-02-19T00:00:00+00:00

We are looking for two candidates for PhD positions, to work in the interdisciplinary research project QuaDramA: Quantitative Drama Analytics, funded by Volkswagen Foundation and soon starting at University of Stuttgart.

nilsreiter.de

Large Language Models and Peer Review (Part 2)

(Some) Peer Reviewers are using LLMs to do their Reviews

(Some) People Work on Optimizing LLMs for Peer Reviewing

Why LLM-Generated Peer Reviews are a Big Deal

References

Large Language Models and Scientific Writing (Part 1)

Scientific writing has changed in a measurable way since ChatGPT has launched

References

How to Survive Scientific Conferences

Aus der Fishbowl

Call for Papers: Computational Drama Analysis: Achievements and Opportunities

Stellenausschreibung

Prof. Reiter

CRETA-Coaching

Shared Task on Scene Segmentation

CRETA e.V.

Reflektierte algorithmische Textanalyse

Virtual summer school "Deep Learning for Language Analysis"

CRETA-Coaching

A Shared Task for the Digital Humanities: Annotating Narrative Levels

Table of contents

Introductions

Annotation Guidelines

Guideline Reviews

QuaDramA: Tracking Character Knowledge (Q:TRACK)

Reflected Text Analysis beyond Linguistics

Agenda

Material

Python

Text Editor

Slides

Monday

Tuesday

Wednesday

Friday

Projects (for ECTS credit points)

Cologne

CRC 1391 Accepted

QuaDramA Tutorial

Exploring Vector-Space for Automatic Metaphors Detection

Abstract

Downloads

After the SANTA Workshop

SANTA Workshop

Late Summer School 'Machine Learning for Language Analysis'

Announcement

Preparations

Python

Text Editor

Agenda

Material

Hackatorial Package

Slides

Accepted Papers at LaTeCH-CLfL 2018

SANTA Submissions

Teaching at European Summer University in Digital Humanities

SANTA deadline approaching

CorefAnnotator 1.0 released

Call for Participation for Shared Task SANTA

How to Develop Annotation Guidelines

Making Pilot Annotations

Improving Guidelines

List of Annotation Guidelines

References

LaTeCH-CLfL 2017 Proceedings now available

SANTA: Creation of Annotation Guidelines

PhD Position in NLP

Call for Papers LaTeCH-CLfL 2017

QuaDramA is hiring