Institute of Cognitive Science

Research Group Computer Vision

Navigation und Suche der Universität Osnabrück

Main content

Top content


Coreference Resolution (CoRe) describes the task of determining which phrases in a text or spoken dialog refer to the same real-world entity and has been an active research field in natural language processing since the 1960s. During the course of the more than 50 years the focus has undergone a gradual change. Whereas heuristic methods marked the beginnings, especially since the 1990s more and more researches focused on machine learning methods to tackle the problem of Coreference Resolution. Although they benefit from the technological advances of modern cpu and gpu computing, they run into the same problems as all machine learning approaches. They heavily rely on the quality and quantity of available training data.[1]


When project sucre was initiated in 2008 the lack of training data has been identified as one source for the stagnating progres in this area of research. Creating high quality annotated datasets however is a highly time-consuming task. Automated methods on the other side are a much faster way to quickly annotate high amounts of data, but unfortunately yield also in a much lower quality. One aim of the project was to combine the quality of a human annotator with the speed of the automated system. The result of this endeavour was the annotation tool COALDA.

COALDA Application

A screenshot of the COALDA Application showing a trained SOM.

COALDA is an annotation tool that uses a semi-supervised approach. Instead of a text centered approach like other corpora annotation tools (MMAX2[2]), Coalda focuses on visualizing the feature space instead. As it is impossible to directly visualize a high-dimensional feature space, an indirect way has to be found. In Coalda, the feature space is visualized by training a Self Organizing Map (SOM) and visualizing this SOM. The software allows the user to:

  • interactively explore the map
  • label feature vectors with coreference information
  • recalculate the map with different settings

The SOM is a good tool to reveal clusters of coreferent and disreferent mentions in a text document. Those cluster can then be annotated at once with the regarding coreference information. Additonally a confidence value can be added, which allows for a fast way of annotating big sets of data. Besides this main functionality the tool also allows the user to inspect the existing feature space in more detail or develope new features.


Besides gaining more data, another group of researches within the project was concerned with explorating the already existing ones. For this purpose they created a modular Framework named Sucre. Following the apparent trend they not only incorporated local features, features that are defined on a pair of mentions, but global features that work on clusters of mentions. This way it can work hand in hand with the annotation tool. Furthermore they participated in several Corefenrence Resolution Challenges among them the SemEval2010 Task1[3] and CoNLL-2011 Shared Task[4], where its performance was tested on multiple languages. Sucre showed a cosistently good performance and even optained the best results in the closed annotation tracks of English, German and Italian.

Big Data and Cross-Document Coreference Resolution

In the recent years one topic has gained increasing interest Big Data. The term describes datasets that are too big, too complex and/or to rapidly changing to process them manually or with traditional methods of data analytics. Accordingly Big Data is often described as a three dimensional problem, whereas the dimensions are also known as the three V's of Big Data: Volume, Variety & Velocity[5][6]. Whilst Volume and Velocity pose problems mostly related to the underlying hardware and the runtime complexity of the used algorithms, Variety poses problems of a completely other kind. A dataset is not solely limit to a single source, can contain incomplete data and even comprise different types of data like auditory, visual or textual information. In short, it contains a lot of unstructured or semistructured content that needs a huge amount of preprocessing before it becomes accessible and analyzable. As the amount of data created by humanity is further growing, especially faster than humanities ability to store data[7], elaborated methods to filter out and categorize the important bits of information are needed if one wishes to harvest big data effiently.

Coreference Resolution is an important preprocessing step and builds the basis for a broad variety of other machine learning techniques. A high precision and accuracy are therefore needed as errors will be carried forth through the processing chain. The latest findings and trends in the area of (Co)Reference Resolution[1] strongly indicate that the area of research is shifting their attention towards big data, focusing on algorithms that work on a more global scale. Cross-Document Coreference Resolution has become a field of peticular interest as it enables the analysis of data among different documents as well as previously processed data, opposed to the analysis of data within a single document, as it has been in the former focus of research.

For a human to properly understand the content of one document and relate it to another one the context is an important component. Traditional Coreference Resolution algorithms usually lack additional knowledge and solely depend on the information that can be extracted from the documents itself. To reliably relate the references from different documents to one another this however is not sufficient.

In order to overcome this problem we decided to harvest the power of knowledgebases like WikiData, DBpedia or YAGO, that contain facts about millions of entities as well as their relations among each other.[8,9,10]. We currently develope a new method that incorporates this background knowledge into the coreference resolution task to find a more accurate and reliable way of making huge collections of data accessible to both humans and machines.


[1] Vincent Ng.
Supervised Noun Phrase Coreference Research: The First Fifteen Years.
In ACL '10: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1396–1411, ISBN: 978-1-932432-67-1, 2010. Association for Computational Linguistics.
| DOI | BibTeX
[2] C. Müller & M. Strube.
Multi-level annotation of linguistic data with MMAX2.
In Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods,pages 197–214,ISBN: 978-3-631-54720-5, 2006.
| BibTeX
[3] M. Recasens, L. Màrquez, E. Sapena, M. Antònia Martí, Ma. Taulé, V. Hoste, M. Poesio & Y. Versley.
SemEval-2010 task 1: Coreference resolution in multiple languages.
In SemEval '10 Proceedings of the 5th International Workshop on Semantic Evaluation, pages 1–8, ISBN: 978-1-932432-70-1, 2010. Association for Computational Linguistics.
| BibTeX
[4] S. Pradhan, L. Ramshaw, M. Marcus, M. Palmer, R. Weischedel & N. Xue.
CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes.
In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27, ISBN: 978-1-937284-08-4, 2011. Association for Computational Linguistics.
| BibTeX
[5] D. Laney
3D Data Management: Controlling Data Volume, Velocity, and Variety.
, 2001. META Group.
| BibTeX
[6] D. Laney & M. A. Meyer
The importance of'big data': a definition.
, 2012. Stamford, CT: Gartner.
| BibTeX
[7] V. Turner, J. F. Gantz, D. Reinsel & S. Minton
The Digital Universe of Opportunities : Rich Data and the Increasing Value of the Internet of Things.
, 2014. IDC.
| URL | BibTeX
[8] Wikidata: a free linked database that can be read and edited by both humans and machines.
Wikimedia Foundation
[9] DBPedia.
University of Mannheim, Openlink Software, University of Leipzig
[10] F. Suchanek, G. Kasneci, J. Hoffart, E. Lewis-Kelham, E. Kuzey, J. Biega, F. Mahdisoltani & G. Weikum
YAGO: A High-Quality Knowledge Base.
Max Planck Institute for Informatics (Saarbrücken/Germany)

Project publications


[1] A. Burkovski & G. Heidemann.
Visualization for Coreference Annotation..
In Conference on Recent Advances in Natural Language Processing (RANLP 2011), pages: 692-697, 2011.
| PDF | BibTeX
[2] A. Burkovski, W. Kessler, G. Heidemann, H. Kobdani & H. Schütze.
Self organizing maps in NLP: exploration of coreference feature space.
In Advances in Self-Organizing Maps, pages: 228-237, ISBN: 978-3-642-21565-0, 2011. Springer.
| DOI | BibTeX


[1] H. Kobdani, H. Schütze, A. Burkovski, S.W. Kessler & G. Heidemann.
Relational Feature Engineering of Natural Language Processing.
In Proc. 19th ACM Conf. on Information and Knowledge Management, CIKM 2010, pages: 1705-1708, 2010.
| PDF | DOI | BibTeX