Automatic Generation of Benchmarks for Entity Recognition and Linking.

Benchmarks are central to the improvement of named entity recognition andentity linking solutions. However, recent works have shown that manuallycreated benchmarks often contain mistakes. We hence investigate the automaticgeneration of benchmarks for named entity recognition and linking from LinkedData as a complement to manually created benchmarks. The main advantage ofautomatically constructed benchmarks is that they can be readily generated atany time, and are cost-effective while being guaranteed to be free ofannotation errors. Moreover, generators for resource-poor languages can fosterthe development of tools for such languages. We compare the performance of 11tools on benchmarks generated using our approach with their performance on 16benchmarks that were created manually. In addition, we perform a large-scaleruntime evaluation of entity recognition and linking solutions for the firsttime in literature. Moreover, we present results achieved on the Portugueseversion of our approach on four different tools. Overall, our results suggestthat our automatic benchmark generation approach can create varied benchmarksthat have characteristics similar to those of existing benchmarks.