Evaluation Metrics for End-to-End Coreference Resolution

Commonly used coreference resolution evaluation metrics can only be applied to key mentions (i.e.already annotated mentions). We here propose two variants of the BCubed (Bagga and Baldwin, 1998) and CEAF (Luo, 2005) coreference resolution evaluation algorithms which can be used to evaluate coreference resolution systems dealing with system mentions (i.e. automatically determined mentions).The algorithms and relevant analysis can be found in details in our SIGDIAL 2010 paper.Both BCubedsys and CEAFsys java classes are available for download, along with a couple of necessary parent classes. They should be used within the BART framework, whose repository is available here.

Download

Click here to download BCubedsys and CEAFsys.

Publications related to evaluation metrics for end-to-end coreference resolution

• SIGDIAL’10 (PDF)
  Cai, Jie; Strube, Michael (2010).
  Evaluation metrics for end-to-end coreference resolution systems.
  InProceedings of the SIGdial 2010 Conference: The 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue,Tokyo, Japan, 24-25 September 2010, pages 28-36.