Loading...
 

Annotation Instructions Library

This idea is part of the A Dollar Worth of Ideas series, with potential open source, research or data science projects or contributions for people to pursue. I would be interested in mentoring some of them. Just contact me for details.


Similar to efforts like Papers With Code, contact authors and publish their annotation instructions. Quality annotation instructions are key for reproducible scientific results and they are difficult to construct, usually involving a long iterative process until convergence.

Even if the data is made available, extending the research work might need to further annotate more data. Sadly, in many scientific communications, the instructions given to annotators are just discussed in abstract terms, while highlighting the the inter-annotator agreement. That is, we know the authors achieved quality annotation instructions for the task... we just don't know what those instructions are. Building a library of such instructions can be a great scientific contribution.