Code for Identifying Incorrect Labels

[GSidiropoulos]

Dear authors, thank you for sharing your work. I was wondering if you can also provide the code for identifying the incorrect labels. From what I understand the label corrections to produce a corrected version are given, however, the code to reproduce is not available.

Best regards,
Georgios

[xuhdev]

We identified the incorrect labels manually, while we utilized some tools to help speed up human review. I believe @frreiss has them available but I don't think they have changed the manual nature of identifying labels in any way.

[GSidiropoulos]

Thank you for your reply. My question mostly refers to the code you have here -> https://github.com/CODAIT/text-extensions-for-pandas/tree/master/tutorials/corpus. How can I obtain the 1054 flagged training samples? Do I have to combine the results from the CoNLL_4.ipynb and CoNLL_3.ipynb notebooks?

[GSidiropoulos]

Can you please clarify how we are supposed to combine the results of CoNLL_2.ipynb, CoNLL_3.ipynb, CoNLL_4.ipynb notebooks in order to obtain the results you report in the paper? In the paper you mention that " We
considered any label where fewer than 7 models agreed with the corpus label to be “flagged”". Is this the only condition you use in order to flag labels?