AI Annotator

Key Responsibilities:

• Perform rigorous human validation on pre-annotated data generated by commercial clinical NLP models (e.g., Amazon Comprehend Medical) or internal LLM/encoder-based tools.
• Contribute to the iterative refinement of annotation guidelines and examples to improve inter-annotator agreement and overall data quality.
• Identify and resolve disagreements between pre-annotations and human validations, ensuring the accuracy and consistency of the annotated datasets.
• Collaborate closely with data science and NLP teams to provide feedback on model performance and contribute to the continuous improvement of pre-annotation models.
• Assist in the creation and maintenance of golden datasets for training and evaluating NLP models.
• Participate in discussions regarding data sensitivity and ensure adherence to all relevant data privacy and security protocols, especially for patient data.
• Work on diverse clinical annotation projects (e.g., alcohol and smoking use cases, oncology (ECOG, Karnofsky scores), biomarkers, SDO, and mental health).

Qualifications:

Must Have (Required Qualifications):

• Bachelor's or Master's degree in a relevant field (e.g., Life Sciences, Linguistics, Computer Science, or a related healthcare discipline).
• Proven experience in data annotation, specifically with clinical or biomedical text.
• Strong foundational understanding of NLP concepts and terminology.
• Exceptional attention to detail and ability to maintain high levels of accuracy in data validation.
• Excellent communication and collaboration skills.
• Ability to quickly adapt to various annotation tools (e.g., Inception, Label Studio, Prodigy).

Good To Have (Preferred Qualifications/Skills):

• Prior experience working within the John Snow Labs (JSL) ecosystem for modeling and annotation, including familiarity with their Health AI Lab and GenAI tool.
• Experience with specific advanced annotation types such as named entity recognition (NER), relationship extraction, and assertion status annotation.
• Direct clinical background or extensive practical experience working with sensitive patient data in a healthcare context.
• Familiarity with incremental batch training processes in machine learning.