AI Annotator
3 Months ago • All levels
Research Development
Job Description
Seeking a skilled Human Subject Matter Expert (SME) specializing in Clinical pre-annotation validation. This role involves validating pre-annotated data from clinical NLP models, refining annotation guidelines, and developing high-quality golden datasets. Responsibilities include performing human validation on data from commercial NLP models and internal tools, contributing to the iterative refinement of annotation guidelines, and resolving disagreements in annotations to ensure data accuracy. The role also requires collaboration with data science and NLP teams to provide feedback on model performance and assist in creating golden datasets for model evaluation. Candidates will work on diverse clinical annotation projects including alcohol and smoking use cases, oncology scores, biomarkers, SDO, and mental health, while adhering to data privacy and security protocols.
Good To Have:
- Experience with John Snow Labs (JSL) ecosystem.
- Experience with advanced annotation types (NER, relationship extraction).
- Direct clinical background or experience with patient data.
- Familiarity with incremental batch training.
Must Have:
- Bachelor's or Master's degree in a relevant field.
- Proven experience in data annotation, specifically with clinical/biomedical text.
- Strong foundational understanding of NLP concepts.
- Exceptional attention to detail and accuracy.
- Excellent communication and collaboration skills.
- Ability to adapt to various annotation tools.
Description
Job Summary:
A highly skilled and detail-oriented Human Subject Matter Expert (SME) specializing in Clinical pre-annotation validation. The successful candidate will play a critical role in our incremental annotation process, focusing on human validation of pre-annotated data, refining annotation guidelines, and contributing to the development of high-quality golden datasets. While core clinical annotation skills are paramount, experience with the Johnson Labs (JSL) ecosystem and advanced NLP concepts are highly desirable.
Key Responsibilities:
- Perform rigorous human validation on pre-annotated data generated by commercial clinical NLP models (e.g., Amazon Comprehend Medical) or internal LLM/encoder-based tools.
- Contribute to the iterative refinement of annotation guidelines and examples to improve inter-annotator agreement and overall data quality.
- Identify and resolve disagreements between pre-annotations and human validations, ensuring the accuracy and consistency of the annotated datasets.
- Collaborate closely with data science and NLP teams to provide feedback on model performance and contribute to the continuous improvement of pre-annotation models.
- Assist in the creation and maintenance of golden datasets for training and evaluating NLP models.
- Participate in discussions regarding data sensitivity and ensure adherence to all relevant data privacy and security protocols, especially for patient data.
- Work on diverse clinical annotation projects (e.g., alcohol and smoking use cases, oncology (ECOG, Karnofsky scores), biomarkers, SDO, and mental health).
Qualifications:
To help recruiters identify the best fit, we've outlined the essential skills for this role and additional valuable assets that would further strengthen a candidate's profile.
Must Have (Required Qualifications):
- Bachelor's or Master's degree in a relevant field (e.g., Life Sciences, Linguistics, Computer Science, or a related healthcare discipline).
- Proven experience in data annotation, specifically with clinical or biomedical text.
- Strong foundational understanding of NLP concepts and terminology.
- Exceptional attention to detail and ability to maintain high levels of accuracy in data validation.
- Excellent communication and collaboration skills.
- Ability to quickly adapt to various annotation tools (e.g., Inception, Label Studio, Prodigy).
Good To Have (Preferred Qualifications/Skills):
- Prior experience working within the John Snow Labs (JSL) ecosystem for modeling and annotation, including familiarity with their Health AI Lab and GenAI tool.
- Experience with specific advanced annotation types such as named entity recognition (NER), relationship extraction, and assertion status annotation.
- Direct clinical background or extensive practical experience working with sensitive patient data in a healthcare context.
- Familiarity with incremental batch training processes in machine learning.
Set alerts for more jobs like AI Annotator
Set alerts for new jobs by Saama
Set alerts for new Research Development jobs in United States
Set alerts for new jobs in United States
Set alerts for Research Development (Remote) jobs
