Description:
If precision matters more to you than speed, this role is a strong fit. The labels you create become training data for AI systems used daily by thousands of students. Accurate behavioral labeling improves the model. Inconsistent labels teach it the wrong patterns.
LearnWith.AI develops AI-powered learning experiences grounded in learning science, data analytics, and subject matter expertise. This position transforms raw student session videos into high-accuracy, rubric-aligned labels the team can rely on. You will observe recorded student sessions, pinpoint critical behavioral events, and apply precise rules to classify what occurred and when. You will also audit LLM pre-annotations, correct errors, and document edge cases to help engineers refine the system.
This is not gig-based, scattered annotation work. It involves a consistent queue within one product domain, supported by direct feedback, calibration against gold standards, and advancement tied to accuracy and consistency. If you value clear expectations, measurable quality standards, and work that directly shapes model performance, we would like to hear from you.
What you will be doing
- Annotate student session videos by identifying, classifying, and timestamping behavioral events according to a detailed rubric
- Audit and correct LLM pre-annotations by eliminating false positives, capturing missed events, and refining timestamps
- Document clear reasoning notes for ambiguous decisions, including rubric citations and the assumptions applied
- Record edge cases and clarification questions for uncertain scenarios and maintain an annotation tracker with session metadata
- Participate in calibration exercises, integrate QA feedback, and apply rubric revisions to enhance accuracy continuously
What you will NOT be doing
- Construct AI models, conduct experiments, or perform research on student behavior
- Design the annotation rubric or redefine category definitions according to personal interpretation
- Prioritize speed over accuracy, consistency, or timestamp precision
- Handle one-off, scattered tasks across unrelated domains without context or quality feedback
Key responsibilities
This role ensures that student session videos are converted into ≥95%-accurate, time-precise labeled datasets that reliably indicate when model performance improves or regresses.
Candidate requirements
- At least 1 year of experience in data annotation, content moderation, QA evaluation, or similar rubric-driven review work
- Strong English reading comprehension and the ability to follow complex written instructions without deviating from the rules
- Ability to maintain focus and accuracy for 4–6 hours of video-based work per day
- Ability to detect subtle visual and on-screen behavioral cues and classify them consistently across many sessions
- Strong written documentation skills for explaining edge cases, assumptions, and clarification questions
- Reliable internet connection capable of streaming video
- Comfort reviewing, correcting, and supplementing AI/LLM-generated annotations
