Lead Reviewer - Egocentric Video Annotation (Physical AI / Robotics)

shift · Bangalore, India

About shift

shift pays everyday workers to record short videos of the tasks they already do (in kitchens,

workshops, job sites, bakeries, and warehouses) to help train the next generation of AI and

robotics. We’ve paid out over $15M to 25,000+ workers across 15+ countries.

About the role

You're the standard other reviewers calibrate against. You audit egocentric video annotation -

segmentation, per-hand text labels, hand tracking, object identification - and judge whether the

work on camera is real. You still annotate at gold standard yourself. When an edge case has no

answer yet, you decide, and you write it down so nobody decides it twice.

What you'll do

• Audit annotation labeling output - in-house and vendor - and own the accept/reject call
• Own the fake-versus-real call: spot staged, faked, or non-genuine task footage, and sharpen the criteria that let others spot it too
• Review machine-generated hand tracking - keypoint accuracy, left/right attribution, hand-object contact - and flag systematic failure modes
• Annotate at gold standard; build and document the golden clips others are measured against
• Rate and calibrate annotators; give feedback specific enough to move their scores
• Adjudicate escalated edge cases and turn each ruling into a rubric or SOP update
• Lead calibration sessions and onboarding for new reviewers
• Surface disagreement patterns and quality trends to the Annotation Quality Lead



What we're looking for

• 6+ years of hands-on video annotation, with meaningful time on egocentric/first-person video
• 4+ years reviewing: auditing others' work, adjudicating disagreements, holding a defined accuracy bar
• Annotation good enough to pass your own audit
• Outstanding written English and communication. Your labels are commands a model learns from, and your feedback has to land with the person receiving it
• You can explain what a 99% versus 95% bar means in practice - and work to it
• Experience contributing to rubrics, SOPs, or edge-case documentation
• Judgment on ambiguous segments, including the discipline to let a borderline pass
• Bonus: You've trained or calibrated annotators, or led a small review team
• Bonus: Robotics, computer vision, or hand/gesture tracking exposure