AI Preference Ranking Evaluator - Remote

About the Role

We're hiring an AI Preference Ranking Evaluator - Remote to join our innovative team at Moonlight. This role involves collaborating with a leading AI lab to enhance preference ranking models for conversational AI systems. As an AI Preference Ranking Evaluator, you'll play a crucial role in assessing and ranking AI-generated responses based on quality, coherence, and helpfulness.

What You'll Do

• Evaluate and compare AI-generated responses across various domains, ensuring high-quality outputs.
• Assign preference rankings to pairs or sets of model outputs, following detailed labeling guidelines.
• Provide brief written explanations for your ranking decisions when required.
• Flag edge cases or inconsistencies in task design or model output to improve the evaluation process.
• Maintain strong attention to detail and consistency across repetitive tasks.
• Adapt to evolving guidelines and work comfortably with ambiguity.
• Commit to a flexible workload of 10–20 hours per week, setting your own hours in a remote and asynchronous environment.

Requirements

• Prior experience in data labeling, content moderation, or preference ranking tasks is preferred.
• Excellent critical thinking and reading comprehension skills.
• Strong attention to detail and ability to maintain consistency across tasks.
• Availability for regular part-time work on a weekly basis.
• Comfortable working with evolving guidelines and ambiguity.

Nice to Have

• Experience in conversational AI or related fields.
• Familiarity with AI evaluation metrics.
• Ability to work independently and manage time effectively.

What We Offer

• Competitive pay of $25–35/hour depending on experience and location.
• Payments issued weekly via Stripe Connect.
• Flexible workload depending on your availability and performance.
• Remote work with the freedom to set your own hours.
• Engagement as an independent contractor with a focus on work-life balance.