LLM AI Quality Analyst (Personalization)
Employment Type: Short-Term Contract (2 Months)
Work Model: Remote
Start Date: Immediate
Open Positions: 50
Experience Required: Minimum 1 Year
Compensation: USD $15 per hour
Schedule: Full-time (3040 hrs/week) with minimum 4-hour overlap with PST
Role Overview
We are seeking professionals to evaluate AI-generated personalized responses. In this role, you will assess how effectively AI uses contextual and personal data to deliver relevant, accurate, and helpful responses. This position combines analytical evaluation with creative prompt design to test personalization quality.
Key Responsibilities
- Design and execute multi-turn conversational prompts based on personal context
- Evaluate AI-generated responses for accuracy, relevance, and personalization quality
- Compare model responses side-by-side (SxS) and rank effectiveness
- Identify grounding issues, incorrect inferences, or misleading outputs
- Assess integration of personal data for natural and helpful responses
- Provide structured feedback and annotations to improve AI performance
- Maintain strict data handling and confidentiality standards
- Ensure evaluation data hygiene and compliance with project guidelines
Required Qualifications
- High proficiency in reading and writing English
- Minimum 1 year experience in customer support, content moderation, AI evaluation, or similar role
- Strong analytical thinking and decision-making ability
- Experience evaluating nuanced or ambiguous digital content
- Willingness to use primary personal Google account with enabled data sources
- Full-time availability with PST time zone overlap
- Reliable laptop/desktop and stable internet connection
- Ability to work independently in a remote environment
Preferred Background
- Experience in data annotation, AI quality evaluation, or content review
- Familiarity with digital support environments (chat, email, platforms)
- Experience handling escalations or customer-facing quality review tasks
- Bachelor's degree (BS/BA) or equivalent experience in a relevant analytical field
Key Skills
- Personalization quality evaluation
- Prompt design and conversational testing
- Analytical reasoning and structured feedback writing
- Attention to detail and quality judgment
- Clear written communication
Offer Details
- Contractor engagement (2 months)
- 30 or 40 hours per week options
- Minimum 4 hours daily overlap with PST
- Global 24-hour operations team
Selection Process
- Job Interest Form
- Online Assessment (complete within 24 hours)
- Pre-onboarding discussion
