Senior Model Evaluation Engineer - AI/​ML Focus

About the Role

We are looking for a Senior Model Evaluation Engineer to join our team at xAI. This Senior Model Evaluation Engineer remote position is an exciting opportunity to work on cutting-edge AI systems that aim to understand the universe and aid humanity. You will be part of a small, highly motivated team focused on engineering excellence and innovation.

What You'll Do

• Conduct comprehensive assessments of AI models to ensure their quality and performance.
• Deep dive into model training and data to identify weaknesses revealed during evaluations.
• Collaborate with modeling and data teams to develop plans for improving model quality.
• Build infrastructure and frameworks for user-friendly model evaluation, utilizing inference frameworks like SGlang and vLLM.
• Develop model assessment and evaluation tasks, including public and in-house benchmarking.

Requirements

• Strong experience with Python, JAX, XLA, Rust, or C++.
• Proven track record in model evaluation and assessment.
• Ability to synthesize data for new evaluations and benchmarks.
• Excellent communication skills to share insights and collaborate effectively with team members.
• Experience in building evaluation frameworks and infrastructure.

Nice to Have

• Familiarity with Spark and large-scale data processing.
• Experience in AI/ML model development and deployment.
• Knowledge of best practices in model evaluation and benchmarking.

What We Offer

• Competitive salary ranging from $180,000 to $440,000 annually.
• Equity options as part of the compensation package.
• Comprehensive medical, vision, and dental coverage.
• Access to a 401(k) retirement plan with employer contributions.
• Short and long-term disability insurance and life insurance.
• Relocation support for candidates willing to move to the Bay Area.
• A flat organizational structure that encourages initiative and leadership.