It takes powerful technology to connect our brands and partners with an audience of hundreds of millions of people. Whether you’re looking to write mobile app code, engineer the servers behind our massive ad tech stacks, or develop algorithms to help us process trillions of data points a day, what you do here will have a huge impact on our business—and the world.

A Little About Us

Our team is at the forefront of redefining the shopping research experience across Yahoo’s global ecosystem. Our team owns the AI-driven product index, evaluation infrastructure, and LLM-powered recommendation systems which form the foundation of Yahoo's smart shopping assistant project. We sit at the intersection of vector databases, data ingestion, and LLM-based product rankings.

About the Role

You are the technical owner of three pillars: the product index that powers shopping discovery, the evaluation systems - both LLM-as-judge and code-based deterministic ones - that hold our recommendation quality bar, and the prompt and LLM architecture decisions that turn those quality signals into shipped improvements. You work at the seam between retrieval, generation, and measurement, building the infrastructure that decides what Yahoo recommends and how we know it's working. You operate with a high level of technical independence as an individual contributor, owning metrics and QA across the entire project.

Responsibilities

- Product Index Ownership - Design, maintain, and scale the product catalog that powers data retrieval, search, and recommendation flows across Yahoo's commerce surfaces. Ensure the index remains accurate, fresh, performant, and comprehensive.
- LLM & Code-Based Evals - Build and maintain LLM-as-judge evaluators in Langfuse, and author deterministic code evals that catch failures before changes make their way to production, as well as monitor production quality.
- Metrics & QA - Define what "good" looks like, instrument it across the stack, and surface quality regressions through tooling.
- Prompt & LLM Architecture Quality - Turn eval signals into action. Translate quality measurements into concrete modifications to prompts, retrieval pipelines, and the broader recommendation architecture.
- AI-Augmented Development - Master AI-assisted development workflows to rapidly prototype, ship, and validate eval and indexing infrastructure.
- Technical Partnership - Partner closely with the project lead and broader squad, providing strong technical opinions on eval design, retrieval architecture, and quality measurement.

Minimum Qualifications

- Experience - 3+ years of shopping recommendation system experience, with a proven track record of shipping production backend or ML infrastructure.
- LLM & ML Expertise - Extensive hands-on experience working with LLMs and traditional ML systems in production, with a clear point of view on what works, what fails, and why.
- LLM Observability & Evals - Deep experience with Langfuse or similar LLM observability tooling, and demonstrated ability to build both LLM-as-judge and code-based evaluation pipelines.
- Product/Catalog Data Experience - Shipped against real-world product or content catalogs at scale.
- AI-Forward Mindset - Strong, positive, and curious outlook on applying LLMs to both product surfaces and engineering workflows. Comfortable using AI-assisted development tools to move faster without sacrificing quality.

Preferred Qualifications

- Backend Languages - Comfort in Python and Go.
- Agent Frameworks & Eval Pipelines - Prior experience designing and shipping an end-to-end agent framework or LLM evaluation platform.
- Vertical Expertise - Prior experience building high-traffic shopping, e-commerce, or product discovery platforms.
- Observability - Proficiency in monitoring tools such as Splunk, Datadog, or Prometheus.
- Mindset - A product-focused approach to engineering with a commitment to measuring what matters and using AI as a tool to maximize personal and team impact.

Yahoo is proud to be an equal opportunity workplace. All qualified applicants will receive consideration for employment without regard to, and will not be discriminated against based on age, race, gender, color, religion, national origin, sexual orientation, gender identity, veteran status, disability or any other protected category. Yahoo will consider for employment qualified applicants with criminal histories in a manner consistent with applicable law. Yahoo is dedicated to providing an accessible environment for all candidates during the application process and for employees during their employment. If you need accessibility assistance and/or a reasonable accommodation due to a disability, please submit a request via the Accommodation Request Form (www.yahooinc.com/careers/contact-us.html) or call +1.866.772.3182. Requests and calls received for non-disability related issues, such as following up on an application, will not receive a response.

Yahoo has a high degree of flexibility around employee location and hybrid working. In fact, our flexible-hybrid approach to work is one of the things our employees rave about. Most roles don’t require specific regular patterns of in-person office attendance. If you join Yahoo, you may be asked to attend (or travel to attend) on-site work sessions, team-building, or other in-person events. When these occur, you’ll be given notice to make arrangements. 

If you’re curious about how this factors into this role, please discuss with the recruiter.

Currently work for Yahoo? Please apply on our internal career site.