← All talks · Keynote
Search, Recommenders and Co-pilots: Evolution of Evaluation Strategies in the Industry
FIRE 2023 — Industrial Track Keynote ·Goa, India ·December 15, 2023
Abstract
Industrial-track keynote at the Forum for Information Retrieval Evaluation.
Traced the evolution of evaluation methodology across search, recommendation,
and emerging LLM-assisted co-pilot products — from offline relevance metrics
to online multi-objective optimization and, increasingly, hybrid evaluation
regimes for generative systems. Drew on production experience scaling
recommenders at ShareChat from 1M to 200M users.
Links & tags
recsysevaluationindustry
Notes:
- Audience: information-retrieval and recsys research community + industry
practitioners attending FIRE.
- Argument: evaluation is the most underrated lever in production ML systems.
As systems shift from ranking to generation, the measurement substrate has
to shift with them — not just new metrics but new experiment shapes.
- Companion resources: the slide deck linked above.