Scalable A/B Testing in Big Tech: Inference, Limitations and Pitfalls
Facebook Event Link: https://www.facebook.com/events/800991147120402
Evan Chow is an experienced data scientist (MSc EME ‘21) who has worked in data science at various Silicon Valley companies including Snap and Uber. At Snap, he joined their first data team to drive forward A/B experimentation, causal inference, and data analytics at scale. Before that, he worked on driver-rider matching algorithms and rider pricing at pre-IPO Uber. Evan is a Princeton graduate and a (soon-to-be) published member of ACM SIGIR.
Recap
-
Talked about A/B Testing, aka. "Randomised Control Trials" in the language of Econometrics / Biostatistics
-
In addition to statistical significance and effect size, in the business setting their are business constraints (controversay, cost)
-
A/B testing - allows for the estimation of $X$ on $Y$, in a controlled setting
-
A/B testing allows for thinking about counterfactuals, iterative rollout in a business setting,
-
A/B testing shortcomings - effects are probabilistic, not deterministic (i.e. might get a different magnitude of results in subsequent experiments); there is no economic explanation beyond X affects Y, if this relationship exists.
-
Interesting mention of network effects of treatment and control. e.g. in social networks) efffect on the treatment might spread to the control groups
-
Careers Advice: one impressive side projects is a good way to get noticed, open source contributions. Get domain knowledge.