Etsy
At Etsy, we leverage our internal A/B testing tool when we launch new campaigns, polish the look and feel of our site, or even make changes to our search and recommendation algorithms. As our experimentation platform scales and the velocity of experimentation increases rapidly across the company, we also face a number of challenges. In this talk, I will talk about how we utilize Julia to investigate and evaluate one of the problems, “peeking” at results early in order to detect maximum significance with minimum sample size. We used Julia to assess the overall problem and how it affected experiments at Etsy. We also used Julia to evaluate a few solutions that have been proposed and applied in industry and academia, keeping in mind the unique challenges we face as a fast-paced e-commerce company. After going through the analysis and evaluation, I will discuss the approach we at Etsy took to tackle the peeking problem.
Kelly Shen is a data engineer at Etsy, where she works on improving the e-commerce company’s in-house A/B testing platform. Previously, she was an undergraduate student at MIT studying computer science and mathematics and used Julia extensively in both her classes and research.
