AB testing

A/B testing is a term for a randomized experiment with two variants, A and B, which are the control and variation in the controlled experiment. A/B testing is a form of statistical hypothesis testing with two variants leading to the technical term, two-sample hypothesis testing, used in the field of statistics.

Other terms used for this method include bucket tests and split-run testing but these terms can have a wider applicability to more than two variants. In online settings, such as web design (especially user experience design), the goal of A/B testing is to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). Formally the current web page is associated with the null hypothesis. A/B testing is a way to compare two versions of a single variable typically by testing a subject’s response to variable A against variable B, and determining which of the two variables is more effective.

Examples of possible statistical tests to use:

• Z-tests (assumption of gaussian): comparing means with known variance.
• Welch’s t test (assumption of Gaussian distribution).
• Fisher’s exact test (assumption of Binomial distribution).
• E-test (assumption Poisson distribution).
• Chi-squared test (assumption of Multinomial distribution).
• Mann-Whitney U test (no assumption of distribution).
• Student’s t-tests are appropriate for comparing means under relaxed conditions when less is assumed.

See also

Statistical testing

Material

• scipy.stats
• “Advanced A/B Testing Tactics That You Should Know ; Testing & Usability”. Online-behavior.com.
• “The Beginners Guide To AB Testing”. Marketizator. Retrieved 2014-10-29.
• “A/B Split Testing ; Multivariate Testing ; Case Studies”. Visual Website Optimizer. Retrieved 2015-09-08.
• “A/B Testing Case Studies”. Optimizely. Retrieved 2015-11-24.