bayesTest wrapper

Wrapper to run multiple bayesian A/B tests on lists of A_data, B_data and priors.

runABbayesTest(
  topScorerHitRate,
  topScorerSampleSize,
  lowScorerhitRate,
  lowScorerSampleSize,
  priors,
  distribution = c("bernoulli", "bernoulliC"),
  nSimulatedData = 10,
  n_samples = 1e+05
)

Arguments

topScorerHitRate

Numeric, hit rate for the top scorer

topScorerSampleSize

Named numeric vector, group size for the top scorer

lowScorerhitRate

Numeric, hit rate for the low scorer

lowScorerSampleSize

Named numeric vector, group size for the low scorer

priors

Named vector or named list providing priors as required by the specified distribution:

  • For 'bernoulli' distribution list("alpha" = val1, "beta" = val2)

  • For 'normal' distribution c("mu" = val1, "lambda" = val2, "alpha" = val3, "beta" = val4)

  • For 'lognormal' distribution c("mu" = val1, "lambda" = val2, "alpha" = val3, "beta" = val4)

  • For 'poisson' distribution c("shape" = val1, "rate" = val2)

  • For 'exponential' distribution list("shape" = val1, "rate" = val2)

  • For 'uniform' distribution c("xm" = val1, "alpha" = val2)

  • For 'bernoulliC' distribution: same prior definitions as 'bernoulli'

  • For 'poissonC' distribution: same prior definitions as 'poisson'

See plotDistributions or the Note section of this help document for more info.

distribution

of underlying A/B test data. See more in details

nSimulatedData

number of trials for the simulated data

n_samples

Number of posterior samples to draw. Should be large enough for the distribution to converge. 1e5 is a good rule of thumb. Not used for closed form tests.

Value

Returns bayesTest results in a tidy table

topScorerSampleSize

numeric - top-scorer sample size

lowScorerSampleSize

numeric - low-scorer sample size

probability

numeric - Probability of top-scorer hit rate to be better than low-scorer hit rate P(topScorerHitRate > lowScorerhitRate)

posteriorTopScorerData

list - distribution over the posterior for top-scorer (if available)

posteriorLowScorerData

list - distribution over the posterior for low-scorer (if available)

Details

  • Only bernoulli & bernoulliC distribution are supported for now but the user can easily add a tidyBayesTest S3 method for the distribution that suits better their data. The distribution that can be implemented are the one available in the bayesAB package see bayesTest.

  • Bernoulli can be used if the data is binary, modeled by 1s and 0s, according to a specific probability p of a 1 occurring. The conjugate Beta distribution for the parameter p should to be used.

  • bernoulli is using Monte Carlo sampling to estimate the posterior probability while BernoulliC is computing the posterior probability analytically (faster but doesn't report the distribution over the posterior)