Implementation of section 6.4.2 of Abraham Wald, Sequential Analysis, Dover 2004.

We will phrase this as purchasing web-traffic from two sources in parallel. Each source (or process) either converts (a success) or does not (a failure). Source 1 converts with (unknown) probability p1 and source 2 converts with (unknown) probability p2. We want to buy traffic from both sources for a while and then with high probability pick the source with the higher conversion rate.

In the sequential analysis all traffic is paired. So we observe tuples of the form (c1,c2) where ci=0 if the traffic did not convert and 1 if it did (i being the traffic source). All tuples of the form (0,0) and (1,1) are discarded as being uninformative about the relative values of p1 versus p2. So the experimental summary is how many (0,1) and (1,0) pairs we have observed. Note this is not a sufficient statistic (such as recording the number of items taken from each source and the number of successes from each source), but instead roughly a pivotal quantity that depends only on the relative sizes of p1 and p2 (and not their absolute rates).

The recording procedure is then:

Our decision surface is a pair of parallel lines: