2.2: Deepbit and "pool hopping"

0. Introduction

Note: the data this post is based on is here.

Strategic miners or 'pool hoppers' (those who only mine while the expected share value is greater than 100%) that mine at a proportional pool cause full time miners to earn less than their expected payout. The amount full time miners lose depends on how much strategic miners add to the pools base hash rate, so in order to determine how much proportional miners lose an accurate estimation of the so called 'hopper boost' is required.

At Deepbit.net this has been difficult to assess since the published data only include a timestamp of when the block was solved, the time taken to solve the block to the nearest minute and the number of shares is took to solve the block.

P4man initially presented findings at bitcointalk.org that indicated Deepbit had been mined strategically; unfortunately these results were equivocal, probably due to the fact he was using the "time to solve" quoted by the Deepbit.net's statistics page (accurate to the nearest minute) rather than the time stamps, which are accurate to the nearest second  (please correct me if I'm wrong about how you generated your graphs, P4man).

If time intervals correct only to the nearest minute are used inaccuracies from rounding partial minutes makes providing an irrefutable proof of a strategic miner hashrate boost impossible, since Deepbit.net's average hashrate equates to nearly a thousand shares per second. Short rounds which are the most profitable are therefore the rounds for which the least accurate information is available.

1. Is Deepbit.net being mined strategically?

To prove that Deepbit.net is being successfully mined by strategic miners, we can plot average hashrate per round as P4man did, but using the time stamp instead. However there are a few details that make this assessment much harder to assess than for other pools.

1. Statistics are delayed by one hour. This means that strategic miner 'hopping software' must wait for a long poll announcing a solved block, and then use various methods to determine the likelihood that the "block solved" announcement was from Deepbit.net. Then the software must determine when 0.43xD shares have been submitted using the time since starting to mine and the pool's published hashrate (the amount of time it takes to reach 0.43xD at current Bitcoin block solving difficulty, D~1490000 and website quoted hashrate, 3800 Ghps 730 seconds). This could have two effects:     

i. Onset time of the strategic miner hashrate boost may vary;    

ii. The total round shares at which strategic miners stop submitting shares may not be exactly 0.43xD .

2.  Since nearly a thousand shares per second are being submitted to Deepbit.net, inaccuracies resulting from rounding seconds could be a problem over very short rounds - for example lasting a few seconds or so.

3. Miners using long polling will sometimes miss the start of rounds, or even start mining mid-round when misinterpreting a block announcement as coming form Deepbit.net. Anecdotal evidence suggests that most strategic miners on Deepbit.net earn 140 to 150% of expected earnings there, and while the maximum theoretical expected round earnings efficiency for a strategic miner is 178.7% most miners find they earn about 160 to 170% in the long term at other proportional pools. Assuming this loss in expected value comes from incorrectly mining when expected share values are low, then at least 18% of strategic mining is on unprofitable times to mine a round. This does not include losses due to missing a round. All this makes determining the strategic mining hashrate boost and expected payout more complicated.

4. The average pool hashrate varies apart from when the pool is being strategically mined. In the data so far, for long rounds when the effect of the strategic mining boost has been reduced, the pool hashrate has varied from 3600 to 4000 Ghps. If the amount of hashrate added by strategic miners  is assumed to be approximately constant per round, then the effect of strategic mining on full time miners will also vary. 

5. The varying pool hashrate can also make the exact strategic miner hashrate boost difficult to determine.

With these provisos, let's look at the average hashrate per round for the last 1013 rounds at Deepbit.net.

You will immediately note that the results show a trend of shorter round having a higher hashrate. From this simple graph it is possible to show that Deepbit.net is likely to have been mined strategically for the timespan of this data.

2. How much of Deepbit.net's hashrate due to strategic miners?

With such a significant spread of data but such clear trends, an analysis of the rise and fall in hashrate is necessary to determine how much of the pool's overall hashrate is due to strategic miners. From point 4 (above) the pool's long term hashrate changes maybe creating the variation. Secondly since most of the action is happening early, a log x-axis and a loess smoothing curve will help to show average trends.

This graph shows the changes in hashrate more clearly, but colour coding by time hasn't helped much. There are a few noteworthy things:

• There is a clear hashrate boost which slowly reduces after about 0.5xD. This is likely due to strategic miners.
• There is a significant number of shares at below the average pool hashrate early on in a round. Why? 
•  At about 0.2 to 0.5xD there are many rounds at below the higher hashrate. This is most likely due to strategic miner software incorrectly guessing a block announcement from Deepbit.net as being from a different pool, preventing strategic miners software from submitting shares and contributing to the pool hashrate.

The first and last points can be addressed by only considering rounds that had a shorter than 0.43xD immediately preceding it. In this case we would expect the hashrate to be at it's boosted maximum immediately, and even if the round was missed (i.e. the block announcement was not accepted as Deepbit.net's by miner software) the hashrate would be boosted until 730 seconds from the start of share submission.  In the case of a very short preceding round this implies mining for most of the next 0.43xD submitted shares. 

In analysing the earlier, more profitable part of a round, it may also be useful to graph hashrate as a function of round solve time rather than total round shares /D. The next graph only considers rounds that have a round shorter than 0.43xD preceding them, against round solve time.

Now that we've removed most of the short rounds that were missed by strategic miners, the trend is much clearer. The average hashrate of a round increases until rounds are about 170 seconds long, then remains stable for the next 490 seconds  after which the average round hashrate begins to decline.

The lower than average hashrate for very short rounds should also be noted. This implies that whatever is causing the reduced average hashrate at the start of the round is affecting all miners equally, not just strategic miners.

Before we can assess the effect of pool hoppers on the average full time miner, we need to answer two questions: 

• Why is there consistently lower than average hashrate at the start of a round?
• What is the actual hashrate increase from the strategic miners rather than the round average increase?

These two questions can be solved together by modelling the estimated actual hashrate, performing an average and comparing the model to the actual recorded hashrates. I considered several mechanisms (or combinations of them) to model:

1. There exists a short period of time at the start of a round during which no shares are submitted, reducing the average hashrate when shares are finally submitted. Strategic miners are assumed to be able to submit as soon as full time miners if they submit shares at all.

2. As per 1, but with strategic miners unable to start the round as soon as full time miners due to inefficiencies in long polling. This option seems the least likely since from the graph above shows that even rounds starting with a strategic miner boost show a reduced hashrate early on.

3. All miners can submit shares at the start of the round.  However, notification of the new block means the miner will drop its current work and start on the new work. This will cause a decrease in overall pool pool hashrate until all miners have submitted work on the new round. An additional effect may be due to stale shares submitted at this time. 

4.  As per 3, but with a gradual tapering of hashrate increase.

The graph below illustrates  these options. The grey shaded area represents the estimated actual hashrate, and the black line the averaged hashrate.

The graphs below compare these models up to 10000 second rounds with the data. Note that the log x-axis makes the vertical lines of option 2 appear slanted, and the straight linear onset of option 3 appear slightly curved. 

After 600 seconds the gradual decline in average hashrate per round is caused by an average of the pools base hashrate with the previously higher hashrate caused by the strategic miners, so this was added to the model. 

The fourth model most closely fits the data (p<0.0001). A linear onset time for all miners did not fit well, and rather than a linear onset, it appears the hashrate increase may be described by some probability distribution. Testing this hypothesis, a log-normal distribution with a mean of log(2.01) and standard deviation of log(5.20) seconds for the estimated actual pool hashrate fit the data up to 500 seconds quite well. I cannot explain why the estimated actual hashrate increase can be determined by a log-normal distribution with these parameters, so I'm simply using it as a function of convenience since it matches the polynomial fitted loess curve quite well. 

Below I've repeated the third chart, with lines representing the chosen model at various boosted hashrates against averaged round times. In all cases, the baseline pool hashrate is 450 Ghps less than the maximum. Equivalently, the model shows good agreement with the data if a strategic miner boost of 450 Ghps over a varying base hashrate is assumed. 

Another comparison between the model's assumed real time hashrate including the strategic miner boost, and below that the round averaged hashrate representing what is detectable when plotting averaged round times against hashrate, up to 10000 seconds.

3. How much do full time miners lose to strategic miners at Deepbit.net?

Since the model seems to fit the data so well (p<0.05), we can assume that this function is close to representing the average pool hashrate over time. It is a constant function from the start, so we can also assume that there is no significant difference between onset times for fulltime miners and strategic miners when the previous round was shorter than 0.43xD. 

Overall, strategic miners represent an essentially constant fraction of the pool hashrate until 0.43xD. Therefore, strategic miners make up the following proportion of the overall pool hashrate (when mining) as follows below. 

NB: these figures should not be considered an exact estimate as they are based on visual analysis of the graphs.

This is important since the boost proportion of the hashrate is how calculate how much fulltime miners lose to strategic miners is determined.  However, anecdotal evidence suggests strategic miners using long polling to find the start of rounds earn a long term average of 145% of expected earning, much less than the real world earnings long term average of a strategic miner at a proportional pool that doesn't delay publishing round which is about 165%. If we assume that this reduction represents times that strategic miners missed the early profitable parts of a round, then we can reduce the average earnings by the pool hoppers. 

Pool base hashrate, Ghps	Strategic miner boost, Ghps	Strategic miner boost, percentage	Fulltime miners' expected round earnings 1	Fulltime miners' expected round earnings 2	Fulltime miners' expected round earnings 3
4500	450	10.00%	96.90%	97.20%	94.28%
4150	450	10.84%	96.70%	97.00%	94.09%
3800	450	11.84%	96.30%	96.80%	93.90%

The values above were calculated using a simulation.

"Fulltime miners' expected round earnings 1",  lists the percentage of expected earnings a full time miner will receive given maximum strategic miner theoretical efficiency.

"Fulltime miners' expected round earnings 2",  lists the percentage of expected earnings a full time miner will receive given estimated real world  strategic miner efficiency.

"Fulltime miners' expected round earnings 3",  lists the percentage of expected earnings a full time miner will receive given estimated real world  strategic miner efficiency, and including the 3% fee Deepbit.net charges to it's proportional miners.

4. Is Pay-Per-Share worthwhile yet?

At the end of post 2.1 Deepbit.net I discussed the variance of full time proportional miners compared to Pay-Per-Share (PPS) rewards, and showed that after only 562 rounds (or 11 days at 3800 Ghps) a proportional miner would, ninety-five times in one hundred,  have earned the same as or more than a PPS miner. Is this still the case?

Obviously, in the long term, a proportional pool miner will earn the more than a PPS miner. But how long a term does it need to be when strategic miners are present? Using an average of 94% from the left most column of the table above, the necessary number of rounds for a proportional miner to be likely to have earned the same as a PPS miner can be calculated the same way.

> sd.clt<-1/sd(x[,5])
> norm.clt<-function(x){
+ abs(pnorm(0.9,mean=0.94,sd=sd.clt/sqrt(x))-0.05)
+ }
>
> optim(1,norm.clt,method='Brent',lower=1,upper=2000)
$par
[1] 1546.412

Three times as many rounds are required to reduce a proportional miner's variance to near PPS payouts 95 times in 100. The graphs below show the difference clearly.

5. Conclusions

• Strategic miners are successfully adding 450 Ghps to Deepbit.net's overall hashrate.
• Due to the presence of strategic miners, full time miners can expect to earn only 94% of expected round earnings,not 97% (including fee).
• This loss in income means that it actually takes over 30 days (at current pool hashrate) for fulltime proportional miners to be confident of earning the same as or more than PPS miners, rather than 11 days without strategic miners present.

organofcorti's Second Rule of Pooled Bitcoin Mining: Avoid proportional payouts unless you're mining strategically.

Thanks go out to P4man at bitcointalk.org for reporting this initially.

Donations help give me the time to investigate pools and write these posts. If you enjoy or find them helpful, please consider a small bitcoin donation:
12QxPHEuxDrs7mCyGSx1iVSozTwtquDB3r