. (mean + 2*stdev)
end
end
samples1.confidence_interval # => 7.92941515120981..12.4705848487902
samples2.confidence_interval # => -3.39084659287681..23.7908465928768
* It is reasonable to assume a normal distribution here. We can safely treat series of server response times as
i.i.d. random variables; therefore, by the central limit theorem, the distribution will converge to normal given
enough samples.
Measurement Tools | 151
We see that the server that produced the first set of samples will usually (95% of the
time) respond in between 8 and 12 ms. On the other hand, the data from the second
server varies so wildly as to be nearly meaningless.
When comparing two data sets, it is important to compare not just their means but
their standard deviations and confidence intervals as well. It may look as if you have
made an improvement, but if the confidence intervals overlap by a significant
amount, there is no statistical significance to the result.
Black-box analysis with httperf
Now that we know how to analyze results, we can benchmark a site. The best reference
is Zed Shaw??™s instructions about tuning Mongrel using httperf (http://mongrel.
rubyforge.org/docs/how_many_mongrels.html). We will not repeat the procedure
here, but we will give some caveats:
??? Ensure that you have the front end server (usually Apache, lighttpd, or nginx)
configured to serve static files.
Pages:
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235