Performance Split A-B Testing with PageSpeed
OverviewYou can manually test page speed, but this this feature allows you to run automated experiments where segments of your traffic get the page rewritten with different settings in order to figure out which filters work best for your site. It reports to your Google Analytics account, storing data in a custom variable.
To run an experiment you must set several options in the configuration file. First, you turn on this feature and tell it your Web Property ID :
pagespeed RunExperiment on pagespeed AnalyticsID UA-XXXXXXX-Y
This will enable several filters needed for experiments including the Insert Google Analytics filter to insert a tracking snippet into each page. If you already have a tracking snippet on your pages, you should remove it so that the one PageSpeed inserts is the only one.
To disable experiments and experiment tracking you can set RunExperiment to 'off':
If you still want PageSpeed to automatically insert the Google Analytics tracking snippet, you need to enable the insert_ga filter:
pagespeed RunExperiment off
pagespeed EnableFilters insert_ga
Once you have turned on RunExperiment and set your Google Analytics id you can set up an experiment. For example, to test how much PageSpeed is speeding up your site you can apply optimizations for only half your visitors:
pagespeed ExperimentSpec "id=1;percent=50;default" pagespeed ExperimentSpec "id=2;percent=50"
ExperimentSpec defines an experimental treatment
or specification with a semicolon-separated list of settings. Some settings are
- A positive integer, unique across all experiments. You can't reuse ids from one experiment to the next; every id must be unique. If you stop running an experiment that had three experiment specs 1, 2, and 3, then your next experiment should start with id 4 or higher.
An integer between 0 and 100 indicating what
fraction of users should receive this treatment. If the sum
percentover all treatments is less than 100 then the remaining users will be assigned to 'no experiment' and continue seeing what they usually would. If the sum exceeds 100 it will report an error when parsing the config file.
ExperimentSpec 1 above adds the optional setting
below, which tells PageSpeed to apply just the filters and settings
it would normally apply. ExperimentSpec 2 specifies nothing and so turns off
_gaq.push(['_setCustomVar', 1, 'ExperimentState', 'Experiment: 1']); _gaq.push(['_setCustomVar', 1, 'ExperimentState', 'Experiment: 2']);
In addition to
percent described above, there
are additional optional settings intended to give you a way to test many
settings you could set in the configuration file:
Apply the filters and settings that would
normally apply. Warning: prior to IISpeed 2.0 / 220.127.116.11 this setting could not be
used in combination with any others. It would override any other
configuration specified in the experiment spec, and the other settings
would be silently ignored. This meant that
enabled=remove_commentswould have no effect. As of 18.104.22.168, however, it is now safe to combine
options=, and it indicates that the current non-experiment options should be used as a starting point. For example, in
default,enabled=remove_comments, you would now get the expected result of server default filters and options, plus
remove_comments. Note that
level=may still not be used with
While all the other settings are in the form
key=value, as in
percent=42, this setting has no 'value' and is just
A comma-separated list of filters explicitly
enabled. For example,
enabled=rewrite_images,inline_js. Equivalent to
A comma-separated list of filters explicitly
- A comma-separated list of options to set. For example, options=JpegNumProgressiveScans=5,WebpRecompressionQuality=72.
The Google Analytics ID you would like the data of
this experiment reported to. If not specified, this experiment will use
the Analytics ID specified with
Google Analytics provides five slots for custom
variables. Here you specify which custom variable slot you would
like to use. The default is slot 1, but you can change this setting
Visitors are assigned to ExperimentSpecs on a week-to-week basis. This means that when you finish one experiment and start another, it will take up to a week for repeat visitors to be assigned to the new experiment.
Starting in IISpeed 2.0 / PageSpeed 22.214.171.124 you can test experiment configurations on a live site
before assigning anyone to them by creating them initially with a percentage of
'0%' and then visiting the page
?PageSpeedEnrollExperiment=<experiment_id>. This will
set a cookie that assigns you to the group you're trying to test. You can stop
running this test by deleting the
PageSpeedExperiment cookie or
loading the page with
Run an experiment on 30% of visitors where half get the default configuration and half get no filters. Use Google Analytics ID UA-XXXXXXX-Y to run the experiment, and default to logging the experiment information into custom variable slot 1:
pagespeed RunExperiment on; pagespeed AnalyticsID UA-XXXXXXX-Y pagespeed ExperimentSpec "id=1;percent=15;default" pagespeed ExperimentSpec "id=2;percent=15
Run an experiment on 30% of visitors where:
- One tenth of visitors get the default configuration.
One tenth get a new configuration which has the core filters
- One tenth get no filters.
- The remaining 70% aren't included in the experiment.
While the RunExperiment feature supports testing many changes, as shown above, the discipline of changing only one variable at a time helps you see the effect and know where it's coming from. The only drawback is that sometimes two filters are more useful in combination, such as minification and inlining, where minifying brings resources down to the inlining threshold.
There is also a tradeoff with the experiment percentages: a larger percentage means you will collect sufficient data for meaningful results more quickly but it also affects more of your users. If you're testing manually with query parameters to be sure your settings aren't making things worse for you, the downside of running with large experiment percentages is low. While you may not have determined the ideal settings right away, you're unlikely to have made things much worse.
After you have been running an experiment for 24 hours Analytics will have results for you. The screenshots below walk through the process of creating a custom report to view them.
Set up advanced segments
- Advanced segments let you label fractions of your traffic. To view speed results broken down by experiments you need to add segments for each experimental group. Click here to import a segment for a sample experimental group.
If prompted, log into Google Analytics:
Select the profile that your experiment data was collected under, enter
a name for the segment, and click
On the next screen you can adjust this custom segment to fit your
situation. If you used a custom variable slot other than the default,
which is 1, change Custom Variable (Key 1)
and Custom Variable (Value 01) to the appropriate
values. Where it says Experiment: 1 replace 1 with the
experiment id you used. If you are using version 126.96.36.199 or later, you
also need to change FuriousState
- Click Save Segment.
- Repeat these steps for the rest of your experiment ids.
Add advanced segments to the speed report
- Click All Accounts.
On the dropdown menu click the name for your profile. In the screenshot
below this is labeled your profile name [DEFAULT].
Click Standard Reporting:
Under Content click
and then Page Timings to bring up
the Site Speed Page Timings page:
- Click Advanced Segments.
Under Custom Segments you will see the segments you added
above. Check the boxes next to them and click Apply:
- This takes you to back to the Site Speed Page Timings page.
Examine experimental results
The Explorer tab shows average timing
information both overall and, below, per-url:
Averages can be misleading, however, because one user who suffers
abnormally high latency before the onload event will have an outsize
effect on the average. Unless your sample is very large, such users can
make an experiment falsely appear to be a success or failure. To overcome
this problem a histogram is helpful. Click
the Performance tab to see page load times grouped into
- To see more fine grained timing detail, click the + icon next to any histogram interval to expand it.
- In this experiment PageSpeed appears to have moved about 8% of visits from the 1-3 second category to the 0-1 second one. For a simple mostly-text site running only the core set of filters, this is a reasonable result. As a next step the webmaster of this site might examine the optimizations list to determine if there are filters that are not in the core set which might be good targets for future experimentation.
- The Explorer tab shows average timing information both overall and, below, per-url:
When a visitor first arrives on a site with this feature enabled PageSpeed
chooses an experiment from among the ExperimentSpecs. If you are running your
experiments on less than 100% of your visitors, some will be assigned to "no
experiment", which PageSpeed represents as
id=0. It generates the
page in accordance with the ExperimentSpec and then sets a cookie valid for one
week, so that on future page loads it knows which ExperimentSpec to apply to
Date: Thu, 10 May 2015 14:19:43 GMT Server: Microsoft IIS .. Accept-Ranges: bytes Set-Cookie: PageSpeedExperiment=3; Expires=Thu, 17 May 2015 14:19:43 GMT; Domain=.www.iispeed.com; Path=/You can test this on your site with the
$ curl -D- -o /dev/null http://yoursite ... Set-Cookie: PageSpeedExperiment=1; ... ... X-IISpeed: ... ...
If you don't see a
Set-Cookie header in the output, this feature
isn't set up properly. If you don't see an
_gaq.push(['_setCustomVar', 1, 'ExperimentState', 'Experiment: 4']);
While it does try to detect when a page already is using Google Analytics and
add to it only what it needs, this is imprecise and can get it wrong. We
recommend that you let PageSpeed insert the only snippet. If you need to
, for example to set
for your site, you can do this by adding to the
<head>. The code to do this for setting the site
sampleRate to 80% would be:
_gaq = _gaq || ; _gaq.push(['_setSampleRate', '80']);