PageSpeed requires publicly cacheable resources to provide maximum benefit. As discussed in the "Cache Extender" filter, the origin TTL specified in the server configuration file dictates how quickly changes made to the source can propagate to users' browser caches. However, using PageSpeed, resources referenced statically from HTML files will be served with a one-year cache lifetime, but with a URL that is versioned using a content hash.
Configuring Server-Side Cache for PageSpeed
In order to rewrite resources, PageSpeed must cache them server-side. A file-system based cache is always employed on each server. memcached may be used as a scalable network-accessible cache in addiiton to the file cache. The file cache is always required since files larger than 1Mb cannot be stored on memcached, and it is important to optimize large images. Finally, a per-process in-memory LRU cache and/or an interprocess shared-memory cache can be configured for rapid access to small objects.
Configuring the File Cache
PageSpeed must be configured with a path where it can write cache files, tuned to limit the amount of disk space consumed. On Linux, the file cache can be placed on a tmpfs partition or on a physical disk. The file cache has a built-in LRU mechanism to remove old files, targeting a certain total disk space usage, and a certain interval for the cleanup process. An example configuration is:
# Note that the configured path must be read/writeable for the IIS application-pool identity! pagespeed FileCachePath c:\PagespeedCache pagespeed FileCacheSizeKb 102400 pagespeed FileCacheCleanIntervalMs 3600000 pagespeed FileCacheInodeLimit 500000
It is important to note that
FileCacheInodeLimit do not define absolute limits
on the cache size and inode count. The cache cleaning process will run at
the time interval defined by
FileCacheCleanIntervalMs, and will only initiate
cleaning if the cache size exceeds
FileCacheSizeKb or the cache inode count exceeds
FileCacheInodeLimit. When cache cleaning is
initiated, the oldest files in the cache will be removed until the cache
size is under
0.75 * FileCacheSizeKb and the
inode count is under
0.75 * FileCacheInodeLimit.
PageSpeed previously reserved another file-path for future use as a shared
database in a multi-server environment. This is no longer in the plan,
GeneratedFilePrefix now generates a deprecation warning.
Configuring the in-memory LRU Cache
To optimize performance, a small in-memory write-through LRU cache can be
instantiated in each server process. Note that in Apache's pre-fork mode
this means dozens of processes, so the total memory consumed
LRUCacheKbPerProcess * num_processes) must fit into the
capabilities of the host machine. Nginx typically runs with many fewer
processes, so a larger
LRUCacheKbPerProcess is appropriate
LRUCacheByteLimit is the limit on how large a
cache entry the LRU cache will accept. A sample configuration is:
pagespeed LRUCacheKbPerProcess 1024 pagespeed LRUCacheByteLimit 16384
pagespeed LRUCacheKbPerProcess 8192 LRUCacheByteLimit 16384
Configuring the Shared Memory Metadata Cache
Note: Deprecated as of IISpeed 2.0
As part of its operation, PageSpeed stores summaries of how to apply optimizations to web pages as part of a metadata cache. Metadata entries are small and frequently accessed. They should ideally be stored in local memory and shared across server processes, as opposed to on disk or a memcached server (see below). That is exactly what the shared memory metadata cache does, and it is the best place to cache your metadata entries.
If this cache is enabled, metadata will no longer be written to the filesystem cache, significantly improving metadata cache performance, but metadata information will be lost upon server restart. This will require resources to be reoptimized for each restart. If a memcached cache is available, cache entries will be written through to the memcached cache so that multiple PageSpeed servers can share metadata and the metadata cache will survive a server restart.
This feature is enabled using
CreateSharedMemoryMetadataCache directive. This directive
takes two arguments. The first is the exact string given as the argument
FileCachePath in any virtual host where you want this cache
active. The second is the size of the cache in kilobytes. Unlike the LRU
cache, this cache is shared among all server processes, so far larger values
are possible. For example:
pagespeed FileCachePath /var/cache/pagespeed/ pagespeed CreateSharedMemoryMetadataCache /var/cache/pagespeed/ 51200
You can see how effective this layer of cache is at the PageSpeed statistics page, where at the bottom of the page every shared memory cache will be listed, including in particular information on its hit rate and how full it is (blocks used).
To enable memcached, specify the list of memcached servers with a comma-separated list of hosts and ports. If the port is omitted, the default memcached port of 11211 is assumed. PageSpeed's memcached integration by uses a background thread for communicating with the memcached servers. This allows PageSpeed to batch multiple Get requests into a single MultiGet request to memcached, which improves performance and reduces network round trips.
pagespeed MemCachedServers "host1:port1,host2:port2,host3:port3"
When you use memcached with PageSpeed, many vital memcached statistics can be found at the statistics pages.These statistics are taken both from the PageSpeed perspective, aggregating all memcacheds as viewed from a single server...
...and also from the perspective of each memcached server, aggregating activity from each memcached client including all PageSpeed instances.
memcached server host1:6765 version 1.4.2 pid 1132 up 343306 seconds
memcached server host2:6765 version 1.4.2 pid 6568 up 343278 seconds
By default, PageSpeed uses a 50 millisecond (50,000 microsecond) timeout for cache operations. If the timeout is exceeded more than 4 times in a 30 second span, PageSpeed assumes that memcached is not healthy and will stop optimizing resources for 30 seconds before trying again.
This 50 millisecond timeout default is tuned for memcached running
on the same local network as PageSpeed. For wide area
networks, or for alternative implementations and proxies of
the memcache protocol such as couchbase or moxi-server, it
may be necessary to increase the I/O timeout. Please monitor
the statistic 'memcache_timeouts' to help tune the timeouts. Select
a new timeout with the
pagespeed MemcachedTimeoutUs timeout_in_microseconds
Flushing PageSpeed Server-Side Cache
Legacy Flushing Of Entire Cache
By default, the system is configured to support only whole-cache
flushes — we'll call this legacy mode. As
of IISpeed 2.0 / PageSpeed version 18.104.22.168, it can be configured to
also allow purging of individual URLs. The two modes operate
differently and you may choose between them for each virtual
host. Legacy mode is on by default, to provide compatibility
with existing scripts and other infrastructure that might have
been built around it. In a future release, individual URL
purging will become the default. Subsequent to that, the legacy
whole-cache flushing mode will be eliminated.
You can choose between the two modes with
EnableCachePurge argument. If set to "on" you
will get the new behavior, with individual URL purging, while
"off" will give you the legacy behavior. The default is "off",
as if your configuration read:
pagespeed EnableCachePurge off
In this mode, simply touch the file "cache.flush" in the directory
sudo touch /cache/pagespeed/cache.flushWindows
echo "" > c:\pagespeedcache\cache.flush
The system may take up to 5 seconds to take effect (changeable via option
Purging individual cache entries or entire cache
Note: New feature as of 22.214.171.124 / IISpeed 2.0
In this mode, the cache may be purged by sending HTTP requests to the server, using a configurable path. The cache can be purged via an HTTP GET, PURGE, or DELETE, once a handler has been configured. The admin site makes this easier by providing a graphical interface to initiate purge requests and see what entries have been purged.
To enable individual URL cache purging, you must configure the admin site, specify a PurgeMethod, or both. Plus you must turn on EnableCachePurge:
pagespeed EnableCachePurge on pagespeed PurgeMethod PURGE (optional)
This establishes three methods to purge the cache of a URL, or to purge the entire cache, assuming PageSpeed is running on example.com:
|Method||Purge single URL||Purge entire cache|
|Admin GUI||Admin interface|
The Purge requests for individual URLs made via the GUI are
executed via an HTTP GET. In either case, the URL being purged
is specified relative to the ORIGIN of the admin site. In these
path/file.ext is combined
example.com to purge the path
http://example.com/path/file.ext from the cache.
When the new mode of cache purging is enabled, the purges take place immediately, there is no five second delay. Note that it is possible to purge the entire cache, or to purge one URL at a time. It is not possible to purge by regular expression or wildcard. The URL purging system works by remembering which URLs are purged and validating each URL coming out of cache against them. There is a limitation to the number of distinct URLs that can be purged. When that limit is exceeded, everything in the cache older than the oldest remaining purge request will be dropped. The limitation is high enough that it's not expected to be exceeded often, but is not currently changeable.
The following limitations apply to both method the legacy and new methods of cache purging.
Caution: In a multi-server system, you must run these commands on every server. All the cache data from VirtualHosts using that cache directory will be flushed. This is true even when using memcached: the cache flush information is kept locally on each machine running PageSpeed, not in the cache itself. This is because of the L1 caches that run locally on each machine, and because memcached does not guarantee persistance.
Caution: Flushing or purging the cache does not delete the old files from the directory, the memcached server, or PageSpeed's in-memory cache, but it tells PageSpeed to ignore those files.
Note: After flushing or purging the cache, the
stale files will eventually be replaced with fresh copies or
removed by the normal file cache cleaning process
You can change the polling interval and name of the cache-flush
file in the configuration file. If you set the polling interval to 0, the
cache flushing feature will be disabled. If you specify the cache flush
filename as a relative path, PageSpeed will look for that file in
FileCachePath directory. If you use an absolute path,
then the caches associated with multiple virtual hosts can be flushed all
Cache Flushing and Purging Options
pagespeed CacheFlushFilename alternate_filename pagespeed CacheFlushPollIntervalSec number_of_seconds
Configuring a Cache Fragment
Note: New feature as of IISpeed 2.0 / PageSpeed 126.96.36.199
By default every site has its own cache. If you have multiple sites served from the same machine that reference common resources you can improve your cache performance and reduce CPU consumption by setting a shared cache fragment:
pagespeed CacheFragment some_token
You must set this to the same value on every site that
shares a cache. The fragment may consist of letters, numbers, underscores,
and hyphens only. The physical caching layer also has to be the same for
the two sites: either you need a shared
FileCachePath or you
need to be using the same
Note: you don't have to do this for simple cases
The default cache fragment is the minimal private suffix, in this
example.com, and is determined from
the public suffix list. If you
however, then they will not share a common minimal private suffix and you
should set the
CacheFragment for better performance.
PageSpeed HTTP fetching timeout
When PageSpeed attempts to rewrite a resource for the first time, it must fetch it via HTTP. The default timeout for fetches is 5 seconds. A directive can be applied to change the timeout
pagespeed FetcherTimeoutMs timeout_value_in_milliseconds
When PageSpeed attempts to rewrite an uncached (or expired) resource, by default it will wait for up to 10ms per flush window to finish and return the optimized resource if it's available. If optimization has not completed within that time the original (unoptimized) resource is returned and the optimizer is moved to the background for future requests. The following directive can be applied to change the deadline. Increasing this value will increase page latency, but might reduce load time (for instance on a bandwidth-constrained link where it's worth waiting for image compression to complete). Note that a value less than zero will cause PageSpeed to wait indefinitely.
pagespeed RewriteDeadlinePerFlushMs deadline_value_in_milliseconds
Implicit cache-lifetime for resources
When PageSpeed fetches a resource via HTTP or HTTPS, it examines the Expires and Cache-Control headers to determine how frequently it should update its cache. When these headers don't specify a timeout, a default timeout of 5 minutes is used. To override this, specify:
pagespeed ImplicitCacheTtlMs implicit_cache_ttl_in_milliseconds
Fetching Resources using Gzip
This option causes PageSpeed to add
requests for resources.
By default, PageSpeed attempts to fetch resources without specifying
Accept-Encoding header. This means the resources will be
sent uncompressed. These requests are often within the LAN, so network
bandwidth to transfer the resources may not be a consideration.
If network bandwidth is a consideration, then PageSpeed can be configured to fetch resources using gzip. This will lower the network transfer bandwidth considerably, but may increase the CPU usage depending on server configuration. The primary concern is the time spent by the origin server compressing the resource, rather than the time spent by PageSpeed inflating it.
pagespeed FetchWithGzip on
Another option to minimize network bandwidth is to use LoadFromFile.
These directives can only be applied at the root or server level, and not in location specific configuration.
PageSpeed uses threads so that resource optimization work does not delay
request serving. There are two kinds of threads in use: rewrite
threads deal with very short-lived bookkeeping tasks that are generally
latency-sensitive, while expensive rewrite threads deal with more
computationally expensive tasks that are not in latency-sensitive
You can tune the thread count per process if necessary via the
Note: IISpeed will autotune the threading settings, for ats_pagespeed manual tuning is required.
Limiting the number of concurrent image optimizations
When optimizing images, PageSpeed can use significant CPU resources. As
the results of the image optimization are cached, this is not ordinarily a
concern once the cache is warm. But when PageSpeed is first installed, or
when a corpus of new images is added to the server, PageSpeed needs to
avoid having each process consume maximum CPU. To accomplish this,
PageSpeed keeps a server-wide counter of active image optimizations. It
avoids running more than
optimizations in parallel across all processes. The default value is 8.
Override this in the configuration file to change this maximum.
Note: Limiting image optimization concurrency only applies to root and server level configuration.
Limiting the size of HTML parsed
When parsing and rewriting large HTML pages, PageSpeed can use
significant memory. This option limits the size of an HTML page that is
parsed. Once the size of the HTML exceeds
further parsing is disabled, and a script is inserted that redirects the
user to the
?PageSpeed=off version of the page. The
default value is 0, indicating that there is no limit. Override this in
the configuration file to change this maximum.
In-Place Resource Optimization
Note: Enabled by default as of IISpeed 2.0 / PageSpeed 188.8.131.52
pagespeed InPlaceResourceOptimization on
Doing browser-specific in-place optimization
PageSpeed has a number of optimizations that are browser-specific. For example, WebP conversion is performed only for browsers that can display WebP images. Ordinarily PageSpeed accomplishes this by serving different rewritten URLs to different browsers depending upon their capabilities. For resources that are rewritten in place, this isn't possible; instead, appropriate headers (such as Vary: Accept and Vary: User-Agent) are added to rewritten resources as necessary. Enable these browser-specific optimizations as follows:
pagespeed EnableFilters in_place_optimize_for_browser
By default, when
in_place_optimize_for_browser is enabled,
Vary: headers are added to resources that are
subject to browser-specific optimization. CSS files are served with
Vary: User-Agent header. Photographic images that are
candidates for WebP conversion are served with
Vary: Accept to
browsers that include
Accept: image/webp in their headers.
Note that while the most recent versions of browsers that display WebP
images include this header in image requests, older WebP-capable browsers do
not — as a result, a slightly smaller subset of browsers will receive
WebP images than would be the case if the URLs were rewritten.
Internet Explorer has difficulty caching resources
Vary: headers (they are either not cached or require
revalidation on every resource access). As a result, browser-specific
in-place resources are instead marked
when served to all versions of Internet Explorer.
Setting the inplace resource rewrite deadline
When InPlaceResourceOptimization is enabled, PageSpeed uses a default deadline of 10ms the when optimizing the resource. If the optimization cannot be completed in 10ms, then the original resource is served to the client, while the optimization continues in the background. Once cached, the optimized resource will be served for further requests. Note that a value less than zero will cause PageSpeed to wait indefinitely.
Also note that in-place-optimized resources generally take at least two refreshes to optimize regardless of the deadline, due to the current architecture of the server modules.
pagespeed InPlaceRewriteDeadlineMs deadline_value_in_milliseconds
This directive can be used at all configuration levels.
Resources optimized by in-place resource optimization are optimized
differently from resources found in HTML, JS, and CSS. First, rewritten
pagespeed URLs contain a content hash that enables the optimized data to be
cached for a year by browser and proxy caches; in-place resources are not
cache extended. Second, in-place resources can't be optimized specially for
the context in which they occur on the page: images can't be resized to the
size they appear on the page, and multiple resources on a page can’t be
combined together. Finally, in-place resources that are eligible for
browser-specific optimizations (such as conversion to the WebP image format)
will be served with the
Vary: User-Agent caching header,
reducing caching at intermediate proxies.
In-place resource optimization will add a new cache entry for every unique URL requested. It will also copy each request into memory once. If you have a large site with many uncacheable resources, this could quickly fill up your cache or cause a lot of expensive string copies.
In-place optimization will also add a small delay to every server response, this should not be large and we have not been able to measure any noticeable slow-down, but if most of your resources are uncacheable, you may wish to avoid this cost.
Rate Limit Background Fetches
Note: Enabled by default, depends on statistics being enabled
To avoid overloading the origin server, PageSpeed will limit the number
of background fetches it makes on a per-domain basis. As PageSpeed makes
fetches it keeps a count of how many ongoing fetches there are for each
domain, and if there are too many then additional fetches will only be
allowed through if they're for user-facing requests. Other fetches
(background fetches) will be queued up for completion later. If the
queue gets too large, PageSpeed will give up on those background
optimizations, leaving them to be reinitiated in response to a later
request. This feature can be disabled by
pagespeed RateLimitBackgroundFetches off