HTTP compression

HTTP compression is a capability that can be built into web servers and web clients to improve transfer
speed and bandwidth utilization.[1]

HTTP data is compressed before it is sent from the server: compliant browsers will announce what methods
are supported to the server before downloading the correct format; browsers that do not support compliant
compression method will download uncompressed data. The most common compression schemes include
gzip and Brotli; a full list of available schemes is maintained by the IANA.[2]

There are two different ways compression can be done in HTTP. At a lower level, a Transfer-Encoding
header field may indicate the payload of an HTTP message is compressed. At a higher level, a Content-
Encoding header field may indicate that a resource being transferred, cached, or otherwise referenced is
compressed. Compression using Content-Encoding is more widely supported than Transfer-Encoding, and
some browsers do not advertise support for Transfer-Encoding compression to avoid triggering bugs in
servers.[3]

Compression scheme negotiation

The negotiation is done in two steps, described in RFC 2616 and RFC 9110:

1. The web client advertises which compression schemes it supports by including a list of tokens in the
HTTP request. For Content-Encoding, the list is in a field called Accept-Encoding; for Transfer-Encoding,
the field is called TE.

GET /encrypted-area HTTP/1.1

Host: [Link]
Accept-Encoding: gzip, deflate

2. If the server supports one or more compression schemes, the outgoing data may be compressed by one or
more methods supported by both parties. If this is the case, the server will add a Content-Encoding or
Transfer-Encoding field in the HTTP response with the used schemes, separated by commas.

HTTP/1.1 200 OK
Date: mon, 26 June 2016 22:38:34 GMT
Server: Apache/[Link] (Unix) (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip

The web server is by no means obligated to use any compression method  – this depends on the internal
settings of the web server and also may depend on the internal architecture of the website in question.

Content-Encoding tokens
The official list of tokens available to servers and client is maintained by IANA,[4] and it includes:
 br – Brotli, a compression algorithm specifically designed for HTTP content encoding,
defined in RFC 7932 ([Link] and implemented in all
modern major browsers.
compress – UNIX "compress" program method (historic; deprecated in most applications
and replaced by gzip or deflate)
deflate – compression based on the deflate algorithm (described in RFC 1951 ([Link]
[Link]/doc/html/rfc1951)), a combination of the LZ77 algorithm and Huffman coding,
wrapped inside the zlib data format (RFC 1950 ([Link]
exi – W3C Efficient XML Interchange
gzip – GNU zip format (described in RFC 1952 ([Link]
Uses the deflate algorithm for compression, but the data format and the checksum algorithm
differ from the "deflate" content-encoding. This method is the most broadly supported as of
March 2011.[5]
identity – No transformation is used. This is the default value for content coding.
pack200-gzip – Network Transfer Format for Java Archives[6]
zstd – Zstandard compression, defined in RFC 8478 ([Link]
478)

In addition to these, a number of unofficial or non-standardized tokens are used in the wild by either servers
or clients:

bzip2 – compression based on the free bzip2 format, supported by lighttpd[7]

lzma – compression based on (raw) LZMA is available in Opera 20, and in elinks via a
compile-time option[8]
peerdist[9] – Microsoft Peer Content Caching and Retrieval
rsync[10] - delta encoding in HTTP, implemented by a pair of rproxy proxies.
xpress - Microsoft compression protocol used by Windows 8 and later for Windows Store
application updates. LZ77-based compression optionally using a Huffman encoding.[11]
xz - LZMA2-based content compression, supported by a non-official Firefox patch;[12] and
fully implemented in mget since 2013-12-31.[13]

Servers that support HTTP compression

SAP NetWeaver
Microsoft IIS: built-in or using third-party module
Apache HTTP Server, via mod_deflate ([Link]
_deflate.html) (despite its name, only supporting gzip[14]), and mod_brotli ([Link]
[Link]/docs/current/mod/mod_brotli.html)
Hiawatha HTTP server: serves pre-compressed files[15]
Cherokee HTTP server, On the fly gzip and deflate compressions
Oracle iPlanet Web Server
Zeus Web Server
lighttpd
nginx – built-in
Applications based on Tornado, if "compress_response" is set to True in the application
settings (for versions prior to 4.0, set "gzip" to True)
Jetty Server – built-into default static content serving and available via servlet filter
configurations
 GeoServer
Apache Tomcat
IBM Websphere
AOLserver
Ruby Rack, via the Rack::Deflater middleware
HAProxy
Varnish – built-in. Works also with ESI
Armeria ([Link] – Serving pre-compressed files[16]
NaviServer - built-in, dynamic and static compression

Many content delivery networks also implement HTTP compression to improve speedy delivery of
resources to end users.

The compression in HTTP can also be achieved by using the functionality of server-side scripting
languages like PHP, or programming languages like Java.

Various online tools exist to verify a working implementation of HTTP compression. These online tools
usually request multiple variants of a URL, each with different request headers (with varying Accept-
Encoding content). HTTP compression is considered to be implemented correctly when the server returns a
document in a compressed format.[17] By comparing the sizes of the returned documents, the effective
compression ratio can be calculated (even between different compression algorithms).

Problems preventing the use of HTTP compression

A 2009 article by Google engineers Arvind Jain and Jason Glasgow states that more than 99 person-years
are wasted[18] daily due to increase in page load time when users do not receive compressed content. This
occurs when anti-virus software interferes with connections to force them to be uncompressed, where
proxies are used (with overcautious web browsers), where servers are misconfigured, and where browser
bugs stop compression being used. Internet Explorer 6, which drops to HTTP 1.0 (without features like
compression or pipelining) when behind a proxy – a common configuration in corporate environments –
was the mainstream browser most prone to failing back to uncompressed HTTP.[18]

Another problem found while deploying HTTP compression on large scale is due to the deflate encoding
definition: while HTTP 1.1 defines the deflate encoding as data compressed with deflate (RFC 1951)
inside a zlib formatted stream (RFC 1950), Microsoft server and client products historically implemented it
as a "raw" deflated stream,[19] making its deployment unreliable.[20][21] For this reason, some software,
including the Apache HTTP Server, only implement gzip encoding.

Security implications
Compression allows a form of chosen plaintext attack to be performed: if an attacker can inject any chosen
content into the page, they can know whether the page contains their given content by observing the size
increase of the encrypted stream. If the increase is smaller than expected for random injections, it means that
the compressor has found a repeat in the text, i.e. the injected content overlaps the secret information. This
is the idea behind CRIME.

In 2012, a general attack against the use of data compression, called CRIME, was announced. While the
CRIME attack could work effectively against a large number of protocols, including but not limited to
TLS, and application-layer protocols such as SPDY or HTTP, only exploits against TLS and SPDY were
demonstrated and largely mitigated in browsers and servers. The CRIME exploit against HTTP
compression has not been mitigated at all, even though the authors of CRIME have warned that this
vulnerability might be even more widespread than SPDY and TLS compression combined.

In 2013, a new instance of the CRIME attack against HTTP compression, dubbed BREACH, was
published. A BREACH attack can extract login tokens, email addresses or other sensitive information from
TLS encrypted web traffic in as little as 30 seconds (depending on the number of bytes to be extracted),
provided the attacker tricks the victim into visiting a malicious web link.[22] All versions of TLS and SSL
are at risk from BREACH regardless of the encryption algorithm or cipher used.[23] Unlike previous
instances of CRIME, which can be successfully defended against by turning off TLS compression or
SPDY header compression, BREACH exploits HTTP compression which cannot realistically be turned
off, as virtually all web servers rely upon it to improve data transmission speeds for users.[22]

As of 2016, the TIME attack and the HEIST attack are now public knowledge.[24][25][26][27]

References
1. "Using HTTP Compression (IIS 6.0)" ([Link]
sServer2003/Library/IIS/[Link]?mfr=true). Microsoft
Corporation. Retrieved 9 February 2010.
2. RFC 2616, Section 3.5: "The Internet Assigned Numbers Authority (IANA) acts as a registry
for content-coding value tokens."
3. 'RFC2616 "Transfer-Encoding: gzip, chunked" not handled properly' ([Link]
m/p/chromium/issues/detail?id=94730), Chromium Issue 94730
4. "Hypertext Transfer Protocol Parameters - HTTP Content Coding Registry" ([Link]
[Link]/assignments/http-parameters/[Link]#content-coding). IANA. Retrieved
18 April 2014.
5. "Compression Tests: Results" ([Link]
[Link]/projects/compression-tests/results). Verve Studios, Co. Archived from the
original ([Link] on 21 March 2012.
Retrieved 19 July 2012.
6. "JSR 200: Network Transfer Format for Java Archives" ([Link]
The Java Community Process Program.
7. "ModCompress - Lighttpd" ([Link]
lighty labs. Retrieved 18 April 2014.
8. elinks LZMA decompression ([Link]
[Link]#CONFIG-LZMA)
9. "[MS-PCCRTP]: Peer Content Caching and Retrieval: Hypertext Transfer Protocol (HTTP)
Extensions" ([Link]
Microsoft. Retrieved 19 April 2014.
10. "rproxy: Protocol Definition for HTTP rsync Encoding" ([Link]
[Link]). [Link].
11. "[MS-XCA]: Xpress Compression Algorithm" ([Link]
[Link]). Retrieved 29 August 2015.
12. "LZMA2 Compression - MozillaWiki" ([Link]
Retrieved 18 April 2014.
13. "mget GitHub project page" ([Link] GitHub. Retrieved
6 January 2017.
14. "mod_deflate - Apache HTTP Server Version 2.4 - Supported Encodings" ([Link]
[Link]/docs/2.4/mod/mod_deflate.html#supportedencodings).
15. "Extra part of Hiawatha webserver's manual" ([Link]
s).
16. "Serving static files part of Armeria's documentation" ([Link]
-[Link]#serving-pre-compressed-files).
17. "How does the gzip compression check work?" ([Link]
[Link], retrieved 10 April 2022.
18. "Use compression to make the web faster" ([Link]
e-compression). Google Inc. Retrieved 22 May 2013.
19. "deflate - Why are major web sites using gzip?" ([Link]
8/why-are-major-web-sites-using-gzip/9186091#9186091). Stack Overflow. Retrieved
18 April 2014.
20. "Compression Tests: About" ([Link]
[Link]/projects/compression-tests/). Verve Studios. Archived from the original ([Link]
[Link]/projects/compression-tests/) on 2 January 2015. Retrieved 18 April 2014.
21. "Lose the wait: HTTP Compression" ([Link]
pression). Zoompf Web Performance. Retrieved 18 April 2014.
22. Goodin, Dan (1 August 2013). "Gone in 30 seconds: New attack plucks secrets from
HTTPS-protected pages" ([Link]
w-attack-plucks-secrets-from-https-protected-pages/). Ars Technica. Condé Nast. Retrieved
2 August 2013.
23. Leyden, John (2 August 2013). "Step into the BREACH: New attack developed to read
encrypted web data" ([Link] The
Register. Retrieved 2 August 2013.
24. Sullivan, Nick (11 August 2016). "CRIME, TIME, BREACH and HEIST: A brief history of
compression oracle attacks on HTTPS" ([Link]
ession-oracle-attacks-https/). Retrieved 16 August 2016.
25. Goodin, Dan (3 August 2016). "HEIST exploit — New attack steals SSNs, e-mail addresses,
and more from HTTPS pages" ([Link]
sns-e-mail-addresses-and-more-from-https-pages/). Retrieved 16 August 2016.
26. Be'ery, Tal. "A Perfect Crime? TIME will tell" ([Link]
CRIME_TIME_Will_Tell_-_Tal_Beery.pdf) (PDF).
27. Vanhoef, Mathy. "HEIST: HTTP Encrypted Information can be Stolen through TCP-windows"
([Link]
[Link]) (PDF).

External links
RFC 2616 ([Link] Hypertext Transfer Protocol –
HTTP/1.1
RFC 9110 ([Link] HTTP Semantics
HTTP Content-Coding Values ([Link] by
Internet Assigned Numbers Authority
Compression with lighttpd ([Link]
ss)
Coding Horror: HTTP Compression on IIS 6.0 ([Link]
[Link]) Archived ([Link]
p://[Link]/blog/2004/08/[Link]) 2014-02-06 at
the Wayback Machine
 15 Seconds: Web Site Compression ([Link]
[Link]/Issue/[Link]) at the Wayback Machine (archived July 16, 2011)
Using HTTP Compression ([Link]
Archived ([Link]
s/[Link]/3514866) 2016-03-14 at the Wayback Machine by Martin Brown of Server
Watch
Using HTTP Compression in PHP ([Link]
[Link]/c/a/PHP/Using-HTTP-Compression-in-PHP-Make-Your-Web-Pages-Load-F
aster/)
Dynamic and static HTTP compression with Apache httpd ([Link]
0430023716/[Link]
pd/)

Retrieved from "[Link]