How to host static content pre-compressed in apache?

Question

I host a JavaScript game which basically consists of an .html and a .data file. If I compress them with gzip, their size shrinks to 25%. So I want to do that.

I'm not 100% sure, but I think using mod_gzip or mod_deflate does the compression on-the-fly, wasting cpu time all the time because the Content doesn't Change.

So I'ld like to precompile the Content. Therefore, I put a .gz next to the uncompressed files and put rewrite rules in .htaccess:

RewriteEngine on 
# If client accepts compressed files 
RewriteCond %{HTTP:Accept-Encoding} gzip 
# and if compressed file exists 
RewriteCond %{REQUEST_FILENAME}.gz -f 
# send .html.gz instead of .html 
RewriteRule ^(.+)\.(html|css|js|data)$ $1.$2.gz [T=text/$2,E=GZIP:gzip,L] 
Header set Content-Encoding gzip env=GZIP 

The Redirect is working, I can request game.html and actually get deliviered game.html.gz. However, the browser doesn't just display it. Instead, it asks me where to save the file. How can I fix that? Or maybe there is another way to achieve my Goal?

Answer 1

This took me a while to figure out but the reason most people are having issues with rewriting requests to pre-compressed files is because they using REQUEST_FILENAME.

It seem REQUEST_FILENAME depending on the context and where you put you rewrite code, determines if you getting the full file-system path or a copy of REQUEST_URI.

REQUEST_FILENAME
The full local filesystem path to the file or script matching the request, if this has already been determined by the server at the time REQUEST_FILENAME is referenced. Otherwise, such as when used in virtual host context, the same value as REQUEST_URI. Depending on the value of AcceptPathInfo, the server may have only used some leading components of the REQUEST_URI to map the request to a file.

Ref: https://httpd.apache.org/docs/2.4/mod/mod_rewrite.html

To fix this I found there are 2 possible options.

One is to use the look-ahead modifier as per the docs.

If used in per-server context (i.e., before the request is mapped to the filesystem) SCRIPT_FILENAME and REQUEST_FILENAME cannot contain the full local filesystem path since the path is unknown at this stage of processing. Both variables will initially contain the value of REQUEST_URI in that case. In order to obtain the full local filesystem path of the request in per-server context, use an URL-based look-ahead %{LA-U:REQUEST_FILENAME} to determine the final value of REQUEST_FILENAME.

The other option that is not reliant on context or look-ahead I found to be using a DOCUMENT_ROOT with REQUEST_URI.

So putting it all together looks something like this.

# ==============================================================================
# Serving pre-compressed content
# ==============================================================================
# Include in a vhost to enable Serving Gzip Files
# ------------------------------------------------------------------------------
# Ref:
# - https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#precompressed
#

<IfModule mod_headers.c>
    <IfModule mod_rewrite.c>
        RewriteEngine On

        # Serve gzip compressed CSS and JS files if they exist
        # and the client accepts gzip.
        RewriteCond "%{HTTP:Accept-encoding}" "gzip"

        # REQUEST_FILENAME is only available as a full path once the requests
        # has been resolved so we need to use a look ahead or
        # DOCUMENT_ROOT+REQUEST_URI
        # RewriteCond "%{LA-U:REQUEST_FILENAME}\.gz"    "-s"
        RewriteCond "%{DOCUMENT_ROOT}%{REQUEST_URI}.gz" "-s"

        # Rewrite to gzip file
        RewriteRule "^(.*)\.(css|js)"                   "$1\.js\.gz" [QSA]

        # Serve correct content types, and prevent mod_deflate double gzip.
        RewriteRule "\.css\.gz$" "-" [T=text/css,E=no-gzip:1]
        RewriteRule "\.js\.gz$"  "-" [T=text/javascript,E=no-gzip:1]

        <FilesMatch "(\.js|\.css)$">
            Header append Pre-Compressed 0
        </FilesMatch>

        <FilesMatch "(\.js\.gz|\.css\.gz)$">
            # Serve correct encoding type.
            Header append Content-Encoding gzip

            # Force proxies to cache gzipped &
            # non-gzipped css/js files separately.
            Header append Vary Accept-Encoding
            Header append Pre-Compressed 1
        </FilesMatch>
    </IfModule>
</IfModule>

It would be nice if Apache's official docs actually point this out or used one of these 2 examples that work in all contexts as this has probably tripped up a lot of people over the years.

Ref: https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#precompressed

Answer 2

The accepted answer seems quite painful. Wolph's answer seems better, but still requires separate configuration for each file extension and lacks support for more advanced negotiation (q-values, status 406, TCN, etc.). Rather than implementing content negotiation yourself using mod_rewrite, you may want to consider using mod_negotiation as discussed in this question. Copying my answer from there:

Options +MultiViews
RemoveType .gz
AddEncoding gzip .gz
<FilesMatch ".+\.tar\.gz$">
    RemoveEncoding .gz
    # Note:  Can use application/x-gzip for backwards-compatibility
    AddType application/gzip .gz
</FilesMatch>

This has the added bonus of working for all .gz files rather than only explicitly-configured ones and being easily extended for brotli or other encodings.

It does have one major drawback, since only requests for files which do not exist are negotiated a file named foo.js would make requests for /foo.js (but not /foo) return the uncompressed version. This can be avoided using François Marier's solution of renaming uncompressed files with a double extension, so foo.js is deployed as foo.js.js.

Answer 3

This is how i fixed once the same problem.

Add new types in .htaccess:

AddEncoding gzip .jsgz .cssgz .htmlgz .datagz
AddType application/javascript .jsgz
AddType text/css .cssgz
AddType text/html .htmlgz       
AddType text/plain .datagz

This was done this way because AddType instruction didn't accept extensions in the form .html.gz.

Then modify your rewrite rule:

RewriteRule ^(.+)\.(html|css|js|data)$ $1.$2gz [L] 

And finally rename your files. Remove dots from .html.gz, .js.gz and so on.

The full .htaccess would look like this:

AddEncoding gzip .jsgz .cssgz .htmlgz .datagz
AddType application/x-javascript .jsgz
AddType text/css .cssgz
AddType text/html .htmlgz       
AddType text/plain .datagz

RewriteEngine on 
# If client accepts compressed files 
RewriteCond %{HTTP:Accept-Encoding} gzip 
# and if compressed file exists 
RewriteCond %{REQUEST_FILENAME}gz -f 
# send .html.gz instead of .html 
RewriteRule ^(.+)\.(html|css|js|data)$ $1.$2gz [L] 

Answer 4

What about the solution outlined here: http://feeding.cloud.geek.nz/posts/serving-pre-compressed-files-using/. Use Apache's built-in MultiViews...

Answer 5

The first question you should ask yourself is, is there any point in doing this? Do you notice too high cpu load and/or performance difference because of this? My guess would be that you are probably not running into this problem :)

Regardless though, there are multiple ways of fixing your problem.

1. Probably the best option for you, use a CDN. They are designed for fast delivery of static files and will make it fast for people in a different geographical area as well as people close to your server. Also, in my experience CDN's are usually much cheaper than your own bandwidth will be.

2. Use Nginx. For hosting static files much faster and has support for pre generating static content like you are doing right now. It will automatically detect if there's a .gz file and serve that instead when needed.

3. Use one of the Apache cache mechanisms like mod_mem_cache or mod_disk_cache to make sure that every regularly used file will be in cache. Tutorial: http://webdirect.no/linux/apache-caching-with-gzip-enabled/

4. Use a caching proxy like Varnish in front of it, these type of servers have a much smarter caching mechanism and will actually cache the files that matter most.

For your current version however, something like this (untested) should do the trick:

RewriteEngine On    
RewriteCond %{HTTP:Accept-encoding} gzip
RewriteCond %{REQUEST_FILENAME}\.gz -s
RewriteRule ^(.*)\.(html|css|js|data) $1\.$2\.gz [QSA]

# Prevent double gzip and give the correct mime-type
RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.html\.gz$ - [T=text/html,E=no-gzip:1,E=FORCE_GZIP]
RewriteRule \.data\.gz$ - [T=text/plain,E=no-gzip:1,E=FORCE_GZIP]

Header set Content-Encoding gzip env=FORCE_GZIP