Routing/Static content delivery performances

We will now put the middle-end in place.
Nginx will have two tasks :

• Delivering the static content at very fast speed
• Routing every non-static or unavailable ressource to Apache backend

Nginx will act as a Reverse Proxy. Remember it should not destroy Cache-control headers in order to let the future frontend to properly cache the static contents.

apt-get install nginx

Debian comes with an exploded configuration of nginx that we will not take in account.
Here is the whole configuration file we’ll which I’ll explain later :

user  www-data www-data;
worker_processes  2;

error_log  logs/error.log;
error_log  logs/error.log  notice;
error_log  logs/error.log  info;

pid        /var/run/nginx.pid;

events {
    worker_connections  128;
    use epoll;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    tcp_nopush     on;
    tcp_nodelay	   on;

    keepalive_timeout  10;
    keepalive_requests 10;

    gzip  on;

	proxy_cache_path          /usr/local/nginx/proxy_temp/ levels=1:2 keys_zone=cache:10m inactive=10m max_size=1000M;

server {

	listen 8081;

	server_name blog.brigato.fr;

	charset utf-8;

        #Pass dynamic content through Apache directly
	location / {

		proxy_buffering           on;
		proxy_cache_min_uses       3;
		proxy_cache_valid         any 10m;
		proxy_ignore_client_abort off;
		proxy_intercept_errors    on;
		proxy_next_upstream       error timeout invalid_header;
		proxy_redirect            off;
		proxy_set_header          X-Forwarded-For $remote_addr;
		proxy_connect_timeout     60;
		proxy_send_timeout        60;
		proxy_read_timeout        60;

	        proxy_pass  http://127.0.0.1:8080;
		client_max_body_size	10m;
		client_body_buffer_size	128k;
		proxy_buffers		32 4k;
		proxy_set_header        Host            $host;
	}

        # Pass static content directly through nginx and set expire for Varnish Cache
        location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|htm|html)$ {
                root  /var/www/blog.brigato.fr/;
                expires           10m;
        }

    }

}

Here we go for explanations :

user  www-data www-data;

The user directive should be set to the same user you’re using Apache. Under Debian it is www-data if there where no modification in Apache configuration.

worker_processes  2;
events {
    worker_connections  128;
    use epoll;
}

Unlike Apache HTTPd, which is Process/Threads based, Nginx is an asynchronous event-based server.
Here, we start 2 process (Number of Vcores + 1) which will each handle 128 concurrent connections at a time, for a maximum of 256 concurrent connection total. Any more connections will be queued of refused by the epoll method, which is the best polling method under linux (won’t be true for a BSD System, for example, wich should use kqueue method). More information here (French).

keepalive_timeout  10;
keepalive_requests 10;

Theses two are really importants, as the Caching Frontend will use a lot of Connection Reuse to avoid the overhead of starting a new TCP connection for every request. A TCP connection is fast to make in itself, but the overhead is very high when making thousands and more TCP connection every seconds.

listen 8081;

Quite explicit. We’ll listen on port 8081 and keep the port 80 for the caching frontend.

location / {

		proxy_buffering           on;
		proxy_cache_min_uses       3;
		proxy_cache_valid         any 10m;
		proxy_ignore_client_abort off;
		proxy_intercept_errors    on;
		proxy_next_upstream       error timeout invalid_header;
		proxy_redirect            off;
		proxy_set_header          X-Forwarded-For $remote_addr;
		proxy_connect_timeout     60;
		proxy_send_timeout        60;
		proxy_read_timeout        60;

	        proxy_pass  http://127.0.0.1:8080;
		client_max_body_size	10m;
		client_body_buffer_size	128k;
		proxy_buffers		32 4k;
		proxy_set_header        Host            $host;
	}

Here we’ll proxy HTTP traffic to Apache (proxy_pass http://127.0.0.1:8080;) and be sure to pass also the Host header.
We’ll add the X-Forwarded-For header to Apache containing the $remote_addr of the client, as nginx will appear in default Apache log with the 127.0.0.1 IP. It will belong to you to edit your Apache logs and Web application settings to take care of this header.
Also, we’ll cache for 10 Minutes every static content which is first passed by apache (proxy_cache_valid any 10m;), which means that for every 1 request passed to apache requesting a static file, Nginx will directly deliver static content for the 10 next minutes. This is the important behavior we were looking for.

Now you can restart Nginx :

/etc/init.d/nginx restart

At this time, Nginx should make his first request directly to Apache, which deliver to Nginx a static cached Page.
Nginx will keep this page as Static Content for the next 10 Minutes, and deliver the page blazing-fast for subsequent request.

A new benchmark should provide performance upgrade by a factor 30, which resulted for me in an overall 521 requests/s.

We are now done with middle-end.