nginx + apache = happy & fast cPanel server

Nginx – the small, lightning fast and very efficient web server is usually used to serve static content or as a reverse proxy/load balancer for Apache.  Till now – the issue has been that many folks have not figured out how to use nginx due to their control panels lack of support.

Case in point - cPanel.

cPanel is an awesome control panel – and while not the least expensive – the support of Dan Muey and his team is second to none !

the cPanel forums like many – are host to flamings – but what is odd – unlike all the others – even the flames come with solutions – the community lacks nothing – except … nginx and openldap (room for another posting later)…

In order to get the cPanel server ready for nginx – you must first install an apache module called mod_rpaf  (written by Thomas Eibner – and he deserves some serious kudos for his excellent work.)

Mod_rpaf will in short – allow for apache to see the visitors ip address rather than the ip address of the nginx front end running on your server. – In short – failure to include this simple module will break the ability for your users to have meaningful logs.

Download (from here: http://stderr.net/apache/rpaf/ ) untar, cd to the newly created directory and run this command as root:

/usr/local/apache/bin/apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

Doing so will install the module into the Apache module directory.

Then in your  Web Host Manager (WHM) follow the tree here:  Main >> Service Configuration >> Apache Configuration > Include Editor > Pre Main Include and add this section there, replacing (place your ips here w/o the brakets) with the list of IP addresses on this Cpanel server:

LoadModule rpaf_module modules/mod_rpaf-2.0.so 

RPAFenable On
# Enable reverse proxy add forward

RPAFproxy_ips 127.0.0.1  (place your ips here w/o the brakets) 

RPAFsethostname On
# let rpaf update vhost settings allowing to have
# the same hostnames as in the "actual" configuration for the
# forwarding apache installation

RPAFheader X-Real-IP
# Allows you to change which header we have mod_rpaf looking for
# when trying to find the ip the that is forwarding our requests

Once this is completed – we are ready to move Apache to another port, let’s take 81 for example.  This – thank goodness to the excellent work of the cPanel team in coding an excellent product is quite simple.  In WHM navigate to the “Tweak Settings” page and replace

0.0.0.0:80 with 0.0.0.0:81

While many like doing it the gui method – others might like using the command line interface (cli)  -so instructions are posted here for those wishing to complete the task that way instead.

vi /var/cpanel/cpanel.config and change port 80 in apache_port assignment to 81:  apache_port=0.0.0.0:81

Next – you need to Run:

/usr/local/cpanel/whostmgr/bin/whostmgr2 –updatetweaksettings

Next – check /usr/local/apache/conf/httpd.conf for any occurrences of port 80, and run /scripts/rebuildhttpdconf to make sure your httpd.conf file is up to date.

Since Nginx is going to be in the front feeding the data requested from browsers to the sites you host – you can now reduce the number of Apache children.   This is done by editing  /usr/local/apache/conf/httpd.conf and replacing the prefork.c section with settings similar to those below…  It is important to note these values may need to be tweaked to fit your needs:

<IfModule prefork.c>
    StartServers 8
    MinSpareServers 2
    MaxSpareServers 5
    MaxClients 80
    MaxRequestsPerChild 0
</IfModule>

Make sure to run /usr/local/cpanel/bin/apache_conf_distiller –update –main so that the changes are picked up and then run  /scripts/rebuildhttpdconf

I promise – were almost there next – we are going to move onto the NGINX settings:

In this final step we are going to build the nginx configuration files based on the domains hosted on your cpanel server.  There are two files: /usr/local/nginx/conf/nginx.conf which is the main configuration file – and the include file with all virtual hosts: /usr/local/nginx/conf/vhost.conf

#!/bin/sh

cat > "/usr/local/nginx/conf/nginx.conf" <<EOF
user  nobody;
# no need for more workers in the proxy mode
worker_processes  1;

error_log  logs/error.log info;

worker_rlimit_nofile  8192;

events {
 worker_connections  512; # you might need to increase this setting for busy servers
 use rtsig; #  Linux kernels 2.6.x change to epoll
}

http {
 server_names_hash_max_size 2048;

 include    mime.types;
 default_type  application/octet-stream;

 sendfile on;
 tcp_nopush on;
 tcp_nodelay on;

 keepalive_timeout  10;

 gzip on;
 gzip_min_length  1100;
 gzip_buffers  4 32k;
 gzip_types    text/plain text/html application/x-javascript text/xml text/css;
 ignore_invalid_headers on;

 client_header_timeout  3m;
 client_body_timeout 3m;
 send_timeout     3m;
 connection_pool_size  256;
 client_header_buffer_size 4k;
 large_client_header_buffers 4 32k;
 request_pool_size  4k;
 output_buffers   4 32k;
 postpone_output  1460;

 include "/usr/local/nginx/conf/vhost.conf";
}

EOF

/bin/cp /dev/null /usr/local/nginx/conf/vhost.conf

cd /var/cpanel/users
for USER in *; do
 for DOMAIN in `cat $USER | grep ^DNS | cut -d= -f2`; do
  IP=`cat $USER|grep ^IP|cut -d= -f2`;
  ROOT=`grep ^$USER: /etc/passwd|cut -d: -f6`;
  echo "Converting $DOMAIN for $USER";

  cat >> "/usr/local/nginx/conf/vhost.conf" <<EOF
   server {
  access_log off;

  error_log  logs/vhost-error_log warn;
  listen    80;
  server_name  $DOMAIN www.$DOMAIN;

  # uncomment location below to make nginx serve static files instead of Apache
  # NOTE this will cause issues with bandwidth accounting as files wont be logged
  #location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css)$ {
  # root   $ROOT/public_html;
  #}

  location / {
   client_max_body_size    10m;
   client_body_buffer_size 128k;

   proxy_send_timeout   90;
   proxy_read_timeout   90;

   proxy_buffer_size    4k;
   # you can increase proxy_buffers here to suppress "an upstream response
   #  is buffered to a temporary file" warning
   proxy_buffers     16 32k;
   proxy_busy_buffers_size 64k;
   proxy_temp_file_write_size 64k;

   proxy_connect_timeout 30s;

   proxy_redirect  http://www.$DOMAIN:81   http://www.$DOMAIN;
   proxy_redirect  http://$DOMAIN:81   http://$DOMAIN;

   proxy_pass   http://$IP:81/;

   proxy_set_header   Host   \$host;
   proxy_set_header   X-Real-IP  \$remote_addr;
   proxy_set_header   X-Forwarded-For \$proxy_add_x_forwarded_for;
  }
 }
EOF
 done
done

Now – Run /usr/local/nginx/sbin/nginx -t to check the configuration, and then /usr/local/nginx/sbin/nginx to start nginx.

We have setup a hook to the cPanel scripts to rerun this script anytime a new account is created as well as anytime a domain is parked however we will leave that for another post.