Rapid Application Development Made Easy

Our current setup we have multiple application and static content servers serving our ecommerce shopping software Veracart. Instead of spending thousands of dollars on an expensive load balancer appliance we decide to go the open source route. After looking at many different solutions what we finally ended up going with for our load balancer machine is stunnel for the ssl decryption and haproxy for the actual load balancing. stunnel is need because haproxy doesn’t support ssl in http mode.

Load balancer hardware: 1U, dual Intel Xeon 2.80GHz cpu’s, 4gigs of ram, sata drives.
Software: freebsd (or linux), stunnel 4.32 with xforwarded-for patch, haproxy 1.4.8

Stunnel Installation

First off if you want your web servers in your clusters to be able to know the connecting clients real ip address instead of the load balancers then you need to patch stunnel with xforwarded-for patch.

wget http://www.stunnel.org/download/stunnel/src/stunnel-4.32.tar.gz
wget http://haproxy.1wt.eu/download/patches/stunnel-4.32-xforwarded-for.diff
tar -zxvf stunnel-4.32.tar.gz
cd stunnel-4.32
patch -p1 < ../stunnel-4.32-xforwarded-for.diff
./configure
make && make install

Haproxy Installation

download: http://haproxy.1wt.eu/#down
or
cd /usr/ports/net/haproxy (from freebsd ports)
make && make install

Stunnel Configuration

Stunnel will receive all the https connections on port 443 and forward as http requests to haproxy on port 81 (or any port you want). From there haproxy will send the http request to the webserver cluster. Below the ip 1.2.3.4 represents your public ip that will goto your cluster. You can have as many [secure.domain.com] sections as you want or have domains with ssl certs for.

stunnel.conf

sslVersion = all
options = NO_SSLv2
setuid = root
setgid = stunnel
pid = /var/run/stunnel.pid
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
output = /var/log/stunnel.log

[secure.domain.com]
cert = /usr/local/openssl/certs/secure.domain.com.crt
key = /usr/local/openssl/private/secure.domain.com.key
accept  = 1.2.3.4:443
connect = 1.2.3.4:81
xforwardedfor = yes
TIMEOUTclose = 0

Haproxy Configuration

Haproxy will receive the decrypted https request, now just an http request, from stunnel on port 81. It will then load balance those requests across however many servers you have listed. I forward the request to port 81 on our apache webservers.

haproxy.conf

global
	daemon
	maxconn 10000
	nbproc 2
	log 127.0.0.1 syslog

defaults
	log global
	clitimeout 60000
	srvtimeout 30000
	contimeout 4000
	retries 3
	option redispatch
	option httpclose
	option abortonclose

listen domain_cluster_http 1.2.3.4:80
	mode http
	balance roundrobin
	cookie SERVERID insert nocache
	option forwardfor except 1.2.3.4
	reqadd X-Forwarded-Proto:\ http
	server server1 10.1.1.2:80 cookie s1 weight 1 maxconn 5000 check
	server server2 10.1.1.3:80 cookie s2 weight 1 maxconn 5000 check

listen domain_cluster_https 1.2.3.4:81
	mode http
	balance roundrobin
	cookie SERVERID insert nocache
	option forwardfor except 1.2.3.4
	reqadd X-Forwarded-Proto:\ https
	server server1 10.1.1.2:81 cookie s1 weight 1 maxconn 5000 check
	server server2 10.1.1.3:81 cookie s2 weight 1 maxconn 5000 check	

listen  lb1_stats [load balancer's public ip]:80
	mode http
	stats uri /
	stats auth username:password
	#stats refresh 5s

Apache Configuration

Lastly there is some things you will want to do to apache so it will know the real clients ip address. What I use is mod_rpaf, which changes the remote address of the client visible to other Apache modules from the X-Forwarded-For header. On freebsd there is a port for this.

cd /usr/ports/www/mod_rpaf2
make && make install

This will install the Apache2 version. After its installed open up your Apache’s http.conf file and add the following lines.

httpd.conf

# if DSO load module first:
LoadModule rpaf_module libexec/apache2/mod_rpaf-2.0.so
RPAFenable On
RPAFsethostname On
RPAFproxy_ips [load balancer's ip]
RPAFheader X-Forwarded-For

Next since we are forwarding https request to our webserver cluster nodes on port 81 we need to add the following line to http.conf

httpd.conf

Listen 81

Conclusion

We don’t have a huge amount of traffic about a steady 5Mbps, and the load balancer is always idling along with barely any load at all. This has worked out great for us. All the ssl decrypting is off loaded from the web servers and haproxy is super fast and has a great stats page to see what’s going on with your cluster’s.