Setting up HAProxy with Transparent Mode on Centos 6.

Transparent mode with HAProxy allows you to see the IP Address of the clients computer while
still having a high availability service using HAProxy.
This posting shows how to setup a blank virgin installation of Centos 6.3 64bit minimum
installation.
This guide works on the assumption that you have a public facing IP Address of [Link] (I
know thats not a real public address) and are using an internal network address space of
10.10.10.x/24 with our two web servers on [Link] and [Link]. So we will have two
network interfaces on our LoadBalancer eth0 will be set with our real world IP of [Link]
and eth1 will be set up with [Link].
After installing our basic Centos 6.3 64bit OS, it maybe worth running a ‘yum update‘ command
first to ensure that the system is fully updated.
As this is a minimum installation you will also need to install a few other packages. These can be
installed with the following command:
yum install make wget gcc pcre-static pcre-devel
I’m using the HAProxy 1.5 dev7 build for this example but at the time of writing dev12 is the latest
available build and I’ll assume that the following will also work with that Development Release.
However, to get all the features that we require we will need to build HAProxy from source and not
from the package repository. The following steps enable us to do just that:
wget [Link]
[Link]
tar -zxf [Link]
cd haproxy-1.5-dev7
make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
cp haproxy /usr/bin/haproxy
cp examples/[Link] /etc/[Link]
The installation is now completed. However, we have only an example configuration file installed
at ‘/etc/[Link]’ this is the file that will store all of the settings that we require to ensure our
website is available for the maximum number of visitors. So we now need to edit this configuration
file I’m going to use ‘vim’ but if you are more familiar with ‘nano’, ‘ee’ or another editor please use
that.
vim /etc/[Link]
Have a quick look through the file if you wish and see the basic structure of the configuration file,
we are going to create a VERY basic config to start with just to make sure that our installation is
working.
global
daemon
log /dev/log local4
maxconn 40000
ulimit-n 81000
defaults
log global
contimeout 4000
clitimeout 42000
srvtimeout 43000
listen http1
bind [Link]:80
mode http
balance roundrobin
server http1_1 [Link]:80 cookie http1_1 check inter 2000 rise
2 fall 3
server http1_2 [Link]:80 cookie http1_2 check inter 2000 rise
2 fall 3
Save the above configuration file and then to start the HAProxy service use the following command
from the command line:
/usr/sbin/haproxy -f /etc/[Link]
If everything starts correctly you should be able to browse to your real IP Address using a different
compute and see you default page, as mine are just two Debian Web Server I get the following:

If you see the above image or the page for your servers. Congratulations your two web servers are
now in High Availability mode. If you do not see your default page stop HAProxy with a killall
haproxy command and run /usr/bin/haproxy -d -f /etc/[Link] this will restart HAProxy with
debugging displayed on the console screen to stop the debug info being printed and the HAProxy
Service simply press Crtl+C
Now that the basic High Availability is working lets move to Transparent mode.
So with a stopped HAProxy service open your /etc/[Link] file again with your editor of choice
and in the ‘listen http1 section’ add the following:
option http-server-close
option forwardfor
source [Link] usesrc clientip
You will now need to edit your iptables rules. I have this as my ‘[Link]’ file:
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local [Link]/0 dev lo table 100
If you now run this file and then start your new modified HAProxy file and retest to your web
server on the Real IP Address you should be able to see in the HTTP Access logs that the address
that your site was visited from is not that of the LoadBalancer.

Install Haproxy 1.5 as transparent

loadbalancer for http and https on Ubuntu
Trusty
I've been searching and testing this quite a while until I had a working setup. There are a lot of
articles out there, but here is a solution that has worked for me.
First install haproxy 1.5:
apt-add-repository ppa:vbernat/haproxy-1.5
apt-get update
apt-get install haproxy

It is important that you use at least version 1.5 of haproxy!

First of all edit /etc/default/haproxy and change the line ENABLED=1. You also may want to
change some flags, for example to give haproxy more memory to work with (256MB in this case):
EXTRAOPTS="-de -m 256"
The actual configuration file is /etc/haproxy/[Link].
If you only want to loadbalance http traffic you can use a listen block like this:
listen appname [Link]:80
mode http
balance roundrobin
option httpclose
option forwardfor
server lamp1 [Link]:80 check
server lamp2 [Link]:80 check

You should change appname to a name that suits your application and [Link] to your public IP
address if you do not want haproxy to listen on all IP addresses. Incoming requests on http port 80
are balanced to [Link] and [Link]. Change these IPs accordingly. You may put only one or
several servers.
It is important however to keep the options httpclose and forwardfor in order to be able to identify
the remote client IP on your backend servers (X-Forwarded-For header) – else you'll only see the
haproxy IP which is quite annoying and will prevent some things like geoip from working.
So far it's quite easy, isn't it? When coming to https, we cannot use the listen block, we have to use
frontend and backend:
frontend appname-frontend
mode http
 bind [Link]:80
bind [Link]:443 ssl crt /etc/haproxy/certs/[Link] crt
/etc/haproxy/certs
option httpclose
option forwardfor
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
default_backend appname-backend

backend appname-backend
mode http
server lamp1 [Link]:80 check

So here the frontend is listening on both, http and https ports and forwarding http traffic to the
backend server [Link]. haproxy is doing the SSL Offload.
In case of an https connection, haproxy will look for your certificate in /etc/haproxy/certs/ using
SNI (Server name identification). If it does not find a corresponding certificate it will serve the
[Link] certificate.
So for example if you call [Link] your certificate file should be
/etc/haproxy/certs/[Link].
To have a .pem file you can simply put your private key and certificate in one file with .pem
extension -no line break at the end of the file: cat [Link] [Link]
> [Link]
The reqadd line will add a special header in case of https which will provide you with an
environment variable that helps you specify wether your on https or http. Remind that haproxy in
this case does all the ssl handling and only sends http traffic to your servers!!
In the above example we're using one frontend for http and https. If you want to force your clients
to use https, you can add this line to your frontend block, right before the default_backend line:
redirect scheme https code 301 if !{ ssl_fc }
Of course you can also use different frontends for http and https, here's an example:
frontend appname-frontend-http
mode http
bind [Link]:80
option httpclose
option forwardfor
default_backend appname-backend

frontend appname-frontend-https
mode http
bind [Link]:443 ssl crt /etc/haproxy/certs/[Link] crt
/etc/haproxy/certs
option httpclose
option forwardfor
reqadd X-Forwarded-Proto:\ https
default_backend appname-backend

backend appname-backend
 mode http
server lamp1 [Link]:80 check

Why use different frontends? Here's an example: you might want to redirect some virual hosts to
https and keep others on http – depending on the hostname. to achieve this, add the following
two lines in your http frontend block, right before default_backend:
acl is_https_host hdr_end(host) -i [Link]
redirect scheme https code 301 if is_https_host

Finally as a security measure you should put these 2 lines in your global block on top of the file:
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!
EXP:!MD5:!aNULL:!eNULL
ssl-default-bind-options no-sslv3

It will disable some unsecure ciphers/protocols and keep you safe from some attacks.
A note on using tcp mode to loadbalance https:
If using tcp, it will work of course, your webservers will receive https traffic and haproxy will not
do the SSL offload. But: on your backend servers you will not be able to see the client IPs, you'll
only see haproxy's IP which is quite annoying. However this is not a limitation of haproxy. HTTP is
on OSI layer 7 and TCP on OSI layer 4. Options like http-close or forwardfor will be ignored
when using TCP.
Nevertheless a quick example:
listen appname [Link]:443
mode tcp
balance roundrobin
server lamp1 [Link]:443 check
server lamp2 [Link]:443 check

When you've finished your configuration file, just restart haproxy: service haproxy restart
One final comment on GeoIP: when using apaches geoip module, you should add this line in
/etc/mods-enabled/[Link] for it to work correctly behind haproxy:
GeoIPScanProxyHeaders On

High availability haproxy (transparent mode)

service with keepalived in CentOS 6.6
Published 24 June, 2015 | By admin
Haproxy is a great piece of software that provides high availability, load balancing, and proxying
for TCP and HTTP-based applications. Joined to keepalived you can easily build a high availability
haproxy service, let’s see how to install in a CentOS 6.6 GNU/Linux distribution.
Let us briefly summarize the situation, we have two nodes (haproxy1 and haproxy2) with haproxy
service running and we need high availability between them, for this we will use keepalived (that
use VRRP protocol) in order to control two virtual IP (non-local IP), one for the frontend
([Link]) and other in the backend side ([Link]). If something goes wrong in the node
(haproxy1) that is giving the haproxy service, keepalived will move these non-local IP from one
node to the other haproxy node (haproxy2), graphically:

Let’s start the installation:

# cat /etc/issue
Scientific Linux release 6.6 (Carbon)
Kernel \r on an \m

In this case we will configure haproxy reverse proxy in transparent mode, first install haproxy in the
two nodes:
# yum install haproxy

For the transparent mode we need T_PROXY support activated by default in 1.5.2 version:
# haproxy -vv
HA-Proxy version 1.5.2 2014/07/12

Build options :
TARGET = linux2628
CPU = generic
 CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1
USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :

epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Now it’s necessary to activate the ip_nonlocal_bind kernel parameter in order to permit haproxy
listen in a non-local IP address:
# echo “1” > /proc/sys/net/ipv4/ip_nonlocal_bind

And also turn on the IP forwarding:

# echo “1” > /proc/sys/net/ipv4/ip_forward

Also include these two lines in /etc/[Link] to activate at the system start:
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1

For the transparent mode haproxy use T_PROXY kernel functionality available for kernel version
later to 2.6.28:
# uname -r
2.6.32-504.16.2.el6.x86_64

First we need to configure iptables to mark all tcp packets:

# iptables -t mangle -N DIVERT
# iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
# iptables -t mangle -A DIVERT -j MARK –set-mark 1
# iptables -t mangle -A DIVERT -j ACCEPT

We can check the iptables configuration it’s ok:

 # iptables -L -n -t mangle

[…]

Chain DIVERT (1 references)

target prot opt source destination
MARK all — [Link]/0 [Link]/0 MARK set 0x1
ACCEPT all — [Link]/0 [Link]/0

Then we use RPDB (Routing Policy Data Base) to send all tcp mark packets to the loopback
interface:
# ip rule add fwmark 1 lookup 100
# ip route add local [Link]/0 dev lo table 100

We can check the configuration:

# ip rule show

0: from all lookup local

32765: from all fwmark 0x1 lookup 100
32766: from all lookup main
32767: from all lookup default

# ip route show table 100

local default dev lo scope host

We configure now the haproxy /etc/haproxy/[Link], first in the global section put:
[…]
user root
group root
[…]

In the frontend section we configure haproxy to listen (in transparent mode) in the virtual IP
([Link]):
[…]
bind [Link]:80 transparent
[…]

Finally in the backend section is necessary to include:

[…]
source [Link] usesrc clientip
[…]

We can now start the haproxy service:

# service haproxy start
# chkconfig haproxy on

And check:
# netstat -lntp | grep 80
tcp 0 0 [Link]:80 [Link]:* LISTEN 887/haproxy
We continue with keepalived, first we need to install it in both haproxy nodes:
# yum install keepalived.x86_64

Let’s set keepalived in the haproxy1 editing /etc/keepalived/[Link] file. We

need to create two vrrp instances, one for the virtual IP ([Link]) in the frontend and another
for the virtual IP ([Link]) in the backend side and include these two instances in a group,
also we need to define a vrrp_script to check the availability of the haproxy service:
global_defs {
router_id LVS_PRO
}

vrrp_sync_group G1 {

group {
eth0
eth1
}

}
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weigth 2
}
vrrp_instance eth0 {
interface eth0
state EQUAL
priority 100
virtual_router_id 156
authentication {
auth_type PASS
auth_pass 12345678
}

virtual_ipaddress {
[Link]
}

track_script {
chk_haproxy
}
}
vrrp_instance eth1 {
interface eth1
state EQUAL
priority 100
virtual_router_id 157
authentication {
auth_type PASS
auth_pass 12345678
}
 virtual_ipaddress {
[Link]
}
track_script {
chk_haproxy
}
}
We put the same configuration file in the other haproxy (haproxy2)
node, and start keepalived:
# service keepalived start
# chkconfig keepalived on

We can check if keepalived it’s running correctly for example in the haproxy1 node showing that
the haproxy node has the two non-local IP:
# ip a
[…]

2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:17:3f:68:ab:1d brd ff:ff:ff:ff:ff:ff
inet [Link]/24 brd [Link] scope global eth0
inet [Link]/32 scope global eth0
inet6 fe80::216:3eff:fe68:aa1d/64 scope link
valid_lft forever preferred_lft forever

3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:17:3e:69:aa:1f brd ff:ff:ff:ff:ff:ff
inet [Link]/24 brd [Link] scope global eth1
inet [Link]/32 scope global eth1
inet6 fe80::216:3eff:fe68:aa1e/64 scope link
valid_lft forever preferred_lft forever
[…]

We will test that keepalived it’s working properly switching off the haproxy server in haproxy1
node and check that the keepalived migrate the non-local Ip to the other node:
# service haproxy stop

We will see for example in the log (/var/log/message) in haproxy2 node that keepalived has
activated the two non-local IP:
# tail -f /var/log/message
[…]
Jun 23 15:31:09 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth1) Transition to
MASTER STATE
Jun 23 15:31:09 haproxy2 Keepalived_vrrp[894]: VRRP_Group(G1) Syncing instances
to MASTER state
Jun 23 15:31:09 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth0) Transition to
MASTER STATE
Jun 23 15:31:09 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth1) Transition to
MASTER STATE
Jun 23 15:31:10 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth0) Entering
MASTER STATE
 Jun 23 15:31:10 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth0) setting
protocol VIPs.
Jun 23 15:31:10 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth0) Sending
gratuitous ARPs on eth0 for [Link]
Jun 23 15:31:10 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth1) Entering
MASTER STATE
Jun 23 15:31:10 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth1) setting
protocol VIPs.
Jun 23 15:31:10 haproxy2 Keepalived_vrrp[894]: VRRP_Instance(eth1) Sending
gratuitous ARPs on eth1 for [Link]
[…]

And check if the virtual IP are up in the haproxy2 node:

# ip a
[…]
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:19:3f:69:ab:1d brd ff:ff:ff:ff:ff:ff
inet [Link]/24 brd [Link] scope global eth0
inet [Link]/32 scope global eth0
inet6 fe80::216:3eff:fe68:aa1d/64 scope link
valid_lft forever preferred_lft forever

3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:18:3e:70:aa:1f brd ff:ff:ff:ff:ff:ff
inet [Link]/24 brd [Link] scope global eth1
inet [Link]/32 scope global eth1
inet6 fe80::216:3eff:fe68:aa1e/64 scope link
valid_lft forever preferred_lft forever
[…]

That’s all.

High availability load balancing using

HAProxy on Ubuntu (part 1)
In this post we will show you how to easily setup loadbalancing for your web application. Imagine
you currently have your application on one webserver called web01:
1 +---------+
2 | uplink |
3 +---------+
4 |
5 +---------+
6 | web01 |
7 +---------+
But traffic has grown and you’d like to increase your site’s
capacity by adding more webservers (web02 and web03), aswell as
eliminate the single point of failure in your current setup (if
web01 has an outage the site will be offline).
1 +---------+
2 | uplink |
3 +---------+
4 |
5 +-------------+-------------+
6 | | |
7 +---------+ +---------+ +---------+
8 | web01 | | web02 | | web03 |
9 +---------+ +---------+ +---------+
In order to spread traffic evenly over your three web servers, we
could install an extra server to proxy all the traffic an balance
it over the webservers. In this post we will use HAProxy, an open
source TCP/HTTP load balancer. (see: [Link] to do
that:
1 +---------+
2 | uplink |
3 +---------+
4 |
5 +
6 |
7 +---------+
8 | loadb01 |
9 +---------+
10 |
11 +-------------+-------------+
12 | | |
13 +---------+ +---------+ +---------+
14 | web01 | | web02 | | web03 |
15 +---------+ +---------+ +---------+
So our setup now is:
– Three webservers, web01 ([Link]), web02 ([Link] ), and
web03 ([Link]) each serving the application
– A new server (loadb01, ip: ([Link] )) with Ubuntu
installed.
Allright, now let’s get to work:
Start by installing haproxy on your loadbalancing machine:
1 loadb01$ sudo apt-get install haproxy

Now let’s backup the original haproxy configuration file and

create a new one with our config which will tell haproxy to listen
for incoming http requests on port 80 and balance them between the
three webservers:
loadb01$ sudo mv /etc/haproxy/[Link]
1 /etc/haproxy/backup_haproxy.cfg
2 loadb01$ sudo vi /etc/haproxy/[Link]
Paste the following configuration there:
1 global
2 maxconn 4096
3 user haproxy
4 group haproxy
5 daemon
6
7 defaults
8 log global
9 mode http
10 option httplog
11 option dontlognull
12 retries 3
13 option redispatch
14 maxconn 2000
15 contimeout 5000
16 clitimeout 50000
17 srvtimeout 50000
18
19 listen webcluster *:80
20 mode http
21 stats enable
22 stats auth us3r:passw0rd
23 balance roundrobin
24 option httpchk HEAD / HTTP/1.0
25 option forwardfor
26 cookie LSW_WEB insert
27 option httpclose
28 server web01 [Link]:80 cookie LSW_WEB01 check
29 server web02 [Link]:80 cookie LSW_WEB02 check
30 server web03 [Link]:80 cookie LSW_WEB03 check
Enable HAproxy by editing the /etc/default/haproxy file
1 loadb01$ sudo nano /etc/default/haproxy
and setting ENABLED to 1:
1 # Set ENABLED to 1 if you want the init script to start haproxy.
2 ENABLED=1
3 # Add extra flags here.
4 #EXTRAOPTS="-de -m 16"
Then, start HAProxy:
1 loadb01$ sudo /etc/init.d/haproxy start
Now open your webbrowser and browse to [Link] (or
whatever IP you have set for loadb01), you should be served a file
from one of the webservers! The loadbalancing is now working, but
let’s take a closer look at some of the things we configured in
the HAProxy configuration:
1 listen webcluster *:80
Listen for incoming connections on all interfaces, port 80 (the *
can also be replaced with a single ip address)
1 stats enable
2 stats auth us3r:passw0rd
This enables HAProxy’s statistics interface which you can access
by browsing to [Link] login with the
username and password given and you should see a nice statistics
report like this:

1 balance roundrobin
This line set’s HAProxy’s balancing algorithm to ’roundrobin’
(which is also the default one), it basically makes sure each
subsequent request is handled by the next server in the line. For
other possible algorithms to use here, please check section 4.2 of
Haproxy’s configuration manual:
[Link]
1 option httpchk HEAD / HTTP/1.0
This option enables HTTP checking on the web servers, HAProxy will
issue HTTP requests to / and check for a valid response, if the
webserver does not give a valid response (for example when it’s
down) haproxy will mark the server as down and will not send any
requests to it anymore. You can also see this in the statistics
interface, here’s an example with the webserver on web02 stopped:

1 cookie LSW_WEB insert

2 server web01 [Link]:80 cookie LSW_WEB01 check
3 server web02 [Link]:80 cookie LSW_WEB02 check
4 server web03 [Link]:80 cookie LSW_WEB03 check
The first line in this block enables the use of cookies,
basically, when a user reaches the webcluster group, the cookie
LSW_WEB will be created and the server id (LSW_WEB01, LSW_WEB02,
LSW_WEB03) will be stored in it. For all next requests in the same
session, HAProxy will look at the cookie and redirect that user to
the same webserver (unless it’s down).
The last three lines define the backend webservers which HAProxy will use, you can easily add
more lines here as the infrastructure grows.
Allright the loadbalancing is working and we are almost there, just one thing left to do in this article
and that’s fixing your webserver logs on the web01/web02/web03 servers. Since requests now
changed from:
1 user --> webserver
To:
1 user --> HAProxy --> webserver
You will see the loadbalancer’s ip in the access log on your
webservers. In order to fix this when you are using Apache
webserver open your /etc/apache2/[Link] file and replace
this line:
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-
1
Agent}i\"" combined
By
#LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-
1 Agent}i\"" combined
2 LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%
{Referer}i\" \"%{User-Agent}i\"" combined
Then restart/reload apache and the logging should be fixed, it
will now include the IP address which is send in the X-Forwarded-
For header (This header contains a value representing the client’s
IP address.) that HAProxy includes in all requests to the backend
webserver. We enabled that earlier by setting the
1 option forwardfor
option in the HAPRoxy configuration.
That’s it!, over the course of next weeks we will be posting some more articles on this subject,
covering:
– Adding high-availability for the loadbalancer (as it’s now a single point of failure ;-))
– MySQL database scalability options.
If there’s anything else you’d like us to cover, or if you have any questions please leave a comment!