Today I found some very useful security and networking cheat sheets:

• Security Cheat Sheets
• Networking Cheat Sheets

This week I have been experimenting with the cloud computing provider, VPS.NET.

The application I am trying to scale is a custom built PHP/MySQL web logging application, so unlike many web apps it has more database writes than reads. This is one of the challenges of scaling it, as a central database will be a single point of failure and a bottleneck.

About VPS.NET

The cheapest Virtual Machines at VPS.NET are £15 a month for 400MHZ CPU, 256MB RAM, 10GB disk space and 250GB/month of bandwidth.

VPS.NET have an interesting approach to choosing the power of a virtual machines. Instead of upgrading (and paying for) individual virtual machines, instead you purchase 'nodes' of resources, which you can allocate to 1 or more virtual machines. This gives a lot of flexibility as you can move 'nodes' around between virtual machines within minutes, without having to change your billing amount.

Effectively this means that the virtual machines themselves are free, but the resources to run them are what you must pay for.

Scaling the Web App

I use a traditional load balancing layout of 1 load balancer (with 1 passive failover) and multiple application VPSs.

The load balancer in question is NingX, a very high performance web server and reverse proxy.

Each application VPS was allocated an internal IP address from VPS.NET so that internal traffic would not be counted towards the bandwidth quota. Then Apache, PHP and MySQL were installed onto each application VPS. The database writes are scaled by having each application VPS write to its own local database, then logging data is then merged together in a central reporting suite later.

CentOS Configuration

CentOS is a free community rebuild of the popular RedHat Enterprise Linux distribution.

The only tuning I made to the CentOS basic image provided by VPS.NET was to turn off the firewall. Obviously this is not very secure, but I intend to re-enable it later, however during the testing phase it was causing me problems as the sheer number of TCP connections was causing the iptables connection tracking table to fill up and drop packets.

service iptables stop
chkconfig iptables off

NingX Load Balancer Configuration

I used the latest stable NginX release and also added in the Fair Upstream add-on module.

This is the NginX configuration:

upstream backend  {
        fair;
	server ip1:80   max_fails=20 fail_timeout=10s;
        server ip2:80	max_fails=20 fail_timeout=10s;
        server ip3:80   max_fails=20 fail_timeout=10s;
        server ip4:80	max_fails=20 fail_timeout=10s;
}

server {
        listen          80 default;
        server_name     default;

        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP	$remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout           5;
        proxy_send_timeout              10;
        proxy_read_timeout              15;

        location / {
                proxy_pass              http://backend;
                proxy_cache             off;
        }
}

Application VPS Configuration

Each application VPS runs Apache and MySQL using 1 node of VPS.NET power.

Admittedly this is probably not how I would run it in production, but I wanted to see just how much I could get out of 256MB of RAM!

Apache was tuned so that each instance would not serve more concurrent requests than it could realistically handle, this way the VPS would not get overloaded and slow to a crawl.

Apache's prefork settings:

StartServers	  10
MinSpareServers   10
MaxSpareServers   20
ServerLimit	 256
MaxClients	 30
MaxRequestsPerChild  4000

I also installed the APC PHP accelerator to get better PHP performance.

Siege Test Results

I used an HTTP benchmarking tool called Siege to repeatedly request the web application over the Internet (not on the same network as the VPSs).

Here are the results:

** SIEGE 2.69
** Preparing 500 concurrent users for battle.
The server is now under siege..      done.
Transactions:                      100000 hits
Availability:                      100.00 %
Elapsed time:                       99.71 secs
Data transferred:               80.20 MB
Response time:                        0.42 secs
Transaction rate:             *1002.91 trans/sec*
Throughput:                        0.80 MB/sec (6.4Mbits/sec)
Concurrency:                      *422.61*
Successful transactions:      100000
Failed transactions:                   0
Longest transaction:               21.01
Shortest transaction:                0.00

Conclusion

I was able to achieve 950-1000 transactions a second using this set-up, although the VPS cluster was being pushed to its limits. During the benchmark the NginX load balancer logged many connection errors to the application VPS (as they're MaxClients was set low), however the load on each application VPS did not exceed 4, and did not start swapping to disk.

By configuring NginX with a short connect and read time out, it was able to redirect the request to another application VPS so that the client would not notice.

The beauty of this system is that adding additional application VPS nodes is simple with VPS.NET and so the cluster can be scaled with no downtime.

I have been thinking about the pros and cons of implementing remote access using VPN (such as OpenVPN) vs. an application level remote access such as SSH or Remote Desktop.

Some of the arguments I have seen made for using Remote Desktop over a VPN is that any viruses or malicious software running on the connecting user's computer cannot directly affect the services running inside the corporate network.

Another argument for Remote Desktop is that unlike VPN, the user's computer network is not directly connected to the corporate network, so any malicious traffic coming from the Internet cannot make its way into the remote network - so called 'Split Tunnelling'

Split tunnelling is when a VPN connection is established on a user's computer, but not all of their network traffic is forwarded down the tunnel to the corporate network's gateway. Instead the VPN is configured only to send traffic for the subnets that belong to the corporate network down the VPN tunnel.

This is much more efficient, as normal Internet browsing still goes out of the end user's connection as normal. However it is arguably opening up a security hole because it could allow packets to be routed from the Internet directly into the internal corporate network via the end user's VPN tunnel.

Most VPN software has the ability to force all traffic down the VPN tunnel which prevents traffic from the external Internet being routed down the VPN accidentally. However this feature can be turned off by malicious clients, and there is always the possibility of clients enabling NAT on their computer to forge external traffic to appear to come from their VPN IP address.

So does mean we shouldn't use VPNs?

I would argue not, whilst accidental split tunnelling could cause problems, it can be mitigated by enabling the features in the VPN software to stop this and by configuring a firewall on the VPN terminator to ensure that traffic only comes down the VPN tunnel from the correct IP addresses (no external IP ranges!).

Any malicious user you are allowing to connect to your servers, either by VPN or by Remote Desktop is likely to be able to cause harm.

Even if they cannot directly route traffic from outside into the VPN, they may still be able to steal sensitive data from the internal network using simple 'copy and paste' technique over Remote Desktop.

My good friend Jon Reed has just started a web development blog.

I am looking forward to a lot of cutting edge web development tips in the coming months!

Yesterday morning at 2AM I got back from my holiday in Egypt.

I had a great time, although feeling a little bit sick now, guessing its the water out there.

Whilst I was there, my friend and I visited the Hed Kandi Beach Bar. What a spot, an open air club, right on the beach, playing House music. Perfect!

Just before I left the apartment where I was staying I received a series of SMS messages from my monitoring system alerting me that the ISP that I use was experiencing connectivity difficulties.

A quick call confirmed that the issues were being worked on, so there wasn't much more I could do, whether I was in Egypt or the UK.

Six hours later I arrived back home, after quickly checking my E-Mails to ensure all was OK (it was) I decided to get some sleep. Just as I clicked to shut down my computer, my phone went off with lots of SMS messages.

This time it was the other ISP we use at work having connectivity troubles. Talk about a Welcome Back Present!

Was browsing the Internet today and found a link to this image showing the differences between the same search on Google and Microsoft BING.

The results are pretty interesting.

The Linux distribution CentOS had a troubling time this week with an open letter to the project's founder, Lance Davis, from some of the project's leading developers.

The problems appear to stem from the team being unable to contact Lance Davis, and his control over the centos.org domain and project finances.

However this morning the CentOS web site is showing an announcement that they have resolved the majority of issues during a "routine meeting" with Lance Davis in attendance.

This is excellent news in my opinion as I am an avid user of the CentOS distribution.

Today I read an article on a vulnerability in ISC's DHCP client.

Apparently a specially crafted DHCP response can trigger a buffer overflow in the client and allow arbitrary code execution.

Because the DHCP client runs as root to allow it to modify network settings, it means that any code executed by the exploit is run as root.

Although this is clearly very bad, you still have to have the attacker running a DHCP server from within your local area network. If this was the case then a remote exploit of the DHCP client is likely the least of your worries, as the attacker could modify your network settings to route all your network traffic through itself and potentially monitor all your network traffic for passwords, or redirect you to fake web sites to gather your login credentials.

RHEL and Fedora are not affected thankfully.

Today I found this article from the CentOS project on securing SSH.

I am pleased to say that I have already followed the majority of them, although the brute force prevention technique using iptables looks interesting.

Securing SSH

During some Googling this evening I found a nice surprise!

University of Portsmouth have put my final year dissertation on-line.

I did this back in 2007. It was a fun project to do and I learnt a lot about the darker side of the net, such as the attempts spammers use to verify if a mail server is acting as an open-relay or not.

Check it out here.