Atlantic Linux Blog

Backscatter and joe jobs

I guess every blogging sysadmin has to blog about spam at least once. I’ll try to keep my frustration with spammers in check, except to note that I’m absolutely dismayed with the amount of time and effort the world has to spend cleaning up after these miscreants (according to some sources, spam makes up 70-80% of all email sent around the net).

We’re running our own mail server – a Debian GNU/Linux based system using Postfix as the mailer and SpamAssassin for spam filtering. SpamAssassin has been doing a great job of filtering our spam, especially since we started using the Bayesian classifier. SpamAssassin successfully catches most of the hundreds of spams I receive every day.

Last Friday, we noticed a fairly dramatic increase in the number of spams that were getting through our filtering – up from 3 or 4 a day to 10 or 20 a day. Updating our SpamAssassin rules to the latest using sa-learn and retraining the Bayesian classifier seemed to do the trick.

This morning, I arrived to find over 1000 mails in my inbox which were bounces from servers which had been spammed by spammers using aplpi.com email addresses, a type of spamming known as a Joe Job. I wasn’t particularly surprised to find our email address being used by spammers – but I was a bit frustrated at the outcome – the bounces or Backscatter was making it pretty difficult to even see genuine emails in my inbox, never mind respond to them. One of the bloggers I regularly read noted a similar trend for their domain – it looks to me like one or more botnets have ramped up their activity significantly in recent times.

Regardless of the cause of all this, I needed a solution to the problem. A cry for help to the SAGE-IE mailing list pointed me at Justin Mason’s blog on dealing with backscatter using some modifications to Postfix and some enhancements to our SpamAssassin configuration. It took a half hour of reconfiguring to apply Justin’s suggested changes – which immediately resulted in a dramatic reduction in backscatter

The postfix changes alone seemed to catch 75% of the bounces. The SpamAssassin VBounce ruleset tagged the rest (note that you must add your mail relay(s) to whitelist_bounce_relays for this to work properly). For now, I’m going to filter the bounces into a separate folder with the following procmail recipe and review them periodically – so far they’re all junk from misconfigured mailservers,

:0:
* ^X-Spam-Status:.*ANY_BOUNCE_MESSAGE.*
$HOME/mail/bounces

We’re running the latest stable version of Debian on our production servers (Debian 4.0 aka Etch). Unfortunately this doesn’t include the very latest SpamAssassin. In order to ensure we’re running with the absolute newest SpamAssassin rules, not only are we running sa-learn regularly, but I’ve reconfigured our servers to use debian-volatile (which is a repository of backported packages for the stable Debian distributions catering specifically for fast-moving targets such as spam filtering and virus scanning).

I’d like to thank Justin Mason for his excellent blog (and work on SpamAssassin) and the folks on the SAGE-IE list for their prompt responses. Heres hoping this keeps the spammers at bay for few more months!

Atlantic Linux becomes a Red Hat® reseller

We’ve just become a Red Hat® Reseller and Ready Partner. This enhances our ability to provide our customers licenses and subscription renewals for their preferred Linux distribution. It also provides us with access to a wide range of Red Hat® products and technologies which we’d be delighted to demo to any potentially interested customers.

Ghost for Linux

We have a number of laptops in the office for pool use – when someone is travelling to a customer site or a conference they can take one of the pool laptops for development, email and so on. Since these are occasionally used and tend to get knocked around a bit, when we purchased them we went for sturdy middle of the road laptops (the HP Compaq nx6310 in case you’re interested – love those memorable URLs HP) . While this made sense when we purchased them, one of the laptops is being used pretty heavily for Windows development at the minute and is showing some signs of stress. The laptops only have 512MB of memory and 5400rpm hard-drives so I figured some upgrades were worth trying before we move to purchasing a faster laptop.

Memory and drive upgrades for laptops are surprisingly cheap these days – 1GB of DDR2 for the nx6310 cost just €20.50. A 7200rpm notebook drive cost a little more but I figured it was worth upgrading both as we were doing any upgrades. Upgrading memory in the nx6310s is very straightforward, there is a memory expansion port on the underside of the laptop accessed through a panel with a single screw – it took all of 30 seconds.

Upgrading the hard drive is physically very straightforward but of course there is one catch – ideally I’d prefer not to spend a half a day to a day reinstalling Windows XP on the new drive including all the post-SP2 updates and hot-fixes and all of the applications installed (unfortunately we’re not big enough yet for me to justify the time it would take to develop a proper customised install image although I have been looking at tools like nlite to see what’s possible). So I need some way of copying or ghosting the contents of the existing hard-drive and restoring them to the new drive when I swap them. The traditional solution to this was to the use the aforementioned Ghost software – but since we use Linux for a lot of our infrastructure I was more interested in seeing if there were viable alternatives on Linux for doing the same thing.

Some research reveals that the wikipedia page for disk cloning summarises the current Linux-based options pretty well. After looking at the various tools and their functionality, I opted to run with partimage which seemed to be lightweight and capable of doing what I required (dumping the Windows partition from the notebook onto a Linux server and restoring this partition onto a new notebook – all over the network). I had briefly considered just using dd after booting the notebook up with a rescue disk – it would work fine (I’ve used this approach in the past to recover a badly corrupted LVM volume to a new disk) but it is a little less user-friendly than a cloning tool like partimage. One of the benefits of using partimage is that it understands a number of filesystems including NTFS and it’s smart enough to only back up the parts of the filesystem that have data on them, rather than copying the whole partition as dd would. It’s also capable of backing up the Master Boot Record and the partition data, and allowing you to restore them independently of restoring the whole drive.

So partimage it was – I needed client software to run on the notebook and server software to run on a Linux box and receive the partition data read from the client. The partimage guys recommend the SystemRescueCd which is a Live Linux CD which you boot off of and which provides a whole bunch of tools including partimage. I’ve used SystemRescueCd before and it’s well put together and does exactly what it says on the tin. So I downloaded the latest version of that which includes partimage 0.6.6. Note that you seem to need the same version of partimage on the client and the server. I’m using Debian 4.0 on our Linux server which includes version 0.6.4 of the partimage server software. To get around the version incompatibility, I had to go with building the partimage server from a source package downloaded from the partimage site. It sounds worse than it turned out in practice! It’s a pretty painless configure, make, make install after you install a few dependencies.

I compiled my partimage server with ssl and login disabled because it was only running on our local network for a short while under my supervision. If you’re running this permanently, you should probably opt for a more secure configuration. After pointing the partimage server at a writable area on the Linux server (you’ll need a good amount of disk space, partimage can compress backed up images, but you should probably still allow close to the raw size of the partition you are backing up to have some headroom), the laptop was rebooted with the SystemRescueCd.

After booting, the partimage command was started and a basic curses dialog was displayed. I selected the partition we wanted to back up (/dev/sda1) and gave it a name of hostname.partition and pointed it at the server with partimage running. This brought me to a second screen where I specified to use a gzip compressed image and put in a description of “sda1”. After this the backup started and partimage told me it was backing up 17.5GB out of the 37GB NTFS partition (the rest was unused).

The backup took about an hour all told (this over a gigabit LAN – I’d imagine the laptop drive was the bottleneck) after which I installed the new drive in the laptop and again booted with the SystemRescueCd.

Before starting partimage to restore the image, I had to create a partition on the new drive. Partimage doesn’t seem to like running against a drive with no partition (even though I planned to restore the partition and mbr from the partimage backup anyway). So I created a throwaway partition of 10MB and then started partimage. First, I selected the option to restore just the MBR and pointed it at the server. I then selected the image I wanted to restore from the server and proceed with a restore of the MBR and the partition table. When this had finished (it took seconds to do the MBR restore), I exited partimage and verified that the throwaway partition table I had created had been replaced with the partition table from the partimage backup (I used cfdisk, but the SystemRescueCd includes a bunch of different partition tools if you prefer something a little more powerful).

The partition table looked exactly as it had on the original drive, so I restarted partimage pointing it at the server again and went for a full restore of the sda1 image to the sda1 partition this time. This took about 40 minutes, which was faster than the original backup. Since writes are normally a bit slower than reads I was surprised – I’m guessing the speed difference is down to the faster laptop drive but it might be something else. Either way, after 40 minutes partimage told me the image had been restored. So the moment of truth had arrived, I rebooted the laptop and waited to see if it gave me the old “Operating System Not Found …” message or whether it booted back to Windows as it had with the original drive. Success! After a few tense moments, the laptop booted to Windows on the new drive and allowed me to login with the same credentials as I’d used on the old drive. A quick inspection of the environment indicated that it all looked as per the original – and there weren’t any wierd errors in the Windows event logs. As a quick smoke test, I ran a defrag of the Windows drive – I figured if there were any problems with the installation, it was a good way of stress testing the filesystem. There were no problems with the defrag, so unless the main user of the laptop notices any problems when I return it to him, I’m pronouncing this a success.

For users of Ghost, I suspect the interface on Partimage may be a bit rough around the edges, but for anyone that is comfortable with command-line Linux and has done some system administration – Partimage is definitely a very useful tool for disk cloning. I can see myself using this regularly both for migrating systems across hard drives and for backing up critical systems at the partition level.

Tags: disk cloning, ghost, partimage