Atlantic Linux Blog

Free Linux Training Notes

Free stuff!

When we started our IT consulting company, Applepie Solutions back in 2004 we looked at working in a number of different areas including Java development, C# development, software engineering consulting and Linux support. At the time, we thought there might be an opening for Linux training also, and we figured it was one way of marketing our services to prospective Linux customers. So we added Linux training to our reportoire of services and I set out to put together some training material. We started with an Introduction to Linux course after finding some interested customers around the Galway area. After looking at our options for training material, including possibly licensing some of the main Linux distributors training material, I decided to go ahead and create our own, little realising what a mammoth task preparing good training materials is. My goals in putting together the course (and subsequent ones) were,

• Provide training material that addressed the three main distributions – Red Hat, SuSE and Debian. So I set out to describe the general concept of whatever area we were looking at in the training material and then discuss the specifics of that concept on each of the main distributions.
• Cover what people need to know but give some context. There are things anyone using a Linux system should know. Then our customers pointed out some things they wanted to cover. Finally, I took a look at the content of exams from various organisations such as LPI and tried to address the key points from their perspective also (without, I should mention, covering things such as how to set up a modem on your Linux system – I know there are people out there that still need this, but they’re in a minority in our world anyway).
• Give enough material in the notes that people had a useful reference afterwards also. Okay, in the early days I found this part quite useful for myself also – I mean, no matter what your Linux knowledge is, giving a 5 day training course without any prompts can be intensive – if you can have some guiding notes for your own reference it helps!

I think we mainly succeeded in these goals – most customers we delivered training to gave us excellent feedback – on both our delivery and the quality of our training material.

After some initial success with our Introduction to Linux (and because preparing one training course wasn’t enough of a struggle), I also prepared 2 subsequent courses, one on Linux System Administration and Support intended to help those comfortable with Linux move to the next level of system administration or providing support to Linux users, and a short one on Linux Performance Tuning which scratches the surface of performance tuning and optimisation (it was intended as a half day or day long course for people comfortable with Linux).

We’re currently looking at restructuring our business and while we’re not exactly sure what the future holds yet, we’re pretty sure it won’t include us delivering Linux training. We could have just let the training materials languish at the bottom of our subversion repository, but really, this stuff doesn’t age well and we’ve always committed to giving something back to the Linux/open source/free software (delete as applicable) community where we could. I think this is one small opportunity for us to contribute back and with that in mind, we decided to release our training materials under a Creative Commons license – specifically, the Attribution Noncommercial Share Alike 3.0 Unported license. See the link for details of what you can and can’t do. In a nutshell, you’re free to share and remix the content of the training material as long as you properly attribute it. You’re not allowed to use this material for commercial purposes without our permission – but drop me a line if you do want to use these for some commercial purpose, we should be able to come to some agreement.

So what’s the catch? Apart from the license above? Nothing much really.

• Some of the materials haven’t been updated in about 2 years so they are showing their age in places. The performance tuning material has probably been most affected by this – but there are still plenty of useful concepts in there.
• The material is not without its bugs. Given the timeframes we worked under to prepare this material, there was never enough time to fully proof-read or correct every error, and there are sections that I’d love to rewrite from scratch if I was doing it again. But hey, they’re free!
• The material was all prepared using OpenOffice – for now we’re only making the material available in PDF format, but if someone is interested in other formats, again, let me know what you want them for and we’ll see what we can do.
• The table of contents don’t include page numbers. Sorry. If someone can tell me how to generate a table of contents for OpenOffice impress that includes slide/page numbers, I’ll be happy to rectify this.

So, without further ado, where are the notes? They are available for download from http://www.atlanticlinux.ie/opensource.php. In the words of the hugely successful Sirius Cybernetics Corporation Complaints division – Share and Enjoy.

Please, if you do find them useful – we’d like to hear about it, please let us know. Similarly, if you find any of the aforementioned bugs, let us know and I’ll see about fixing them, time permitting.

Tags: creative commons, free, linux, notes, training

What a difference a Gig makes

We’re working on a project at the moment that involves deploying various Linux services for visualising Oceanographic modelling data using tools such as Unidata’s THREDDS Data Server (TDS) and NOAA/PMELS’s Live Access Server (LAS). TDS is a web server for making scientific datasets available via various protocols including plain old HTTP, OPeNDAP which allows a subset of the original datasets to be accessed and WCS. LAS is a web server which, using sources such as an OPeNDAP service from TDS, allows you to visualise scientific datasets, rendering the data overlaid onto world maps and allowing you to select particular variables from the data which you are interested in. In our case, the datasets are generated by the Regional Ocean Modeling System (ROMS) and include variables such as sea temperature and salinity at various depths.

The data generated by the ROMS models we are looking at uses a curvilinear coordinate system – to the best of my understanding (and I’m a Linux guy, not an Oceanographer, so my apologies if this is a poor explanation) since the data is modelling behaviour on a spherical surface (the Earth) it makes more sense to use the curvilinear coordinate system. Unfortunately, some of the visualisation tools, in particular LAS prefers to work with data using a regular or rectilinear grid. Part of our workflow involves remapping the data from curvilinear to rectilinear using a tool called Ferret (also from NOAA). Ferret does a whole lot more than regridding (and is, in fact, used under the hood by LAS to generate a lot of the graphical output of LAS) but in our case, we’re interested mainly in its ability to regrid the data from one gridding system to another. Ferret is an interesting tool/language – an example of the kind of script required for regridding is this one from the Ferret examples and tutorials page. Did I mention we’re not Oceanographers? Thankfully, someone else prepared the regridding script, our job was to get it up and running as part of our work flow.

We’re nearly back to the origins of the title of this piece now, bear with me!

We’re using a VMware virtual server as a test system. Our initial deployment was a single processor system with 1 GB of memory. It seemed to run reasonably well with TDS and LAS – it was responsive and completed requests in a reasonable amount of time (purely subjective but probably under 10 seconds if Jakob Nielsen’s paper is anything to go by). We then looked at regridding some of the customer’s own data using Ferret and were disappointed to find that an individual file took about 1 hour to regrid – we had about 20 files for testing purposes and in practice would need to regrid 50-100 files per day. I took a quick look at the performance of our system using the htop tool (like the traditional top tool found on all *ix systems but with various enhancements and very clear colour output). There are more detailed performance analysis tools (include Dag Wieers excellent dstat) but sometimes I find a good high-level summary more useful than a sea of numbers and performance statistics. Here’s a shot of the htop output during a Ferret regrid,

What is interesting in this shot is that

• All of the memory is used (and in fact, a lot of swap is also in use).
• While running the Ferret regridding, a lot of the processor is being spent in kernel activity (red) instead of normal (green) activity.

High kernel (or system) usage of the processor is often indicative of a system that is tied up doing lots of I/O. If your system is supposed to be doing I/O (a fileserver or network server of some sort) then this is good. If your system is supposed to be performing an intensive numerical computation, such as here, we’d hope to see most of the processor being used for that compute intensive task, and a resulting high percentage of normal (green) processor usage. Given the above it seemed likely that the Ferret regridding process needed more memory in order to efficiently regrid the given files and that it was spending lots of time thrashing (moving data between swap and main memory due to a shortage of main memory).

Since we’re working on a VMware server, we can easily tweak the settings of the virtual server and add some more processor and memory. We did just that after shutting down the Linux server. We restarted the server and Linux immediately recognised the additional memory and processor and started using that. We retried our Ferret regridding script and noticed something interesting. But first, here’s another shot of the htop output during a Ferret regrid with an additional gig of memory,

What is immediately obvious here is that the vast majority of the processor is busy with user activity – rather than kernel activity. This suggests that the processor is now being used for the Ferret regridding, rather than for I/O. This is only a snapshot and we do observe bursts of kernel processor activity still, but these mainly coincide with points in time when Ferret is writing output or reading input, which makes sense. We’re still using a lot of swap, which suggests there’s scope for further tweaking, but overall, this picture suggests we should be seeing an improvement in the Ferret script runtime.

Did we? That would be an affirmative. We saw the time to regrid one file drop from about 60 minutes to about 2 minutes. Yes, that’s not a typo, 2 minutes. By adding 1 GB of memory to our server, we reduced the overall runtime of the operation by 97%. That is a phenomenal achievement for such a small, cheap change to the system configuration (1GB of typical system memory costs about €50 these days).

What’s the moral of the story?

1. Understand your application before you attempt tuning it.
2. Never, ever tune your system or your application before you understand where the bottlenecks are.
3. Hardware is cheap, consider throwing more hardware at a problem before attempting expensive performance tuning exercises.

(With apologies to María Méndez Grever and Stanley Adams for the title!)

Tags: htop, las, linux, memory, performance, thredds, tuning

It’s not easy to “Be with AIB”

So, given the recent armageddon^H^H^H^H^H … err, turbulence in the financial marketplace, we considered it prudent to follow the old adage of not putting all of our eggs in the one basket. I figured it’s as easy to open a few bank accounts while we have all the documentation pulled together. For reference, if you are a limited company and you want to open a bank account you’ll need the following,

• The certificate of incorporation for your company (some banks seem to require an original, some seem happy with a photocopy)
• The memorandum and articles of association of the company (a certified copy seems ok)
• For at least 2 directors, proof of id e.g. a drivers license and a copy of a recent utility bill for each of those 2 directors.

Finally, you’ll need to complete an application form when you go to the bank.

After some quick research on the internet to find the most local branches of our desired banks, we checked their opening times. This was our first surprise, most banks seem to only open at 10:00am. Or 10:30am for some banks on some days of the week. 10:30am? Since when did banks start working half days (they close again from 1:00pm to 2:00pm and again at 4:00pm, it’s like they don’t want our business).

First to our local Ulsterbank branch, one of the early risers, opening their doors at 10:00am. Clearly these guys want our money! Why Ulsterbank you might ask? No strong reason – their branch is nearby and I’d like us to put some of our eggs in non-Irish banks just to have a good mix (Ulsterbank is owned by Royal Bank of Scotland). The only downside to dealing with a non-Irish bank at the moment is that they only guarantee your deposits up to a maximum of €100,000 as per the Irish government’s recent announcement rather than yesterdays Irish government announcement which only covers Irish banks. In we go and ask to speak to someone about opening a deposit account for our limited company. We’re quickly introduced to a business banking person who takes us into his office, quickly explains our options and then proceeds to take the necessary documentation from us and copy it while we complete a nice minimal set of paperwork which asks us for sensible details about our company, the directors, what kind of account we require and so on. All told, I think it took as 15 minutes. The staff member thanked us at the end of the process and told us our account should be open by tomorrow morning.

Not bad, I thought to myself – clearly they value their customers and are interested in doing business with them.

Since we were on a roll, and we had all the documentation pulled together – we decided to also open an account with an Irish bank due to their recent success in obtaining the full backing of the Irish government (I’d have to start a whole new blog to discuss the wisdom of this, other than to wonder what troubled businesses will next be backed by the government … builders? Oh wait, they’ve committed to doing that already). The banks given unlimited coverage in yesterdays move by the Irish government include,

• Allied Irish Bank
• Bank of Ireland
• Anglo Irish Bank
• Irish Life and Permanent
• Irish Nationwide Building Society
• Educational Building Society

After a long strategic discussion (we checked which one had a branch closest to us) we opted to open our second account with Allied Irish Bank. They’re big, they’re guaranteed by the Irish government and they have lots of branches (and staff, I think I read somewhere before that AIB are by far the largest banking employer in Ireland).

So off we went to their local branch, wondering whether AIB would be even more efficient or whether they’d greet us with the smugness of a bank who’s just been given a large unsigned cheque by the Irish government. Opening their doors at 10:30am on Wednesday, AIB are one of the late risers of the Irish banking scene. We entered the branch, approached the customer service desk and explained that we wished to open a deposit account for a limited company.

“Do you want to do this today?” we were asked.

I wanted to open the account in the next 10 minutes if at all possible but I kept it to a simple “yes” to avoid any confusion.

After clarifying that we had all the required documentation, we were issued with a large lengthy application form to be completed in full (her emphasis, not mine).

I handed over our documentation to be copied – the same documentation that had sufficed not a half hour earlier at Ulsterbank.

“This certificate of incorporation is a copy, we need the original”.

“Why?” I asked.

“We just do”

As luck would have it, we had come prepared and had our pristine original. We handed it over to be reviewed and watch in horror as the AIB staff member proceeded to start stamping everything with a magic AIB stamp, including our original certificate of incorporation. This is a valuable legal document and there was AIB happily stamping it. I pulled my hands back from the counter, fearing where the stamping frenzy would stop and then demanded back our certificate of incorporation (my colleague wasn’t quite so polite and expressed his confusion as to why AIB had defaced our property).

“It has to be stamped” we were informed.

Anyone else that we have to present our certificate of incorporation to will be very impressed that AIB have stamped it – it harkens back to the days of bank managers being trusted … oh wait, they’re still elevated to such a position by the government.

After safely securing the return of our other documentation, our other identification was demanded. After getting the AIB staff member to promise not to stamp our drivers licenses (maybe the Gardai let your drive faster if you’re license has been stamped by AIB?) we handed them over to be copied and went to complete the form.

The first thing that struck me about the form was how much more detail we had to supply. The second thing that struck me is that the form had been designed to look pretty rather than be easy to complete.

I think we’d already spent more time with AIB at this stage than Ulsterbank had consumed, but we felt we must be there and must be within minutes .. or at least 24 hours of having an AIB account. We returned the completed application and received back our forms of identification.

“That should be up in about a week” we were told, in a tone of a voice that suggested we were lucky to be the recipients of such efficient service.

“What?” I asked, “we just opened an account with another bank and were told it would be up and running tomorrow morning”.

“Well, we have to run security checks on your identification and your company”.

“But a week …. ?” I asked?.

The staff member was having none of it. It takes a week to open an account with AIB and thats the end of the matter. Clearly if we didn’t like that state of affairs, we could take our business elsewhere. It’s not like AIB is stuck for cash – oh wait, all of the banks are stuck for cash at the minute, wouldn’t they be glad of your business? So, the Irish government can introduce a new piece of legislation for banks in about 24 hours while it takes one of those banks 7 days to create an account, into which we’ll be putting our cash.

I’ll let you know if we fail the security checks – but it’s clear to me now why at least one Irish bank needed to have the kind of guarantees that the Irish government has offered. They’re not getting business in the door because of how they treat their customers.

Tags: aib, banks, bureaucracy, customer service, ulsterbank