Back to Debian

A couple of weeks ago, my main workstation went insane. I put this down to the fact that I was running Debian unstable, which made it, well… unstable. I’d always had a lot of trouble with it. Occasionally, files would become corrupt (mostly apt’s status file), or programs would randomly segfault. The most hilarious instance being where fsck segfaulted on startup. Finally, after putting it off for too long and an hour or two of repeated crashes, I decided to force a full fsck.

I wish I hadn’t.

There were so many tragic errors. My /bin directory got wiped and replaced with a single shell script I’d written myself. My /home directory also got wiped and replaced with a 415-bit text file.

I wept and wept. And then went about the process of getting it back on-line.

The reason I chose to run Debian Unstable as opposed to, say, Debian Stable is because I liked the better choice of packages. Debian Stable is stuck with Gnome 2.4, whereas Debian Unstable is up to Gnome 2.6 (2.7 if you include the stuff from Experimental). But not wanting to risk killing my machine again, I decided to look elsewhere. The first thing that came to mind was [fedora][1]. I grabbed the isos from bittorrent and started installing.

Initially, I loved Fedora. Its crisp interface (much nicer than Debian thanks to the improvements in the Xorg that Fedora use) was very easy on the eyes, and the fact that everything just worked really scored big points with me. Between you and me, even the graphical boot screen made me go “Ooh”. For a while, I was pretty happy.

Until, that is, I tried to get it working the way I was comfortable, with the software I like.

It was then that I realised the simple beauty of Debian. For example, I compose my entries into this blog using BloGTK. To install it on Fedora was a matter of

  • Fire up a web browser
  • go to google
  • search for fedora blogtk
  • check a few false-hits
  • download the package
  • rpm -Uvh fedora_package.rpm
  • find out it was missing a few depencies
  • go to google
  • search for fedora dependcyname
  • download the package

Here’s me installing BloGTK on Debian:

Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
  libzvt2.0-0 python-glade2 python-gnome2 python2.3-glade2 python2.3-gnome2
  python2.3-pyorbit
Recommended packages:
  aspell
The following NEW packages will be installed:
  blogtk libzvt2.0-0 python-glade2 python-gnome2 python2.3-glade2
  python2.3-gnome2 python2.3-pyorbit
0 upgraded, 7 newly installed, 0 to remove and 10 not upgraded.
Need to get 477kB of archives.
After unpacking 2048kB of additional disk space will be used.
Do you want to continue? \[Y/n] y```

Do you see how simple that is?

I read somewhere that Fedora has its own apt-alike, called Yum, but I disliked this intensely.  I asked it to install a particular piece of software, something trivial and without too many dependencies, and I watched as the machine sat there for a good ten minutes, calculating its depencies.  Clearly this would not do.

So on Monday, I decided to switch back to Debian.  This time, I swore I would be more careful, I would stay on _stable_ for as long as I could.  I think I lasted an hour or two before I was upgrading to _unstable_.  And almost immediately, I was back to where I began - my `apt` was crashing randomly, programs were segfaulting for no reason.  I went home on Monday night an unhappy bunny.

Yesterday, before I did anything else, I made sure I had all the latest packages, especially `e2fsprogs` and forced a full fsck on reboot with

`shutdown -F now`

It took a while (full fsck on a 120GB disk is about 10 minutes).  But since then, I haven't had an ounce of trouble.  Of course, I'm touching wood with every part of me that isn't needed for working right now, but I'm pretty hopeful about the stability of my _unstable_ box.

I'm not much for morals, but I'd say that if there had to be a moral to this story, it would be: fsck regularly.  Especially if you've just upgraded from _stable_ to \_unstable.

[1]:	http://fedora.redhat.com/

Helpdesk Systems: Eventum

For the last two years, our IT helpdesk has been “powerered” by Zope’s ‘Tracker’ Product. I laughingly refer to it as ‘powered’ because it’s anything but. Right now, we’ve managed to create an ad-hoc system based around Tracker but at the same which tries to avoid it at every step, because Tracker is just plain nasty.

We’re trying to make things better, so for the past couple of weeks, we’ve been trying out a few different products, such as Footprints (good, fully-featured, very expensive) and Auscomp’s IT Commander (cheap, bizarre feature-set). We still haven’t had success finding something that’s suited to our needs, but we’re still looking.

Since I had nothing better to do all weekend, I went looking for other replacement candidates. I stumbled across eventum. Eventum is currently in use by the boys in MySQL AB to handle their technical support. I liked the look of the screenshots, so I installed it on a linux machine at home and tried it out. Here’s what I learned:

  • Very simple install Just unpack the .tar.gz file into a web-accessibly directory on a server with Apache and MySQL
  • Very fast It’s doing a lot, but seems to be pulling the data out pretty quick. And the server I’m running it on isn’t beefy.
  • Intuitive workflow No faffing about trying to figure out how to do certain things.
  • Powerful report generator Provides many different views of your data, which is useful for say, weekly summaries.
  • Multiple projects Create a different eventum tracker for different tasks (IT Helpdesk, Mail Server Upgrade Project etc.)
  • Role-based accounts Unlike ITCommander, these roles are actually useful and tied into your account, so you’re never confused as to what you can do.
  • Anonymous posting No need to sign in to report a problem.
  • Email integration Didn’t get this working at home (because.. well.. I really don’t have a mail server set up at home), but this is tied heavily into the issues. You have the option to notify the person who opened the issue at every change (and also change the list of people who get notified)
  • Time tracking Complete time-tracking integration for proper project management.
  • Phone call tracking There’s an option to update an issue with details of phone calls you’ve made/received regarding this issue. I really like this idea. No more (“Hey, did you ring John Doe about buying that software?"). This is also nicely tied into the time tracking system.
  • Easy to configure/tinker with It provides a powerful administration interface, but it’s also written in really simple PHP. I was able to get my head around the code in an hour or so.

Here are the things I didn’t get to configure and play with, but sounded really bloody nifty from the INSTALL file:

  • Reminder System The reminder system was designed with the objective as serving as a safe net for issues that need attention.
  • Heartbeat Monitor The heartbeat monitor is a feature designed for the administrator that wants to be alerted whenever a common problem in Eventum is detected, like the database server not being available anymore.
  • Command-line interface The Eventum command-line interface allows you to access most of the features of the web interface straight from your command shell. From a personal perspective, this means I can easily automate many things without having to write some custom web-scraping script.

Now, it’s not all roses. Here are the problems I’ve noticed:

  • Possible bugs When I go to close an issue, it will sometimes not budge, and look like it’s not done anything. But it has, it just hasn’t told me. Similarly, when I put in an anonymous issue, it didn’t move. I hit submit five or six times, then finally checked the issue list, and there were five or six anonymous issues. Although this COULD just be the fact that I’ve configured email integration without a working email setup. I’ll have to check this out.
  • Very developer-centric Out of the box, it looks as if it’s geared towards software developers rather than IT helpdesk. However, after an hour or so of changing options in the Admin interface, I managed to make it look and feel more like what we’d need.
  • Open source project As an open-source project, you’re pretty much on your own with very little documentation to guide you. Although there is an active eventum-users mailing list, which could be a pretty good source of support for.

Wario Ware Inc.

Strip the flashy graphics, DTS surround sound and heavy production costs from most of today’s best-selling video games. Strip them right down to their bare-bones essentials and what are you left with? A truly interesting game will manage to maintain its element of “fun” without these things. Nintendo understand this and this is why, pound for pound, they’re producing the most fun of the next-generation giants.

Wario Ware Inc. is a perfect demonstration of this understanding. There are no flashy graphics. The few glimpses of 3D come in the form of static menu screens or non-interactive FMV. What they have produced is stripped-down, bare-bones fun in the form of 200 mini-games. Since most of the first video games were also stripped-down, bare-bones fun, Nintendo have decided to emulate this for their graphical style. Many of the mini-games achieve a visual aesthetic similar to classic Atari games. With a knowing grin, Nintendo acknowledge this, having you play all of these games through an on-screen GameBoy Advance.

In keeping with the faux-GBA interface, this is perfect opportunity gaming. Waiting for a kettle to boil? No problem. Turn on your GameCube and bash about for a couple of minutes - if you can put it down, that is. As well as being opportunity gaming, it’s also a perfect example of just-one-more-go gaming. You fly through the mini-games at such a furious rate that it’s hard to put down. And as your gaming pride kicks in at being beaten by something so simple as glorified “wack-a-mole”, it becomes even more difficult to say no at the “play again” screen.

Also entertaining is watching its effect on non-gamers. I live in a house of non-gamers, and each of my housemates’ reactions to the game has been the same:

“What’s this?”
“Haha, this looks insane”
“Wow, they’re really going for the old-school graphics”

At this point, I usually step out for a cup of coffee or something to keep me twitching like a ten-year old who ate too much sugar (the essential state of being for many of the mini-games) and come back in to find them playing my game.

Project Idea no. 5629

Another idea for a project I may or may never get around to completing (or in this case, starting):

  • list of configuration files/binaries to ‘check’
  • in the case of text-based configuration files, read the file into the database
  • nightly, check the files to see if they’ve been modified (MD5sum)
  • if the configuration file has been modified, diff it and the file already in the database
  • put the output of diff into a database

This would allow an admin quick and easy access to view any changes made to the configuration structure of their machine.

Advantages over using CVS for monitoring

  • automated, no chance of ‘forgetting’ to check a change into CVS
  • non-intrusive
  • easy to view changes
  • easier configuration, easier to roll out onto multiple servers
  • “server roles” automatically selecting default packages - eg “Apache web server” would automatically add /etc/apache/httpd.conf, /usr/sbin/httpd, etc.
  • all done via a central database, eg select samba_config from db_machinename;

update

I’ve been doing some more thinking about this. Here’s one proposed database structure and what each column should store:

db_machinename

  • file_name (name of the config file we’re backing up)
  • config_orig (config file as it was originally)
  • date_orig (date the config file was read in)
  • config_cur (config file as it is today)

db_machinename_diffs

  • file_id (id of the config file we’re dealing with)
  • diff_date (date we took the diff)
  • diff_text (text of the actual diff)

404 Cheat

When our CEO came to me and told me that I had two weeks to design and implement a new website, I knew I was in trouble. In the years since our site first went live, it had grown a lot of ‘cruft’, information that was now completely useless. I started by going to our head of marketing and discussing what didn’t need to be transitioned across to the new website.

Although the CEO was happy with the look of the new site and liked the way we’d trimmed it down, he still insisted that all content be available. So I cheated a little.

I moved the old website to another directory (called ‘oldsite’), and set up a new virtualhost for ‘oldsite.ourdomain.com’. I gave this its own 404 page saying “Your page could not be found”. In the new site, I also gave it a custom 404 page - actually, a PHP script, which would redirect to oldsite.ourdomain.com

So, someone requesting a document we hadn’t transitioned to the new site:

It’s a simple procedure, but one which saved my neck. In the hopes of saving someone else’s neck, here’s that simple 404.php page

<?php
$uri=getenv("REQUEST_URI");
header("Location: http://oldsite.ourdomain.com$uri");
echo $uri;
?>

On Why We Don't Use Challenge-Response

I was recently sent an email from one of our users, evidently impressed by a challenge/response mechanism set up by one of our clients on their mail server (even more impressed by the client’s claim that he received “no spam, ever”). He asked us why we didn’t implement something like this.

Hi Luke,

We’ve been keeping an eye on the challenge/response (C-R) debate for quite some time now. I remember we spent a good while debating over whether we should include it in our anti-spam arsenal. After a lot of consideration, I think we’re going to leave it alone for now, and treat it as a “last line of defense”.

A few of the reasons we are choosing not to roll out a C-R solution:

  1. Increases the amount of non-legitimate mail traffic. This is actually contrary to the goals of an anti-spam solution.
  2. Doesn’t provide as much protection as you’d think. I doubt Eric’s claim of “no junk mail ever”, especially since we regularly get spam emails that are ‘spoofed’ to be from @ourdomain.com addresses.
  3. Trivial to work around. Spammers, for all their misdeeds, are inventive, creative little sods. For example, there was a story recently about spammers getting around Yahoo’s automated-account-creation-prevention tool. When you try to create a Yahoo account, you’re given an image with a word on it, which is hard for machines to easily guess. So what the Spammers redirected this image onto their pornography sites. People joining these sites would type in the word they saw, and this would be fed directly into Yahoo. Sneaky, but impressively so.
  4. Any kind of automated response will just lead to the auto-responding address being added to the spammer’s list of “active” emails. This results in more spam hitting the address.
  5. This, in turn, results in heavier burden on the system.
  6. Speaking of which, most spam comes from non-working or false email addresses. A C-R response to each of these could easily result in a DOS attack on our system.

I could go on, but I think you should see by now that there’s a lot to be said AGAINST C-R systems.

However, one of the things we’re keeping a very close eye on for our anti-spam toolkit is the idea of ‘greylisting’ (www.greylisting.org). A brief rundown on the greylisting method:

  • Unknown person (john.doe@unknowndomain.com) sends an email to myaddress@ourdomain.com
  • ourdomain’s mail server responds with “oops, temporarily unavailable, try again in a minute”
  • ourdomain’s mail server notes that it’s got unknowndomain.com’s mail server in its queue of mails
  • if unknowndomain.com is a proper mail server, it will wait a couple of minutes and try again
  • if unkowndomain.com is using spam software, it will just barf
  • unknowndomain.com’s mail server tries sending the mail again, ourdomain.com’s mail server notes that it passed verification, and “whitelists” @unknowndomain.com

It’s like C-R, but without any of the nasty downsides I listed above. One thing I particularly like about this system is that it doesn’t involve any human interaction. My Grandmother could email me and not get confused by the Challenge-Response mechanism.

We’ll probably be testing out greylisting on our secondary mail server soon, and if all goes well, we’ll roll it out onto our primary mail server.

Open-Source groupware

Something that’s come up quite a bit in work recently has been the idea that we need groupware. What we specifically need is:

  1. shared address book
  2. shared calendar
  3. shared mail directories

The third of these, we’ve managed to hack together using courier-imap. Unfortunately for us, most of our users are very reluctant to move away from POP3, so they’re largely unaware of the availability of this really cool technology.

The second of these, we’ve… well… sort of managed to hack together. Using Outlook 2000/XP/2003’s Free-Busy publishing tool, our users publish to a shared folder which they can all read from. I even wrote a nifty little hack for our ‘resources’ (meeting room, projector, etc.). But one of the limitations of the free-busy information is that it doesn’t list the reason for being busy. It wil just say “John is busy from 10am until 11pm tomorrow”, not “John is in a meeting with Jane and Bob from 10am until 11pm in Meeting Room 1”. So again, this gets underused.

Playing about with Mozilla’s Calendar, we have a tool that does what we want, and allows us to share calendars amongst a team. And it does it all using open standards, so we can get in there and hack around it, if we want. Unfortunately, Sunbird (the codename for the calendar software) is still at a very early stage (0.0.2?), and is barely-usable. Definitely not usable within a production environment.

Shared address books? Nightmare. We have a company-wide address book, with all our email addresses in LDAP (one of these days I’ll get around to integrating this with Sendmail, I promise). But we can’t add to this remotely. Perhaps we can, and I just haven’t figured out how yet. Either way, we need something better.

Yesterday, a crazy thought entered my head. Bear with me, because it sounds a little ‘out there’. Would it be possible to remove Outlook completely, and have everyone work through a ‘thin-client’ (read: browser-based) solution? We already provide a much-loved web interface to mail (using the horde application framework), so would it be possible to extend this some more? Horde offer a module that lets people browse CVS, which I’m sure will appeal to developers. It also allows them to set up filtering rules and vacation notices and, and, and…

Well, this is useless. Of course I can see the advantages of it. I’m already sold on it. Based on their reluctance to give up POP3, I’m worried about how reluctant they’d be to give up their Outlook. People get remarkably attached to their email.

First Post!

Finally got around to watching Danny O’Brien’s Life Hacks talk from NotCon (video). This talk is essentially a “Seven Habits of Highly Effective People” for dorks - Habits of Highly Prolific Geeks. It’s terrifying how many of these habits I seem to have formed myself, and yet am nowhere near being what you would call ‘prolific’ (I sometimes stick a mirror under my nose to make sure I haven’t died).

One of the figures that stood out for me from this talk was that “7% of all posts to livejournal are marked as ‘private’". These are the posts that are visible to noone but the poster. I use these a lot on my own livejournal to keep track of important memories, or even nuggets of code that I might use a lot (the most-used being wget -r -l1 -H -t1 -nd -N -np -A.mp3 http://www.site.com). There are also countless nerd-related thoughts that run through my head on any given day that I’d love to expand upon, but don’t want to do it through livejournal. I’ve got everyone thinking I’m some kind of puerile semi-moron, and I like that. I wouldn’t want to spoil that by posting something that’s actually slightly useful.

So that’s the purpose of this blog. Hopefully I won’t get bored too quickly.