More of Michael’s Not So Quick Tips


I migrated my filesystems to ext4 (I’ve come from an all0ext3 F10 to F11 upgrade, FWIW) and just like to mention that if you’re migrating your root filesystem you might want to regenerate your initrd via mkinitrd; the stock one I had didn’t seem to like ext4 (complained about unsupported filesystem options at boot time and failed dismally) but a quick initrd rebuild in rescue mode had me up and running in minutes.

I didn’t see this mentioned anywhere – perhaps I’m a corner case – but I thought I’d record it for posterity in case someone else finds it useful – or desperately needs it!


If you’re looking for a simple, cron-capable no-fuss delta/differential backup solution (looks over in Peter Gordon’s direction) I’d consider the “rdiff-backup” package, which has been in Fedora for quite some time, does rsync-like backups of reversed diffs – the actual data transfer is small, you can do “point in the past” backups with little fuss plus if you want just the last copy, you can just grab it from the backup tree as-is, without a need to invoke rdiff-backup’s restore (rdiff-backup -r) option. All you need is SSH set up between two hosts, ideally pubkey auth or GSSAPI (ie passwordless or pre-authenticated)

To back up a home dir (~fred) to a remote server, barney:

rdiff-backup /home/fred barney::/home/fred.backup

To restore tmp/wilma from last week (7 days)

rdiff-backup -r 7D barney::/home/fred.backup/tmp/wilma /home/fred/tmp/wilma

How simple is that?

Hacks for database admins

This is an ugly idea, but it’s functional and insanely simple:

Much of my day-to-day systems admin work is with web developers and frameworks. Some frameworks in my experience, whilst being insanely great and powerful for a developer have an unfortunate tendency to hide things under the hood – one example is where the framework defines the database schema (via ORM et. al) but keeps it fairly opaque to the developer – leaving it to the DBA / system admin to work out what may have changed.

This can be a PITA for both developer and systems mangler alike – the developer isn’t always sure if it’s introducing a regression and the DBA/sysadmin wondering about the performance difference.

Both PostgreSQL (pg_dump -sΒ  or –schema) and MySQL (mysqldump -d or –no-data) allow you to save schema-only dumps of databases (I’m not sure about Firebird or others, I’ve not tried them out lately)

I take this at regular intervals and check it in to version control – I can then see changes via standard VC diff commands. It’s a simple hack but it’s functional and requires no extra tools.

This came about because I need to maintain a script that purges old data from a pgsql database, including foreign keys (manually as the schema doesn’t grok DELETE CASCADE) and a change in the FK relations means that my script broke – if I can follow the schema changes it becomes trivial to add in the FK changes needed. πŸ™‚

That Fleming Gent Meets Leonidas

My long-running repository at ThatFlemingGent (or “Enlartenment” for those who haven’t caught up) is live and ready for use.

As releases have progressed the list I offer has become smaller, as many have been pulled into Fedora proper (either by me or other Fedora contributors), died upstream or just dropped due to lack of my/visitor interest

There’s only 58 packages this time – long gone are the old (pre-)Extras days when I had 140+ πŸ™‚

Highlights: well, there’s um… the GNOME Internode Applet![1]. yet another webserver in Hiawatha! (Think lighttpd with a security focus) – the AIM and MSN Python Jabber/XMPP Transports… A bunch of WordPress plugins (oh how I’m jonesing for a 2.7/2.8 package for F10!) and other small but useful tools, especially for random hackers and systems people.

There’s a couple of non-starters that I’m still working on – namely the Courier suite (authlib won’t build at the moment, I’m trying to work out why) and the MySQL-memcache UDF functions (memcached_functions_mysql in F9/F10). They’ll be added as soon as they build correctly.

Enjoy, and feel free to drop me a line if you have a suggestion / problem πŸ™‚

[1] Internode is my ISP, one of the highest regarded in Australia and for good reason, they’re stable -and it’s owner/CEO Simon isn’t averse to being pranked either:

Mike’s Mailer Cookbook: SMTP auth, SASL and MySQL

I’m putting this up here as both a reminder to myself and just in case it’s useful to others.

I serve my IMAP user credentials (Courier in my case) from a MySQL backend and know from experience that my users find it very convenient to use the same set of credentials when sending mail with SMTP AUTH.

Now I can extract details and add them to /etc/saslauthdb2 et. al but a) it’s a little too fiddly especially when you’ve got all the security bells and whistles turned on and b) it seems like needless replication, and I’m big fan of KISS theory in systems admin.

So why not use the Cyrus-SASL SQL plugin:

  • Install the SQL plugin – “yum install cyrus-sasl-sql”
  • (For postfix) Create an “smtpd.conf” file in /usr/lib(64)/sasl2″ containing something similar to the following:

pwcheck_method: auxprop
auxprop_plugin: sql
mech_list: cram-md5 digest-md5
sql_engine: mysql
sql_hostnames: <my-db-server>
sql_user: <my-user-with-select-grant>
sql_passwd: <my-sql-password>
sql_database: <my-db-with-imap-creds?
sql_select: SELECT clear FROM passwd WHERE id = ‘%u@%r’ AND active = 1

  • Enable SMTP AUTH inΒ  your config (Read The Fine Material for that)
  • Give it a test run with your MUA of choice.

I personally only offer CRAM/DIGEST (not plaintext LOGIN/PLAIN) because I’m paranoid – you mileage my vary; I also use the stock standard SQL schema that courier-authlib prefers and use user@domain (“%u” – user, “%r” SASL realm, commonly your domain) for virtual login names – adjust the sql_select statement above to suit your environment, and switch “clear” for “crypt” if you want to offer PLAIN / LOGIN instead.

I’ve been using this for what feels like forever but it’s cheap and has served me well; I’m always open to improvements though.

Random musings for an autumn evening.

  • Is it some extension of Sod’s Law that the day after I build a brand new package, upstream will put out a new version that fixes some serious bug? libmemcached is a good example. I’ve just pushed 0.28 the ThatFlemingGent repo less than a day after 0.27. Are the folks watching me? πŸ™‚
  • Collary: Software useful to me / my work will have an overly generic name, leading to packaging naming / namespace fun. Would anyone like to hazard a guess at what “statsproxy” does? (Tip: It’s not for webserver logs). *grumble*
  • Two reverse proxies, two webservers running both Apache *and* lighttpd and two PostgreSQL boxes to run one site: A sign of a systems architect trying to be too fscking clever?
  • Said setup is active-passive. Trying to be too clever and failing? πŸ˜› (ps. it’s not mine)
  • I seem to have a real menagerie of Twitter followers; After following Neil “The Game” Strauss I got a gaggle of pickup artists, I have the requisite band of “social media” professionals (hey, beats working πŸ˜› – except @lozz ‘cos she’s working for a good cause) I even picked up a couple of boozers after mentioning vodka in a tweet. I often wonder “what on earth are you following little ol’ me for?”. The real-life friends I understand.. The motivations behind others would be interesting..
  • I am a systems admin by trade (audience: “You don’t say!”) but I’ve really gone off coffee, which might have something to do with most of the local blends tasting like wet road base. Yeah, I know I’m probably going to get drummed out of the Secret Society of Systems Admins (TINSocietyOfSystemsAdmins) but I’m a tea man – chai mainly, but for those times when you need to be alert at some insane hour ie. most days it doesn’t seem to cut it. Suggestions for a sane caffeine hit for those of us who absolutely positively have to be up most of the night? πŸ™‚

Oh, and if you’re reading this via Planet Fedora – hi there (*waves from his home in Brisbane, .au*). I’m the GeoIP / ModSecurity package guy, among other things.

I’m more BOFH than hacker/coder (not that it’s stopped me patching / rewriting things before in an emergency). I’ll post something with more signal in the future – I give you my word as a gentleman πŸ™‚

Memcache related packages

Oh, and the libmemcached and MySQL-memcache UDF packages I mentioned last night? They’re up and available for Fedora and RHEL in the usual place (look down the page if you’re lazy). Please give them a whirl, feedback welcomed.

I’m also pleasantly surprised to see a new version of phpldapadmin pop up in Fedora updates tonight – I thought that was deader than General Franco πŸ™‚

Oh, and I’m giving serious consideration to the Planet Fedora idea – if my hackergotchi scares children and small cats, well – you’ve been warned. πŸ™‚

Where to from here?

I seem to have found myself at a crossroads. Not a personal one (well, not really) but hobby / professional. Make of that what you will.

Firstly, I’ve been doing IT support and systems administration for a living for nigh on nine years now. I’d been a Linux tinkerer well before then (I got into mail systems first due to spam fighting; Sendmail and UNIX / Linux with Slackware back in 1997). I did a long stint doing Windows Server systems admin – which is actually fairly interesting tech sometimes, even if it can be clunky and buggy – and do Red Hat for a living again now.

But it can get old. Over the years IT has become something of a “regular” service, the support staff and admin like janitors and there’s no glamour in the job. Depending on the area and your businesses’ line of work there’s fewer interesting “toys” to tinker with. This describes my situation well.

In short I think I need a good project to sink my teeth into. DotProfile is doing things I’ve already done to death (XMPP and messagingΒ  / DNS) and I’m not really a programmer per se. I’ve done OpenID, as you can see here. Ditto IPv6, also on show here. My infrastructure does DKIM, GeoIP, SPF, greylisting, XMPP. I’ve even had a disasterous run as a Tor exit node.

I’ve been asked if I’d do webhosting – no thanks; there are far too many kids (in the literal and figurative sense) overselling their little VPS to small business for a razor-thin profit. I don’t want to get in amongst that shit, I take pride in quality servers and reliability.

Even the package repository for Fedora feels thin. There’s only so much out there in the way of interesting software to package, and even new packages in Fedora proper are getting a little obscure. I suspect most of my users are here for Courier IMAP anyway πŸ˜‰

I do memcached at work (and I’m building libmemcached in the background here, just because) but I personally have little use for it. Do I go back to my usenet roots and run INN? (Am I that much of a masochist? I remember cyclic overview rebuilds even now, they weren’t fun). I suck at art so design is out, and I just can’t consider being a bigger blogger / pundit or SEO hack without laughing at the thought. Cloud computing? Yes please – but how / who will bankroll? πŸ™‚

I’m a systems architect at heart. /me needs a challenge, fast.

(Or I end up getting ITIL / PRINCE2 qualifications, sell out and become management ;-))

Gentlemanly Gaming

Per my musing yesterday, I’ve run up some game servers on qbert.

Crossfire is turned off and I’ll wager no one notices. πŸ˜€

I have Battle For Wesnoth and Teeworlds running right now, ready to go. The Linux crowd will know Wesnoth well, and Teeworlds is good fun for a bit of brainless blasting.

Armacycles/Armagetron is installed, just a matter of configuration before running it up.

The Windows crowd need not fret – all have Windows clients too.

The Game Servers page has all the info you need.