I’m IPv6 ready – are you?

Firstly for those wondering about Courier-IMAP / authlib / maildrop+authlib packages for Leonidas:  I’ve built them successfully – only a minor adjustment needed after all that –  and it’s available in the usual place. Enjoy, and let me know if there’s any bugs / issues.

(For a change I managed to get them out the door before someone emailed me asking where they were. Miracles happen! :-P)

I was quite surprised – and pleased – to check my website stats and find that my most frequent visitor is an IPv6 address:

IPv6 in AWStats for ThatFlemingGent
IPv6 in AWStats for ThatFlemingGent

(If only the GeoIP database had an idea about IPv6 netblock ownership…)

A good friend of mine is a network administrator for a fair size network – two AS’ under his control and a network covering the Australian eastern seaboard. He’s often tasked with finding additional IPv4 address space

Because IPv4 addressing is becoming scarce the registrars in many locales (APNIC in his and my case) set a high bar for new allocations to network service providers (must use 80% of existing allocation, justify new allocations for a max of a /22 last I heard) – and rightfully so. They’re not toffees and they are indeed becoming quite scarce, moreso with increasing takeup of internet-enabled mobile devices and broader broadband availability.

Yes, there are other options such as NAT (Network Address Translation) and name-based virtual hosting to mitigate many issues – but not all applications play nice behind NAT, Voice apps and some games being good examples – and port forwarding isn’t simple for the novice user.

IPv6, step up to the plate! Support in Linux has been around for aeons and it’s rock solid. If you’re already IPv6-enabled, you’re likely talking to me over it now[1]. It’s even on by default with “link-local” fe80:: class addressing ubiquitous on new installs (even if there’s a lot of frankly ordinary advice on turning it off!)

For Fedora, there’s a number of options for public IPv6 – the documents for the “initscripts” package show the basics of IPv6 quasi-native tunnelling and “6to4” tunnelling and are a good starting point

The latter is easier and a good option if you don’t have a nearby tunnel broker / point of presence like SiXXS, Hurricane Electric or a provider offering a Hexago-like service.

(Australia is a good example – the AARNet educational network offers such a service, as does Internode for it’s customers; Telstra may still do so but that’s it, with Hurricane Electric a higher-latency option down here. Other points of presence are just too distant to be useful)

Wolfgang Rupprecht has a Fedora-specific howto, which applies just as well for F11 or even RHEL/CentOS.

The aiccu package is in the Everything repository if you’re eyeing off a SiXXS tunnel connection.

The “go6” client from Hexago is another that hasn’t been packaged yet (to my knowledge and while I use it due to my provider’s use of their broker software I’m not really a fan)

HE.NET (Hurricane Electric) lets you use the standard tools, no extra apps needed (bless ’em!)

The simplest method? 6to4. It’s not as fast as full tunnelling or “native” direct IPv6, but it will get you “on the road” so to speak. Unfortunately NetworkManager currently gets in the way,  going from my testing, but on a headless gateway not using NM it works a charm:

  1. Make sure IPv6 is on in your network config: (NETWORKING_IPV6=”yes” in /etc/sysconfig/network)
  2. Tell the network the default IPv6 interface to use (set “IPV6_DEFAULTDEV=tun6to4” in the above file)
  3. Add the following lines to your network interface:
    • IPV6INIT=yes
    • IPV6TO4INIT=yes
  4. That’s about it – restart the network service and you should be rollin’.

It will use anycast to (default anycast prefix host for 6to4) to find the nearest 6to4 broker and use it as the endpoint. Test by going to a site like www.kame.net (if you see an animated turtle, it’s working) and enjoy.

I’m moving servers next week (a Xen VPS with a fatter pipe) and rest assured it will be IPv6-aware!


[mfleming@qbert ~]$ host -t AAAA www.thatfleminggent.com
www.thatfleminggent.com has IPv6 address 2001:44b8:62:1b0::1

Some updates, and a little “open sermon”

  • I’m in the unusual position of being inside, yet rugged up with a jacket and scarf with what feels like a cold. I can’t say I’m happy about it.
  • Upgraded WordPress here to 2.7.1 (Fedora 11 RPM rebuilt for F10) without too many hassles. It kept retrying the database upgrade(!) but disabling the plugins and a little bit of tinkering (clean cookies, rebuild config, even restarted memcached / set SELinux to Permissive) got it working. I suspect clearing memcached was the trick.
  • Sorry to my Planet readers for some cruft in the RSS, the Related Posts plugin needed to be *ahem* disciplined. It should be fixed now.
  • Loving the Leonidas release – the adventure during upgrade (died partway, restarted, left behind most of F10 for some reason) seems to have cleared out a lot of stuff I didn’t need or use and the rest is a definite improvement. Well done to all concerned!
  • Disappointed that our local news sources are running the front pages with soft non-news pap, when there’s rioting and apparent vote fraud in Iran. Their people are suffering and getting shafted, and you’re running crap about actresses doing nude scenes to get ahead…

To our dear Fedora users:  Please don’t attribute  malicious intent where it’s not warranted. I’ve had one comment here and just responded to a thread on fedora-list from users making some frankly melodramatic claims around how / why decisions are made and features disabled/changed/not kept up to date.

A large chunk of us are not on the Red Hat payroll,  we’re volunteers. Why do we do this? Because we enjoy what we do and are passionate about it. These users should remember that we use it too (“eating our own dogfood”) and want to deliver a top quality distribution.

However you can’t please all of the people all of the time – but just because a feature / change doesn’t suit you, doesn’t mean that the developer / packager  is out to get you

The distinct advantage of an open community is just that: it’s an open community.

If you don’t like a feature, suggest/contribute changes and/or send a patch. If the documentation is lacking, why not write up a how-to and publish it, help update the wiki or the distro documentation? Likewise if the art isn’t to your taste,  I’m sure the Art team welcome volunteers. If you’re fairly knowledgeable, share it with other users on the lists / IRC / forums.

A “This is broken, you guys suck and out to get us” attitude is not helpful, please let such attitudes die off.


A user, packager, infrastructure hacker and occasional developer (since Red Hat 5.1)

That Fleming Gent Meets Leonidas

My long-running repository at ThatFlemingGent (or “Enlartenment” for those who haven’t caught up) is live and ready for use.

As releases have progressed the list I offer has become smaller, as many have been pulled into Fedora proper (either by me or other Fedora contributors), died upstream or just dropped due to lack of my/visitor interest

There’s only 58 packages this time – long gone are the old (pre-)Extras days when I had 140+ 🙂

Highlights: well, there’s um… the GNOME Internode Applet![1]. yet another webserver in Hiawatha! (Think lighttpd with a security focus) – the AIM and MSN Python Jabber/XMPP Transports… A bunch of WordPress plugins (oh how I’m jonesing for a 2.7/2.8 package for F10!) and other small but useful tools, especially for random hackers and systems people.

There’s a couple of non-starters that I’m still working on – namely the Courier suite (authlib won’t build at the moment, I’m trying to work out why) and the MySQL-memcache UDF functions (memcached_functions_mysql in F9/F10). They’ll be added as soon as they build correctly.

Enjoy, and feel free to drop me a line if you have a suggestion / problem 🙂

[1] Internode is my ISP, one of the highest regarded in Australia and for good reason, they’re stable -and it’s owner/CEO Simon isn’t averse to being pranked either:


Experiences in both success and failure

I did my bit:

I really do like where things are headed and I’m sure the good work will continue 🙂 There’s been some lively debates on the mailing lists of late, but it’s worked out smoothly and courteously in this humble hacker’s opinion.

All the tested LiveCDs and pre-releases of F11 I’ve tried have been painless which bodes well, as the certainty of me finding something broken / failing / misbehaving with a freshly distribution-upgraded system is often close to 1 🙂

Speaking of breakage on the other hand…

I tried to convert my home ejabberd server from the standard mnesia backend to MySQL. using “ejabberdctl convert2odbc” to output the data to flat SQL scripts which you can then import into almost any server (ah, simple, standard transactional SQL, how I love thee :-))

This went really well until I found that I had no MySQL driver for Ejabberd/Erlang installed, and there’s none packaged (ProcessOne has one in their ejabberd-modules Subversion repository, but finding out after the fact is of little comfort, *sigh* )

Oops. Might have to fix that little oversight before retrying. Mea Maxima Culpa

Fortunately I took a backup of the mnesia database before all this (when in doubt, take a backup; when you’re certain – still take a backup!) and I needed it as even after reverting my configuration changes and restarting ejabberd it still wanted to connect to the MySQL service (there were references still in the spool/ on-disk database to it). The restore fixed it in minutes though, fortunately.

I’m planning to release an updated PyICQ transport once Leonidas is out and things have settled (and I have time to test a local scratch build). I also had a look at the python-based Yahoo! Transport, which isn’t very good in my opinion and I already have the MSN Transport packaged here; it’s good and works with the current MSN servers with a little patch. It’s upstream development is slow/”undead”, however which makes me a little hesitant to push it to Fedora proper.

Twitter Fail: Even mentioning Yahoo! tongue in cheek / in passing gets you retweeted by the Yahoo! News bot. Really, if you’re going to let loose any form of artificial intelligence (I use the term loosely) the “intelligence” part is important, yknow. 🙂

(On a slight tangent I’m fairly certain a Markov/MegaHAL style bot, if set loose on Twitter, will post more interesting content than most “celebrities” using it to pimp themselves, Stephen Fry being an exception)

I’d advise folks interested in signal to try an open, laconi.ca based solution (like Identi.ca) instead 🙂

PS. Yes, I thought about PostgreSQL as the driver is there. Alas WordPress is tied to MySQL only

PPS. Laconica seems very fail-whale free 🙂

Scale and speed and spam

Obligatory Laugh-and-head-shake digression: I’ve been fighting email spam for 12 years plus, and I still see stupid spammers – pardon the tautology – indiscriminately junkmailing abuse@  role addresses. Idiots. Why not just offer crack to a cop?

With that thought out of the way…

Mike McGrath’s memcached plug prompted me to give it a go here, for well, two reasons.

a) I’ve had some (good) experiences in the workplace with it – it’s a boon for database intensive web apps especially and b) because I can and it’s there (which is always a good reason in my ever humble view)

Memcached itself is always a fairly simple install for Fedora – Install via yum (including memcached-selinux if you’re running SELinux – and if you aren’t why not! :-)), give it some options via /etc/sysconfig/memcached eg. CACHESIZE=”64″ (at home, usually “1024” at work because their app is a lot heavier) start it up and point clients at it.

WordPress was a touch trickier – there isn’t an “official” WordPress plugin, with a client available buried in WordPress Plugins version control (http://plugins.trac.wordpress.org/browser/memcached/trunk/) – which has worked well – if you’re reading this it’s not killed my blog.

To install, grab the above file(s) and drop the object-cache.php file in /usr/share/wordpress/wp-content, set “WP_CACHE” to “true” in wp-config.php and you’re most of the way there.

On the server side, memcached-tool’s “stats” command should start seeing increases in cache hits/misses and cached object numbers.

Adding Andy Skelton’s batcache plugin can help to fine-tune what and how it caches – it’s functional but not as “click-and-drool” as many WP plugins, but how much tweaking do you need to do really?

Drupal was a similar adventure I’ll go into elsewhere; there’s a reasonably simple to install plugin from drupal.org – download, drop into /etc/drupal/all/modules, configure and enable –  and the results just as good.

I was surprised to find related Perl packages not in the main repository (Other major languages are covered – my workplace couldn’t survive without the Python bindings :-)) so I whipped up a package of Cache::Memcached 1.26 (also for RHEL/CentOS) on my own repository, plus I’ll be uploading it for review for Fedora proper[2] as a Perl-using systems admin it’s just too useful not to have (monitoring / stats-gathering scripts for a start :-))

In my continued masquerade as a web developer/SEO maven (which isn’t fooling anyone, I know!) I’ve spent too much time looking at analytics to the point of my poor old eyes turning square and developing line graphs burnt in to my retinas.

At least that’s been a little successful. I have one sticking point in the development side, which is avoiding / dumping web form spam. I could use CAPTCHA but I forsee a lot of visitors finding it off-putting, which is undesirable (it’s for my girlfriend’s business venture). I could use Akismet but that seems more suited to blogging, alas (and I’d need to package the PHP PEAR apps for it anyway).

I’m welcome to other suggestions as always.

[1] well, if you’re not reading Planet Fedora via an aggregator anyway..
[2]  Update at 9:07pm AEST: Bug #504403 if someone is keen.

Result may be fit, social with an open messaging standard on top.

Techie Happenings:

I’ve been tinkering with more XMPP (Jabber to you oldies) messaging in general, as it happens. I’ve had a look at the python Yahoo transport and found it a little lacking – it’s config is a little too different to the python MSN / ICQ / AIM transports than I’d personally like, which I’m already packaging and running plus it tends to die/misbehave at unusual times. It has been worth a look though and if enough people would want it, I’ll make the RPM available – but be warned it’s not really to my usual standard.

Speaking of which, there’s a persistent room for chatting around the packages generally – just as an experiment and because I can – at thatfleminggent-rpm@conference.thatfleminggent.com.

I’m still tinkering / debugging my publish-subscribe nodes for thatfleminggent.com (Ejabberd for those interested). I’m sure I’ve configured it right with PEP (Personal Eventing) but I wonder how strong support is client side? Gajim does some PEP (Mood/Activity/Tune for instance) but it updates sporadically if at all.

(Which leads me to wonder how widespread support for that and Service Discovery protocol are client-side, as aside Gajim/PSI it seems sparse and many XMPP goodies are unseen without it. I’d be keen to know more from those in the know)

I’ve signed up on identi.ca (as “thatfleminggent“) and liking it – there’s less noise and spambot followers than Twitter and some very nice XMPP/OpenID integration too.

Better support from the Windows clients would be nice (twhirl does but it’s in the minority, and I’ve preferred TweetDeck or TwitterFox when on a Win desktop) but the Linux clients have been fabulous, especially the current Gwibber version in Fedora.

While there’s been a few “oh no, a slip!” comments being bandied around the tubes regarding the Leonidas release, it’s being done for good reasons. Your august poster here has seen his share of *ahem* “Gold” releases from commercial vendors (no, not just Microsoft but I’ve seen lots of theirs over the years) that many FOSS devs would regard as paper-bag. There’s nothing worse than that corner case bug that inevitably bites us (because Murphy loves a systems tech) so it’s good to delay and get it right than get it out quick and brace for a mess.

Oh, and a protip for those converting local shell accounts to LDAP (or similar) – be prepared to get some uid/gid mismatches unless you’ve been REALLY careful :-). I just did and thought I had consistent ID’s before, but no such luck. It’s trivial to reorganize though.

Life In General:

My girlfriend, bless her heart is a fitness buff and that means that of course yours truly will be convinced to give her current “hey here’s an idea to improve our health” ideas a go.

Thursday’s “let’s go for a good walk” turned out to be a marathon, possibly literally. I’m not sure if I covered a complete 25miles (~42km for those of us on the metric system) on Thursday but it felt like it. This was followed by Friday’s “just a bit of a jog up some stairs” – a 30 degree incline and a 200m stretch! (colloquially known as “The Hill” at Teneriffe in Brisbane) a half dozen times with minimal rest..

Of course I can’t feel anything but lactic acid from the knees down now (I’m not in possession of a runner’s build – I was a weightlifter / shotputter in my youth)

My better half is of course as good as gold 😀

Went to see Angels and Demons (the inner UNIX geek keeps wanting to write it as “Daemons”) on Wednesday night. Not as good as The DaVinci Code, but still better than a lot of the fluff in my local cinemas at the moment. It also helps my local cinema is licensed; a bottle of wine helps get through some poorly chosen movies..

I’m headed back to the office on Monday – feeling better after a break, knowing nothing’s gone amiss in my absence, with a few ideas on how to improve things (equals “make my job easier”) in the back of my mind 🙂

A Poutpourri and the GFC.

It’s felt like a long week so far but I must admit I’m faring better than some – I’m looking over resumes for friends at the moment ( I think I’ve seen more skill matrices than shell prompts today!) who’ve lost work over the past few weeks due to the GFC – redundancies and in one case a business owner who took it as an opportunity to sell up and give his entire staff a week’s notice! If the financial crisis has left you out of work, keep your chin up and don’t lose hope.

I feel that between Bacula’s rotation and volume management keeping me on my toes (tip to the new – RTFM twice, then it makes perfect sense), slavery involving OpenLDAP and MySQL (I should consider Fedora Directory Server and DRBD for master-master replication, but first things first..) and Puppet configuration management (manna from heaven for an admin’s sanity especially if you have more than a handful of servers) I’ve achieved something so far this week. 🙂

Regarding the latter package I’ll put up any useful facts/recipes I write up that work – you have my word. Reductivelabs have a number, but there’s a few omissions that I and perhaps some of you may find handy.

Building some Ruby gems has not been so successful, and if anyone has been successful in rebuilding RPMs of hpricot / json gems on Red Hat Enterprise Linux I’d be interested in your tale.

I’ve mitigated (fixed is the wrong word) my Koji issues, but not in the way I’d like. 🙁 At least I have more information  to work with and things are running in the meantime.

Dan Walsh’s post on confining services through SELinux is definitely getting printed and stuck on my workmates’ noticeboard. I’ve seen a lot of instances where SELinux is switched off instead of thought through and used correctly, which is unfortunate so anything that encourages changes in that behaviour that is a plus :-). It’s not voodoo and it won’t eat your brane, so especially if you have services is sensitive / “wild side” areas the time taken to learn it will be well spent.

Now to sort out those WSGI socket AVC’s of mine… 🙂

I hate it when that happens…

I had a bit of free time – and a half-day at work yesterday to boot – so I thought I’d modernise my set of hacked-up shell scripts (calling mock/createrepo/rsync+ssh) and install the all-singing & dancing Koji suite.

Now I’ve done this at work (1.2 under RHEL 5) with a fair amount of success. No problems on my slightly venerable but functional Fedora 9 server – or so I thought

(The fun that is Kerberos I’ll not go into, I do have a working KDC and other services are fine. I’d also be keen to see a day where Koji is database-independent :-P)

Packages installed OK, configuration looks good, principals look sane and added to keytabs. PostgreSQL database looks spiffy. All is well?

No such luck! Fired up kojiweb and Firefox gives me nothing. Checking the error log, it appears that httpd is an unhappy camper – it’s segfaulted.

So I tell Apache to generate some coredumps (CoreDumpDirectory) and run some gdb magic. I obtain the following enlightenment:

  • I have far too much hooked into Apache and could really do with some cleaning (auth modules in particular lightly used)
  • Something httpd did has tripped up a call to lua libraries. “bt” stops at this gem: block = (*g->frealloc)(g->ud, block, osize, nsize);
  • The only Apache module that uses lua is mod_security, which is too useful to disable. Crap – might have a chat to the fine gents @ Breach about that if I can’t find anything useful.

If anyone else has had a successful crack at Koji+ModSecurity+SELinux (if you’re running a server you do have it turned on and Enforcing, yes!?) or some insight into my problem, I’d be very keen to hear from you 🙂 I have core dumps but they’re 50mb apiece, so I’m not going to post it here (or the backtrace, it’s quite long)

To top things off, my otherwise very useful Blackberry Storm[1] crashes occasionally with an “Error 534”. C’mon Research In Motion, make your error messages useful! Making me pine for the elegance[2] of Windows error messages is bad indeed, k?

I think I’ll avoid production equipment for a little while, I seem jinxed. *sigh*

[1] I’m not enough of a conformo to go with an iPhone and the local telco (not Telstra!) gives free BB/RIM traffic on their plans, which is an Epic Win in my book.
[2] Detect Obvious Sarcasm (y/n/duh)

The Sir Humphrey Appleby Award goes to…

ObFedoraContent: I have now permanently retired the Fedora 7 and Fedora 8 sections of my repository. As I mentioned a few nights ago, those with older releases should really consider upgrading to a supported release or move to RHEL/CentOS. Cheers.

Skip the rest if you’re only wanting things Linux/Fedora related. I don’t mind 🙂

Telstra lays down the law on Twitter


Most of this is the standard “if you do this in an official capacity identify yourself appropriately and don’t disparage the company” wording you’d expect to see from business and that is fair enough.

However, the verbal flotsam around unofficial personal usage is both hilarious and worrying all at once:

They are required to complete an accreditation process and undergo training to update their “knowledge on emerging social trends and evolving best practice in social media”.

This coming from Telstra, whom Australians know understand very little aside taking their subscriber’s money and providing poor to no service? It’s like getting lessons in etiquette from a hillbilly. It will keep the many Brazil-like layers of middle mismanagement there busy for maybe a week, most of it time to discuss forming a committee to investigate the feasibility of research into the impact of such social media.

And this:

A difficult aspect of the guidelines to enforce is the section governing the use of the sites in a personal capacity. If the employee refers to Telstra, they are expected to identify themselves as an employee of the company and ensure they do not imply they are authorised to speak on Telstra’s behalf. “Use a permanent disclaimer if you are referring regularly to Telstra or Telstra-related issues,” the guidelines specify.

Again I think Captain Obvious comes to the rescue here. In this country almost no one has a good word to say about Telstra. If it’s critical it’s sure as hell not officially sanctioned making disclaimers redundant. If it’s positive (and not a press release or prank) then I’d be checking out my window for four shifty characters on horseback.

Asking the poor sods slaving for Telstra to have to publicly out themselves as such seems a bit cruel and unusual 🙂

And frankly a) what right does a company have to influence the personal views of their employees in such a manner anyway and b) are their management  / PR / media people such sensitive flowers that any sort of criticism needs to be suppressed in such a manner? (Overtly or otherwise – noone’s going to mention them if some lame disclaimer is mandated!) My $DEITY people! The rod up your arses must have rods up their arses! A good corporate culture needs constructive criticism and open debate if it’s going to go anywhere…

Fortunately I am not a Telstra employee, nor have I played one on TV (I am however tempted to play one on Twitter for the ironic value)

I have however had the misfortune of working for a company whom Telstra outsourced to (sort of; we took over one of their online divisions, my team effectively making several disgruntled DBAs effectively redundant) so I can say this just as I want to:

Telstra: You couldn’t get a clue if you were in a room full of in-heat clues covered in clue musk. The Peter Principle really wasn’t meant to be used as a management KPI but you’ve managed to do just that and the poor serfs underneath them are treated like expendable worker drones. Trying to counsel them on media relations is insulting to their intelligence as your PR/Marketing has treated the public like morons for years. Don’t pee on my foot and tell me it’s raining. Thanks.

Oh, we know why you’re doing it, don’t tell us it’s not about Fake Stephen Conroy. 🙂 If Leslie Nasser hadn’t done it someone would have – we’ve all been completely Stephen Conroy after about a dozen schooners and a couple of shots. Don’t look so surprised.

Don’t deny the public some potentially comic genius from employees and – more importantly – what really happens within the effective monopoly telco in my great land.

A Koan

A novice once came to a systems software roshi seeking wisdom:

“I am troubled. My project has become large and I am having trouble reproducing the environment across test servers. I have all my sources and I’ve installed them from source in the ways our forefathers have, but some behave in strange ways and clash with other software at odd times. What can I do?”

The older  asked “Find your dependencies and bring them here.”

The novice searched for days amongst his tarballs and local installations, but could only come back with sources, some parchments containing approximate versions, notes and some ldd output.

The master took the notes and the sources, put them in a box, and put the box on the novice’s head.

“In future use these, perhaps build your own – it is not hard. They will manage your installed software better and ease your suffering if you use them wisely and diligently.”

On hearing this the novice was struck with awe.