If you or someone you know has symptoms of Too Fscking Clever syndrome, often identifiable as a frequent need to overengineer what should be a simple solution, invite design by committee or overthink a problem ignoring practical usage cases and requirements,  suggest seeking immediate help before TFC develops into Solution Looking For A Problem Disease, which can in turn lead to madness or a career in Windows Server administration.

Treatment often involves simple counselling (of the W. Venema “what problem are you actually trying to solve” method or similar), introduction to Occam’s Razor, or in extreme cases flogging the subject with a copy of Plan 9 until s/he has an epiphany and tries to simply just get the job done.

This has been a Public Service Announcement.

Firstly, I thought I’d never ever see a web framework’s object relationship model used as a database stress tester (cheers Django!). I don’t think it was intended as such but some of the queries it’s generating are just hideous and driving load averages to insane levels.

Secondly - I have two USB keys in front of me. Nothing really unusual about that, except they have “Windows + PHP: Platform of Choice” on them. Yes, I raised an eyebrow too, and I’ve actually done Windows on PHP before at an old workplace.

Choice, PHP and Windows Server from experience boils down to this, in my experience: a) Run as FastCGI or b) Endure a world of suffering (the ISAPI is awful and suphp doesn’t exist).

I’ll of course advocate c) Run it on a Linux box as a DSO / suphp / under mod_fcgid process as your needs dictate - at least they’ll all actually work as you’d expect.

I got the keys from a presentation / open session run by a good friend and former work colleague Jorke Odolphi, now working for Microsoft (but still genuinely interested in F/OSS, I’ve trained him fairly well) and “Professional Geek” Nick Hodge (also a pleasant and well versed chap).

Microsoft having an open session on OSS was not something my curiousity would let me miss, so off I went. I applaud Nick and Jorke for having a go - the turnout wasn’t huge but the session was very lively and definitely worthwhile.

Writing up an auto-installer for FOSS web apps under Win/IIS was a nice touch, and now you know how difficult it is to implement a depsolver IronPython / IronRuby are interesting ideas (the Parrot project / Perl6 peeps are doing almost the same thing and Java has been doing it for ages) but your usage case might be a bit of a corner one. We are all aware in the OSS world how charmingly limited the PHP database drivers can be (hence PECL alternatives and native drivers) so you’re part of a large-ish crowd there

But chaps you’re going to be pushing harder than Sisyphus to get some real FOSS cred for your bosses, if they genuinely want them.

I’m not going to delve deeply the licensing side of things (for good reason, there’s a post of it’s own) but a BSD-like license isn’t likely to get you the community you might want - it’s not really Free and there’s nary a nod to those making code / docs or other contributions (contributor doesn’t mean distributor by the way, if anyone from MS is reading)

The existing culture and mindset and that of Microsoft, it’s partners and some users is fatal to any “open source” initiative in my view (this doesn’t mean I’m unhappy to see an attempt, it shows the strength and relevance of communities like Fedora’s - or any other distribution’s for that matter)

They’re a cathedral; the faithful are handed tools and protocols blessed by the Powers That Be, with such tools/protocols invariably created because they help maintain the status quo (commercially advantageous to the “church”, closely coupled together to ensure/encourage adherence / lock in etc.) - many don’t know of or even see anything outside of the cathedral! Compare to the marketplace that’s Free Software - don’t like a tool/protocol? There’s other stalls with alternatives. Pick up a few and you’re building your own in no time and trying out ideas they may not even had considered before.

That’s the worrying part for the “high priests” - if the “faithful” are choosing their own tools for their own purposes (not just building using blessed tools for those protocols deemed desirable or “holy”) what’s to stop them straying from the rest of the flock? If the market allows you to build your own stall and offer your own wares, why go to the cathedral and be told how what to use?

I’m sure Microsoft’s upper management are happy (even if their outward demeanor may not show it) to see the projects around the Codeplex - still centred almost solely around their own technologies - .NET / IIS / Sharepoint / Windows Live Auth. Nothing really innovative and widely interoperable (and decoupled from other MS tech) like XMPP or memcached / OpenID / Laconica..

The real test will be projects that tie to *genuinely* open tech, a completely non-MS solution. I suspect the “high priests” may quietly sideline such “heresy”, even if the frontline preachers do not.

Lastly: To answer a question posed to me by James Morris via identi.ca: I asked about the IE8 and $10K giveaway: Technically it was mind bogglingly simple: IE8 has a little “feature” where it will display these “web slices” on certain pages like a favourite / bookmark. Find the “splice” with the winning content (on an MS partner site of course) and the 10K is yours.

The splice itself is just a boring standard DIV element with a “hslice” class.
But the hack worked for Microsoft Australia marketing, so I suppose it serves it’s purposes.

Of course there were Chrome and Firefox extensions to support this behaviour almost immediately

The post is bought to you by lekhonee v0.4

I can’t say I’m unhappy to see this or i’ll miss it when it’s gone. An arbitrary definition of “spam” is not so good; providing almost no information to administrators and end users is just plain poor and demanding a “donation” for removal is just plain bovine excrement.

Something I learned from my formative years as a neophyte mail admin-in-training on news.admin.net-abuse.email was that if you wanted to run a blacklist and be taken seriously, you need a fair deal of transparency (ie provide info on why/how a server got listed and a means to resolve the issue) and fairly sane and personable demeanour, and a clear and stricly enforced policy on listing.

Unfortunately SORBS failed all of these in my experience.

One of my old jobs was to handle abuse@ at a Large Australian Hosting Provider (now part of MelbourneIT) along with my regular systems admin / support duties.

Alas, as unfortunately happens in large network / hosting ops, a customer spews some junk. We found and terminated the perp, but not before getting blacklisted.

A quick check of the major lists found the evidence / reason for listing and after informing them that we’d resolved the issue removal was quite swift.

But not SORBS. After jumping through a couple of hoops to find out how / when the servers got listed, no evidence for it’s addition was found aside a single “Recieved:” email header - which is easily forged (and at the time quite popular with spammers to confuse less experienced users/admins)

Our request for more information was met with little more than “I have proof, but I’m not sharing any more” and removal was met with “Donate to the fund supporting Mr Anti-Spammer, who’s being sued for defamation by WeSpamYou Pty. Ltd and I’ll remove it” (names spared to protect the innocent).

W.T.F? Of course the answer was “no” (with the backing of management) especially after I pointed out the case had been settled, in the anti-spammer’s favour. This was changed to a “donate to $charity” after I reminded Mr/Ms Sullivan of that fact.

It still didn’t act as a deterrent (even Legal pointing out that it’s potentially extortion didn’t work!) so I just gave up and stopped bothering with him. You know what they say about arguing with an idiot - they bring you to their level and beat you with experience.

Henceforth, I’ve been advising mail administrators not to use SORBS zones. Customers getting bounces mentioning SORBS got a boilerplate response outlining the situation and why using opaque and arbitrary lists are a Bad Thing (worded appropriately for on-forwarding to ISPs as applicable). I don’t recall ever getting one complaint, as most of the major ISPs here didn’t use it to block mail anyway and smaller players generally got the message once made aware.

There are far better alternatives that don’t generate so many false positives, catch more genuine spam and don’t shake down mail admins / abuse guys for removal. I personally use zen.spamhaus.org for my DNS blacklist needs and it’s never let me down in over 6 years (tied into a multitude of Postfix and Exim installs for small and large mail providers alike)

I migrated my filesystems to ext4 (I’ve come from an all0ext3 F10 to F11 upgrade, FWIW) and just like to mention that if you’re migrating your root filesystem you might want to regenerate your initrd via mkinitrd; the stock one I had didn’t seem to like ext4 (complained about unsupported filesystem options at boot time and failed dismally) but a quick initrd rebuild in rescue mode had me up and running in minutes.

I didn’t see this mentioned anywhere - perhaps I’m a corner case - but I thought I’d record it for posterity in case someone else finds it useful - or desperately needs it!

Backup

If you’re looking for a simple, cron-capable no-fuss delta/differential backup solution (looks over in Peter Gordon’s direction) I’d consider the “rdiff-backup” package, which has been in Fedora for quite some time, does rsync-like backups of reversed diffs - the actual data transfer is small, you can do “point in the past” backups with little fuss plus if you want just the last copy, you can just grab it from the backup tree as-is, without a need to invoke rdiff-backup’s restore (rdiff-backup -r) option. All you need is SSH set up between two hosts, ideally pubkey auth or GSSAPI (ie passwordless or pre-authenticated)

To back up a home dir (~fred) to a remote server, barney:

rdiff-backup /home/fred barney::/home/fred.backup

To restore tmp/wilma from last week (7 days)

rdiff-backup -r 7D barney::/home/fred.backup/tmp/wilma /home/fred/tmp/wilma

How simple is that?

Hacks for database admins

This is an ugly idea, but it’s functional and insanely simple:

Much of my day-to-day systems admin work is with web developers and frameworks. Some frameworks in my experience, whilst being insanely great and powerful for a developer have an unfortunate tendency to hide things under the hood - one example is where the framework defines the database schema (via ORM et. al) but keeps it fairly opaque to the developer - leaving it to the DBA / system admin to work out what may have changed.

This can be a PITA for both developer and systems mangler alike - the developer isn’t always sure if it’s introducing a regression and the DBA/sysadmin wondering about the performance difference.

Both PostgreSQL (pg_dump -s  or –schema) and MySQL (mysqldump -d or –no-data) allow you to save schema-only dumps of databases (I’m not sure about Firebird or others, I’ve not tried them out lately)

I take this at regular intervals and check it in to version control - I can then see changes via standard VC diff commands. It’s a simple hack but it’s functional and requires no extra tools.

This came about because I need to maintain a script that purges old data from a pgsql database, including foreign keys (manually as the schema doesn’t grok DELETE CASCADE) and a change in the FK relations means that my script broke - if I can follow the schema changes it becomes trivial to add in the FK changes needed.

For those users of my package set, you can now use yum-presto to grab deltarpms of my packages for Fedora 10 and 11. This should make your life easier - I know my upstream link isn’t always fast so the less traffic over it the better for you guys - it’s a good thing I don’t do 120+mb packages like Danger From The Deep anymore eh? :-). Feedback welcomed as always.

The last push of Fedora updates hit my local mirror and mostly played nice, except Gallery2 which leads me to:

General Non-Linux-Specific Stuff:

… the gallery, which has borked itself. Oh Gallery2, why do you mock me?

I used to keep a bunch of old photos - taken or just collected, under /gallery.

As some were of an ex-acquaintance of mine, Tina Wallmann, they proved quite popular and widely linked to (she’s a bikini model and has large *ahem* tracts of land :-))

But they’re old, we’ve long since parted company (have not seen her in a couple of years and she seems disinterested in old friends back here in Brisbane, having moved on to a different crowd in Sydney) and it tends to detract from more interesting content on the site, I’ve decided to let ‘em go. I’ll get less page impressions, but it’s quality not quantity I’d prefer. Gentlemen will just have to look elsewhere for their “evening private entertainment”.

The remnants I’ve pushed up to Flickr and integrated into my Wordpress install via Fidgetr. It’s a decent compromise I think.

(For a change I managed to get them out the door before someone emailed me asking where they were. Miracles happen! :-P)

I was quite surprised - and pleased - to check my website stats and find that my most frequent visitor is an IPv6 address:

IPv6 in AWStats for ThatFlemingGent

(If only the GeoIP database had an idea about IPv6 netblock ownership…)

A good friend of mine is a network administrator for a fair size network - two AS’ under his control and a network covering the Australian eastern seaboard. He’s often tasked with finding additional IPv4 address space

Because IPv4 addressing is becoming scarce the registrars in many locales (APNIC in his and my case) set a high bar for new allocations to network service providers (must use 80% of existing allocation, justify new allocations for a max of a /22 last I heard) - and rightfully so. They’re not toffees and they are indeed becoming quite scarce, moreso with increasing takeup of internet-enabled mobile devices and broader broadband availability.

Yes, there are other options such as NAT (Network Address Translation) and name-based virtual hosting to mitigate many issues - but not all applications play nice behind NAT, Voice apps and some games being good examples - and port forwarding isn’t simple for the novice user.

IPv6, step up to the plate! Support in Linux has been around for aeons and it’s rock solid. If you’re already IPv6-enabled, you’re likely talking to me over it now[1]. It’s even on by default with “link-local” fe80:: class addressing ubiquitous on new installs (even if there’s a lot of frankly ordinary advice on turning it off!)

For Fedora, there’s a number of options for public IPv6 - the documents for the “initscripts” package show the basics of IPv6 quasi-native tunnelling and “6to4″ tunnelling and are a good starting point

The latter is easier and a good option if you don’t have a nearby tunnel broker / point of presence like SiXXS, Hurricane Electric or a provider offering a Hexago-like service.

(Australia is a good example - the AARNet educational network offers such a service, as does Internode for it’s customers; Telstra may still do so but that’s it, with Hurricane Electric a higher-latency option down here. Other points of presence are just too distant to be useful)

Wolfgang Rupprecht has a Fedora-specific howto, which applies just as well for F11 or even RHEL/CentOS.

The aiccu package is in the Everything repository if you’re eyeing off a SiXXS tunnel connection.

The “go6″ client from Hexago is another that hasn’t been packaged yet (to my knowledge and while I use it due to my provider’s use of their broker software I’m not really a fan)

HE.NET (Hurricane Electric) lets you use the standard tools, no extra apps needed (bless ‘em!)

The simplest method? 6to4. It’s not as fast as full tunnelling or “native” direct IPv6, but it will get you “on the road” so to speak. Unfortunately NetworkManager currently gets in the way,  going from my testing, but on a headless gateway not using NM it works a charm:

1. Make sure IPv6 is on in your network config: (NETWORKING_IPV6=”yes” in /etc/sysconfig/network)
2. Tell the network the default IPv6 interface to use (set “IPV6_DEFAULTDEV=tun6to4″ in the above file)
3. Add the following lines to your network interface:
   • IPV6INIT=yes
   • IPV6TO4INIT=yes
4. That’s about it - restart the network service and you should be rollin’.

It will use anycast to 192.88.99.1 (default anycast prefix host for 6to4) to find the nearest 6to4 broker and use it as the endpoint. Test by going to a site like www.kame.net (if you see an animated turtle, it’s working) and enjoy.

I’m moving servers next week (a Xen VPS with a fatter pipe) and rest assured it will be IPv6-aware!

[1]

[mfleming@qbert ~]$ host -t AAAA www.thatfleminggent.com
www.thatfleminggent.com has IPv6 address 2001:44b8:62:1b0::1

To our dear Fedora users:  Please don’t attribute  malicious intent where it’s not warranted. I’ve had one comment here and just responded to a thread on fedora-list from users making some frankly melodramatic claims around how / why decisions are made and features disabled/changed/not kept up to date.

A large chunk of us are not on the Red Hat payroll,  we’re volunteers. Why do we do this? Because we enjoy what we do and are passionate about it. These users should remember that we use it too (”eating our own dogfood”) and want to deliver a top quality distribution.

However you can’t please all of the people all of the time - but just because a feature / change doesn’t suit you, doesn’t mean that the developer / packager  is out to get you

The distinct advantage of an open community is just that: it’s an open community.

If you don’t like a feature, suggest/contribute changes and/or send a patch. If the documentation is lacking, why not write up a how-to and publish it, help update the wiki or the distro documentation? Likewise if the art isn’t to your taste,  I’m sure the Art team welcome volunteers. If you’re fairly knowledgeable, share it with other users on the lists / IRC / forums.

A “This is broken, you guys suck and out to get us” attitude is not helpful, please let such attitudes die off.

Cheers,

A user, packager, infrastructure hacker and occasional developer (since Red Hat 5.1)

As releases have progressed the list I offer has become smaller, as many have been pulled into Fedora proper (either by me or other Fedora contributors), died upstream or just dropped due to lack of my/visitor interest

There’s only 58 packages this time - long gone are the old (pre-)Extras days when I had 140+

Highlights: well, there’s um… the GNOME Internode Applet![1]. yet another webserver in Hiawatha! (Think lighttpd with a security focus) - the AIM and MSN Python Jabber/XMPP Transports… A bunch of Wordpress plugins (oh how I’m jonesing for a 2.7/2.8 package for F10!) and other small but useful tools, especially for random hackers and systems people.

There’s a couple of non-starters that I’m still working on - namely the Courier suite (authlib won’t build at the moment, I’m trying to work out why) and the MySQL-memcache UDF functions (memcached_functions_mysql in F9/F10). They’ll be added as soon as they build correctly.

Enjoy, and feel free to drop me a line if you have a suggestion / problem

[1] Internode is my ISP, one of the highest regarded in Australia and for good reason, they’re stable -and it’s owner/CEO Simon isn’t averse to being pranked either:

http://www.youtube.com/watch?v=B3w3R0FkkVQ

I really do like where things are headed and I’m sure the good work will continue There’s been some lively debates on the mailing lists of late, but it’s worked out smoothly and courteously in this humble hacker’s opinion.

All the tested LiveCDs and pre-releases of F11 I’ve tried have been painless which bodes well, as the certainty of me finding something broken / failing / misbehaving with a freshly distribution-upgraded system is often close to 1

Speaking of breakage on the other hand…

I tried to convert my home ejabberd server from the standard mnesia backend to MySQL. using “ejabberdctl convert2odbc” to output the data to flat SQL scripts which you can then import into almost any server (ah, simple, standard transactional SQL, how I love thee :-))

This went really well until I found that I had no MySQL driver for Ejabberd/Erlang installed, and there’s none packaged (ProcessOne has one in their ejabberd-modules Subversion repository, but finding out after the fact is of little comfort, *sigh* )

Oops. Might have to fix that little oversight before retrying. Mea Maxima Culpa

Fortunately I took a backup of the mnesia database before all this (when in doubt, take a backup; when you’re certain - still take a backup!) and I needed it as even after reverting my configuration changes and restarting ejabberd it still wanted to connect to the MySQL service (there were references still in the spool/ on-disk database to it). The restore fixed it in minutes though, fortunately.

I’m planning to release an updated PyICQ transport once Leonidas is out and things have settled (and I have time to test a local scratch build). I also had a look at the python-based Yahoo! Transport, which isn’t very good in my opinion and I already have the MSN Transport packaged here; it’s good and works with the current MSN servers with a little patch. It’s upstream development is slow/”undead”, however which makes me a little hesitant to push it to Fedora proper.

Twitter Fail: Even mentioning Yahoo! tongue in cheek / in passing gets you retweeted by the Yahoo! News bot. Really, if you’re going to let loose any form of artificial intelligence (I use the term loosely) the “intelligence” part is important, yknow.

(On a slight tangent I’m fairly certain a Markov/MegaHAL style bot, if set loose on Twitter, will post more interesting content than most “celebrities” using it to pimp themselves, Stephen Fry being an exception)

I’d advise folks interested in signal to try an open, laconi.ca based solution (like Identi.ca) instead

PS. Yes, I thought about PostgreSQL as the driver is there. Alas WordPress is tied to MySQL only

PPS. Laconica seems very fail-whale free

With that thought out of the way…

Mike McGrath’s memcached plug prompted me to give it a go here, for well, two reasons.

a) I’ve had some (good) experiences in the workplace with it - it’s a boon for database intensive web apps especially and b) because I can and it’s there (which is always a good reason in my ever humble view)

Memcached itself is always a fairly simple install for Fedora - Install via yum (including memcached-selinux if you’re running SELinux - and if you aren’t why not! :-)), give it some options via /etc/sysconfig/memcached eg. CACHESIZE=”64″ (at home, usually “1024″ at work because their app is a lot heavier) start it up and point clients at it.

Wordpress was a touch trickier - there isn’t an “official” Wordpress plugin, with a client available buried in Wordpress Plugins version control (http://plugins.trac.wordpress.org/browser/memcached/trunk/) - which has worked well - if you’re reading this it’s not killed my blog.

To install, grab the above file(s) and drop the object-cache.php file in /usr/share/wordpress/wp-content, set “WP_CACHE” to “true” in wp-config.php and you’re most of the way there.

On the server side, memcached-tool’s “stats” command should start seeing increases in cache hits/misses and cached object numbers.

Adding Andy Skelton’s batcache plugin can help to fine-tune what and how it caches - it’s functional but not as “click-and-drool” as many WP plugins, but how much tweaking do you need to do really?

Drupal was a similar adventure I’ll go into elsewhere; there’s a reasonably simple to install plugin from drupal.org - download, drop into /etc/drupal/all/modules, configure and enable -  and the results just as good.

I was surprised to find related Perl packages not in the main repository (Other major languages are covered - my workplace couldn’t survive without the Python bindings :-)) so I whipped up a package of Cache::Memcached 1.26 (also for RHEL/CentOS) on my own repository, plus I’ll be uploading it for review for Fedora proper[2] as a Perl-using systems admin it’s just too useful not to have (monitoring / stats-gathering scripts for a start :-))

In my continued masquerade as a web developer/SEO maven (which isn’t fooling anyone, I know!) I’ve spent too much time looking at analytics to the point of my poor old eyes turning square and developing line graphs burnt in to my retinas.

At least that’s been a little successful. I have one sticking point in the development side, which is avoiding / dumping web form spam. I could use CAPTCHA but I forsee a lot of visitors finding it off-putting, which is undesirable (it’s for my girlfriend’s business venture). I could use Akismet but that seems more suited to blogging, alas (and I’d need to package the PHP PEAR apps for it anyway).

I’m welcome to other suggestions as always.