Posted on

How polite!

I’m testing the Prosody XMPP server package I’d mentioned some time back (I’ve worked with jabberd/jabberd2 and ejabberd, figured a new tinker toy was on the cards) and got this when trying to check the service status – as a non-privileged serf/peon

mfleming@pong ~]$ /etc/init.d/prosody status
Prosody is not running

Note:
You will also see this if prosodyctl is not running under
the same user account as Prosody. Try running as root (e.g.
with ‘sudo’ in front) to gain access to Prosody’s real status.
[mfleming@pong ~]$ su –
Password:
[root@pong ~]# /etc/init.d/prosody status
Prosody is running with PID 17701

If only more software packages were that polite and helpful 🙂 (looks in the direction of some larger software concerns that shall remain nameless)

Posted on

New hosting arrangements

If you’re reading this (and not a “Page Cannot Be Displayed” or “Internal Server Error”) then I’ve successfully moved my site(s) to the new server.

After years of hosting off my own gear, either at datacentres or literally in-house I’ve moved to a Xen VPS at Slicehost, running CentOS 5. Setup and migration was relatively painless (fat-fingered a DNS record however, I won’t be doing that again..) and their default bare-bones install is an absolute boon for keeping an instance clean – I hate cluttered installs full of cruft I don’t need/want.

I’ve even got low-ping IPv6 to most of the world now via Hurricane Electric (just as I remember it – simple and do it yourself, which also meant pinging the service to keep the tunnel up :-D)

The only nit I’ve found has been the lack of SELinux support on their guests – I’ve had it in Enforcing mode on my servers for as long as I can remember it being offered – it just feels weird and wrong not having it now 🙂

I’ve also found that there’s a few Fedora packages not available in EPEL that I just can’t live without (postgrey and linux_logo!). I’m not sure why they’ve not been picked up but I’m tempted to do so myself if there’s been no takers – give EPEL some love, it’s nice and stable and won’t do the dirty on you 🙂

Posted on

Too Fscking Clever

  • If you have six discrete crontabs for a relatively small set of tasks instead of two (or even just one) you may be Too Fscking Clever.
  • If your SQL statement has five or more JOINS in it, you may be Too Fscking Clever (or a web development framework ORM)
  • If you are asked to forward ports to a host and instead DNAT the entire IP address, you may find that your Too Fscking Clever-ness will bite your arse when the usage case changes. Thinking it won’t is a sign of severe Too Fscking Clever Syndrome.
  • “We should cache this data for as long as humanly possible (what’s stale data?)” or “We should only cache this in RAM for 5mins (and refetch from a  large table?)” are classic Too Fscking Clever symptoms.

If you or someone you know has symptoms of Too Fscking Clever syndrome, often identifiable as a frequent need to overengineer what should be a simple solution, invite design by committee or overthink a problem ignoring practical usage cases and requirements,  suggest seeking immediate help before TFC develops into Solution Looking For A Problem Disease, which can in turn lead to madness or a career in Windows Server administration.

Treatment often involves simple counselling (of the W. Venema “what problem are you actually trying to solve” method or similar), introduction to Occam’s Razor, or in extreme cases flogging the subject with a copy of Plan 9 until s/he has an epiphany and tries to simply just get the job done.

This has been a Public Service Announcement.

Posted on

Things I thought I’d never see

If I hadn’t been present for these, I’d probably call myself a bullspit artist. But I swear on a stack of $documents I was there and my experiences are 100% true.

Firstly, I thought I’d never ever see a web framework’s object relationship model used as a database stress tester (cheers Django!). I don’t think it was intended as such but some of the queries it’s generating are just hideous and driving load averages to insane levels.

Secondly – I have two USB keys in front of me. Nothing really unusual about that, except they have “Windows + PHP: Platform of Choice” on them. Yes, I raised an eyebrow too, and I’ve actually done Windows on PHP before at an old workplace.

Choice, PHP and Windows Server from experience boils down to this, in my experience: a) Run as FastCGI or b) Endure a world of suffering (the ISAPI is awful and suphp doesn’t exist).

I’ll of course advocate c) Run it on a Linux box as a DSO / suphp / under mod_fcgid process as your needs dictate – at least they’ll all actually work as you’d expect. 🙂

I got the keys from a presentation / open session run by a good friend and former work colleague Jorke Odolphi, now working for Microsoft (but still genuinely interested in F/OSS, I’ve trained him fairly well) and “Professional Geek” Nick Hodge (also a pleasant and well versed chap).

Microsoft having an open session on OSS was not something my curiousity would let me miss, so off I went. I applaud Nick and Jorke for having a go – the turnout wasn’t huge but the session was very lively and definitely worthwhile.

Writing up an auto-installer for FOSS web apps under Win/IIS was a nice touch, and now you know how difficult it is to implement a depsolver 🙂 IronPython / IronRuby are interesting ideas (the Parrot project / Perl6 peeps are doing almost the same thing and Java has been doing it for ages) but your usage case might be a bit of a corner one. We are all aware in the OSS world how charmingly limited the PHP database drivers can be (hence PECL alternatives and native drivers) so you’re part of a large-ish crowd there 🙂

But chaps you’re going to be pushing harder than Sisyphus to get some real FOSS cred for your bosses, if they genuinely want them.

I’m not going to delve deeply the licensing side of things (for good reason, there’s a post of it’s own) but a BSD-like license isn’t likely to get you the community you might want – it’s not really Free and there’s nary a nod to those making code / docs or other contributions (contributor doesn’t mean distributor by the way, if anyone from MS is reading)

The existing culture and mindset and that of Microsoft, it’s partners and some users is fatal to any “open source” initiative in my view (this doesn’t mean I’m unhappy to see an attempt, it shows the strength and relevance of communities like Fedora’s – or any other distribution’s for that matter)

They’re a cathedral; the faithful are handed tools and protocols blessed by the Powers That Be, with such tools/protocols invariably created because they help maintain the status quo (commercially advantageous to the “church”, closely coupled together to ensure/encourage adherence / lock in etc.) – many don’t know of or even see anything outside of the cathedral! Compare to the marketplace that’s Free Software – don’t like a tool/protocol? There’s other stalls with alternatives. Pick up a few and you’re building your own in no time and trying out ideas they may not even had considered before.

That’s the worrying part for the “high priests” – if the “faithful” are choosing their own tools for their own purposes (not just building using blessed tools for those protocols deemed desirable or “holy”) what’s to stop them straying from the rest of the flock? If the market allows you to build your own stall and offer your own wares, why go to the cathedral and be told how what to use?

I’m sure Microsoft’s upper management are happy (even if their outward demeanor may not show it) to see the projects around the Codeplex – still centred almost solely around their own technologies – .NET / IIS / Sharepoint / Windows Live Auth. Nothing really innovative and widely interoperable (and decoupled from other MS tech) like XMPP or memcached / OpenID / Laconica..

The real test will be projects that tie to *genuinely* open tech, a completely non-MS solution. I suspect the “high priests” may quietly sideline such “heresy”, even if the frontline preachers do not. 😛

Lastly: To answer a question posed to me by James Morris via identi.ca: I asked about the IE8 and $10K giveaway: Technically it was mind bogglingly simple: IE8 has a little “feature” where it will display these “web slices” on certain pages like a favourite / bookmark. Find the “splice” with the winning content (on an MS partner site of course) and the 10K is yours.

The splice itself is just a boring standard DIV element with a “hslice” class.
But the hack worked for Microsoft Australia marketing, so I suppose it serves it’s purposes.

Of course there were Chrome and Firefox extensions to support this behaviour almost immediately 🙂

The post is bought to you by lekhonee v0.4

Posted on

Vale SORBS, we’ll hardly miss ye…

SORBS is on death’s door.

I can’t say I’m unhappy to see this or i’ll miss it when it’s gone. An arbitrary definition of “spam” is not so good; providing almost no information to administrators and end users is just plain poor and demanding a “donation” for removal is just plain bovine excrement.

Something I learned from my formative years as a neophyte mail admin-in-training on news.admin.net-abuse.email was that if you wanted to run a blacklist and be taken seriously, you need a fair deal of transparency (ie provide info on why/how a server got listed and a means to resolve the issue) and fairly sane and personable demeanour, and a clear and stricly enforced policy on listing.

Unfortunately SORBS failed all of these in my experience.

One of my old jobs was to handle abuse@ at a Large Australian Hosting Provider (now part of MelbourneIT) along with my regular systems admin / support duties.

Alas, as unfortunately happens in large network / hosting ops, a customer spews some junk. We found and terminated the perp, but not before getting blacklisted.

A quick check of the major lists found the evidence / reason for listing and after informing them that we’d resolved the issue removal was quite swift.

But not SORBS. After jumping through a couple of hoops to find out how / when the servers got listed, no evidence for it’s addition was found aside a single “Recieved:” email header – which is easily forged (and at the time quite popular with spammers to confuse less experienced users/admins)

Our request for more information was met with little more than “I have proof, but I’m not sharing any more” and removal was met with “Donate to the fund supporting Mr Anti-Spammer, who’s being sued for defamation by WeSpamYou Pty. Ltd and I’ll remove it” (names spared to protect the innocent).

W.T.F? Of course the answer was “no” (with the backing of management) especially after I pointed out the case had been settled, in the anti-spammer’s favour. This was changed to a “donate to $charity” after I reminded Mr/Ms Sullivan of that fact.

It still didn’t act as a deterrent (even Legal pointing out that it’s potentially extortion didn’t work!) so I just gave up and stopped bothering with him. You know what they say about arguing with an idiot – they bring you to their level and beat you with experience.

Henceforth, I’ve been advising mail administrators not to use SORBS zones. Customers getting bounces mentioning SORBS got a boilerplate response outlining the situation and why using opaque and arbitrary lists are a Bad Thing (worded appropriately for on-forwarding to ISPs as applicable). I don’t recall ever getting one complaint, as most of the major ISPs here didn’t use it to block mail anyway and smaller players generally got the message once made aware.

There are far better alternatives that don’t generate so many false positives, catch more genuine spam and don’t shake down mail admins / abuse guys for removal. I personally use zen.spamhaus.org for my DNS blacklist needs and it’s never let me down in over 6 years (tied into a multitude of Postfix and Exim installs for small and large mail providers alike)

Posted on

I’m IPv6 ready – are you?

Firstly for those wondering about Courier-IMAP / authlib / maildrop+authlib packages for Leonidas:  I’ve built them successfully – only a minor adjustment needed after all that –  and it’s available in the usual place. Enjoy, and let me know if there’s any bugs / issues.

(For a change I managed to get them out the door before someone emailed me asking where they were. Miracles happen! :-P)

I was quite surprised – and pleased – to check my website stats and find that my most frequent visitor is an IPv6 address:

IPv6 in AWStats for ThatFlemingGent

(If only the GeoIP database had an idea about IPv6 netblock ownership…)

A good friend of mine is a network administrator for a fair size network – two AS’ under his control and a network covering the Australian eastern seaboard. He’s often tasked with finding additional IPv4 address space

Because IPv4 addressing is becoming scarce the registrars in many locales (APNIC in his and my case) set a high bar for new allocations to network service providers (must use 80% of existing allocation, justify new allocations for a max of a /22 last I heard) – and rightfully so. They’re not toffees and they are indeed becoming quite scarce, moreso with increasing takeup of internet-enabled mobile devices and broader broadband availability.

Yes, there are other options such as NAT (Network Address Translation) and name-based virtual hosting to mitigate many issues – but not all applications play nice behind NAT, Voice apps and some games being good examples – and port forwarding isn’t simple for the novice user.

IPv6, step up to the plate! Support in Linux has been around for aeons and it’s rock solid. If you’re already IPv6-enabled, you’re likely talking to me over it now[1]. It’s even on by default with “link-local” fe80:: class addressing ubiquitous on new installs (even if there’s a lot of frankly ordinary advice on turning it off!)

For Fedora, there’s a number of options for public IPv6 – the documents for the “initscripts” package show the basics of IPv6 quasi-native tunnelling and “6to4” tunnelling and are a good starting point

The latter is easier and a good option if you don’t have a nearby tunnel broker / point of presence like SiXXS, Hurricane Electric or a provider offering a Hexago-like service.

(Australia is a good example – the AARNet educational network offers such a service, as does Internode for it’s customers; Telstra may still do so but that’s it, with Hurricane Electric a higher-latency option down here. Other points of presence are just too distant to be useful)

Wolfgang Rupprecht has a Fedora-specific howto, which applies just as well for F11 or even RHEL/CentOS.

The aiccu package is in the Everything repository if you’re eyeing off a SiXXS tunnel connection.

The “go6” client from Hexago is another that hasn’t been packaged yet (to my knowledge and while I use it due to my provider’s use of their broker software I’m not really a fan)

HE.NET (Hurricane Electric) lets you use the standard tools, no extra apps needed (bless ’em!)

The simplest method? 6to4. It’s not as fast as full tunnelling or “native” direct IPv6, but it will get you “on the road” so to speak. Unfortunately NetworkManager currently gets in the way,  going from my testing, but on a headless gateway not using NM it works a charm:

  1. Make sure IPv6 is on in your network config: (NETWORKING_IPV6=”yes” in /etc/sysconfig/network)
  2. Tell the network the default IPv6 interface to use (set “IPV6_DEFAULTDEV=tun6to4” in the above file)
  3. Add the following lines to your network interface:
    • IPV6INIT=yes
    • IPV6TO4INIT=yes
  4. That’s about it – restart the network service and you should be rollin’.

It will use anycast to 192.88.99.1 (default anycast prefix host for 6to4) to find the nearest 6to4 broker and use it as the endpoint. Test by going to a site like www.kame.net (if you see an animated turtle, it’s working) and enjoy.

I’m moving servers next week (a Xen VPS with a fatter pipe) and rest assured it will be IPv6-aware!

[1]

[mfleming@qbert ~]$ host -t AAAA www.thatfleminggent.com
www.thatfleminggent.com has IPv6 address 2001:44b8:62:1b0::1
Posted on

Mike’s Mailer Cookbook: SMTP auth, SASL and MySQL

I’m putting this up here as both a reminder to myself and just in case it’s useful to others.

I serve my IMAP user credentials (Courier in my case) from a MySQL backend and know from experience that my users find it very convenient to use the same set of credentials when sending mail with SMTP AUTH.

Now I can extract details and add them to /etc/saslauthdb2 et. al but a) it’s a little too fiddly especially when you’ve got all the security bells and whistles turned on and b) it seems like needless replication, and I’m big fan of KISS theory in systems admin.

So why not use the Cyrus-SASL SQL plugin:

  • Install the SQL plugin – “yum install cyrus-sasl-sql”
  • (For postfix) Create an “smtpd.conf” file in /usr/lib(64)/sasl2″ containing something similar to the following:

pwcheck_method: auxprop
auxprop_plugin: sql
mech_list: cram-md5 digest-md5
sql_engine: mysql
sql_hostnames: <my-db-server>
sql_user: <my-user-with-select-grant>
sql_passwd: <my-sql-password>
sql_database: <my-db-with-imap-creds?
sql_select: SELECT clear FROM passwd WHERE id = ‘%u@%r’ AND active = 1

  • Enable SMTP AUTH in  your config (Read The Fine Material for that)
  • Give it a test run with your MUA of choice.

I personally only offer CRAM/DIGEST (not plaintext LOGIN/PLAIN) because I’m paranoid – you mileage my vary; I also use the stock standard SQL schema that courier-authlib prefers and use user@domain (“%u” – user, “%r” SASL realm, commonly your domain) for virtual login names – adjust the sql_select statement above to suit your environment, and switch “clear” for “crypt” if you want to offer PLAIN / LOGIN instead.

I’ve been using this for what feels like forever but it’s cheap and has served me well; I’m always open to improvements though.

Posted on

Random musings for an autumn evening.

  • Is it some extension of Sod’s Law that the day after I build a brand new package, upstream will put out a new version that fixes some serious bug? libmemcached is a good example. I’ve just pushed 0.28 the ThatFlemingGent repo less than a day after 0.27. Are the tangent.org folks watching me? 🙂
  • Collary: Software useful to me / my work will have an overly generic name, leading to packaging naming / namespace fun. Would anyone like to hazard a guess at what “statsproxy” does? (Tip: It’s not for webserver logs). *grumble*
  • Two reverse proxies, two webservers running both Apache *and* lighttpd and two PostgreSQL boxes to run one site: A sign of a systems architect trying to be too fscking clever?
  • Said setup is active-passive. Trying to be too clever and failing? 😛 (ps. it’s not mine)
  • I seem to have a real menagerie of Twitter followers; After following Neil “The Game” Strauss I got a gaggle of pickup artists, I have the requisite band of “social media” professionals (hey, beats working 😛 – except @lozz ‘cos she’s working for a good cause) I even picked up a couple of boozers after mentioning vodka in a tweet. I often wonder “what on earth are you following little ol’ me for?”. The real-life friends I understand.. The motivations behind others would be interesting..
  • I am a systems admin by trade (audience: “You don’t say!”) but I’ve really gone off coffee, which might have something to do with most of the local blends tasting like wet road base. Yeah, I know I’m probably going to get drummed out of the Secret Society of Systems Admins (TINSocietyOfSystemsAdmins) but I’m a tea man – chai mainly, but for those times when you need to be alert at some insane hour ie. most days it doesn’t seem to cut it. Suggestions for a sane caffeine hit for those of us who absolutely positively have to be up most of the night? 🙂

Oh, and if you’re reading this via Planet Fedora – hi there (*waves from his home in Brisbane, .au*). I’m the GeoIP / ModSecurity package guy, among other things.

I’m more BOFH than hacker/coder (not that it’s stopped me patching / rewriting things before in an emergency). I’ll post something with more signal in the future – I give you my word as a gentleman 🙂

Posted on

Where to from here?

I seem to have found myself at a crossroads. Not a personal one (well, not really) but hobby / professional. Make of that what you will.

Firstly, I’ve been doing IT support and systems administration for a living for nigh on nine years now. I’d been a Linux tinkerer well before then (I got into mail systems first due to spam fighting; Sendmail and UNIX / Linux with Slackware back in 1997). I did a long stint doing Windows Server systems admin – which is actually fairly interesting tech sometimes, even if it can be clunky and buggy – and do Red Hat for a living again now.

But it can get old. Over the years IT has become something of a “regular” service, the support staff and admin like janitors and there’s no glamour in the job. Depending on the area and your businesses’ line of work there’s fewer interesting “toys” to tinker with. This describes my situation well.

In short I think I need a good project to sink my teeth into. DotProfile is doing things I’ve already done to death (XMPP and messaging  / DNS) and I’m not really a programmer per se. I’ve done OpenID, as you can see here. Ditto IPv6, also on show here. My infrastructure does DKIM, GeoIP, SPF, greylisting, XMPP. I’ve even had a disasterous run as a Tor exit node.

I’ve been asked if I’d do webhosting – no thanks; there are far too many kids (in the literal and figurative sense) overselling their little VPS to small business for a razor-thin profit. I don’t want to get in amongst that shit, I take pride in quality servers and reliability.

Even the package repository for Fedora feels thin. There’s only so much out there in the way of interesting software to package, and even new packages in Fedora proper are getting a little obscure. I suspect most of my users are here for Courier IMAP anyway 😉

I do memcached at work (and I’m building libmemcached in the background here, just because) but I personally have little use for it. Do I go back to my usenet roots and run INN? (Am I that much of a masochist? I remember cyclic overview rebuilds even now, they weren’t fun). I suck at art so design is out, and I just can’t consider being a bigger blogger / pundit or SEO hack without laughing at the thought. Cloud computing? Yes please – but how / who will bankroll? 🙂

I’m a systems architect at heart. /me needs a challenge, fast.

(Or I end up getting ITIL / PRINCE2 qualifications, sell out and become management ;-))

Posted on

The one about yoga, marketing and DNS.

I’ve been fairly flat out this week. No blog entries, even my Twitter updates are sparse in comparison to the norm for me.

Speaking of Twitter – dear Marketing people: I know you won’t listen because you don’t really care unless you can make a buck out of it, but we would like at least one popular use of the Internet that has a decent signal-to-noise ratio. It’s not there as a cheap way for you to flog products, it can be a useful and simple communications tool for the savvy masses.

I point blank refuse to buy anything spammed in such a manner and I’d rather eat my own testicles than shill for you in any way shape or form. I’d rather follow someone with something compelling to read – Stephen Fry is an excellent example.

Even while my dearest is out of town I’m continuing with the “hot yoga” classes. It’s getting a little easier although doing so in loose shirts is probably not a great idea. Late in the lesson when you’ve sweat about a litre they tend to hang off me like a heavy towel, which is not fun. A singlet would be more sensible.

I did say I’d move the “technical” blogging to DotProfile, but I’ll make a break here – my network is BIND-free; I’m the maintainer of the MaraDNS package in Fedora (having inherited it from the previous maintainer who has moved on to other things) but haven’t used it as much as I should have.

As there’s a newer version (and a buglet in the current package’s init script) I rolled a new one and tried it on my internal servers – it’s running very nicely now, a touch quicker than BIND and while the zone format is rather odd, it’s still simpler to configure than BIND. So now I’ve got one PowerDNS box (qbert) and one MaraDNS (pong). I’m starting to run out of nameservers to try (having run djbdns, nsd, unbound, dnsmasq and posadis in the past here, and MS-DNS at previous workplaces.)

Oh, expect a new maradns release for Fedora very soon. It’s already in Rawhide.