Getting a PowerDNS recursor up and going, fast!

This post is another one of my “quick and dirty” service tutorials

This time, I’ll cover getting a recursive DNS service up and going, using the PowerDNS recursor package. Traditionally Red Hat/Fedora users would opt for BIND (with or without the old “caching-nameserver” package of old) but I like to be a little different. Plus:

  • PowerDNS has an excellent security record (was not affected by the Kaminsky DNS vulnerability)
  • It’s small and does only the job it’s intended for in the traditional small-tool UNIX philosophy (Authoritative DNS is the job of it’s “bigger brother” PowerDNS package)
  • It’s fast and very easy to configure (compare to djbdns for example, which is neither)

Installing the software

For Fedora users, it’s in the Everything repository so you can just install the package as below. Red Hat Enterprise Linux  / CentOS et. al will need to  add the EPEL repository first

To install, simply

yum install pdns-recursor

.. which will install the package and it’s dependencies (just lua and boost if you’re on a fairly fresh install)


It only needs a single configuration file in /etc/pdns-recursor/recursor.conf., so open it in your preferred editor

As it uses key = value pairs, it’s very easy to follow, well commented and the defaults are quite sensible.

Firstly, for security, change the “allow-from” to match your local subnets – this determines which address blocks our server will permit and answer recursive queries for.


If  you have local authoritative zones (especially private internal DNS) you may want to set forward-zones to tell the recursor to query those servers for domains

#format is zonename=dns.server.ip

forward-zones =

If  you have a number of zones to forward queries for, you can use the forward-zones-file directive, which should point to a file containing the key-value pairs as above

By default, PowerDNS will listen on all interfaces but in practice will still prefer an explicit interface to listen on, so setting a local address via local-address is generally a good idea, especially if you’re multi-homed. It takes multiple addresses or even 🙂

# Listen on localhost and my NIC IP

local-address =,

For spotting common issues I like to have a little logging, but not much, so I set it to send common errors to syslog


For most uses, that’s all you need! Start the server via service pdns-recursor start and test it via dig/host

[mfleming@qbert ~]$ dig a @

; <<>> DiG 9.5.1-P3-RedHat-9.5.1-3.P3.fc10 <<>> a @
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6559
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;    IN    A

;; ANSWER SECTION: 2044    IN    A

;; Query time: 4 msec
;; WHEN: Sun Aug  9 14:19:19 2009
;; MSG SIZE  rcvd: 57

Oh, and before anyone asks: see the 3rd answer in the FAQ regarding presence/absence of Authority records in dig etc. output. It’s a feature, not a bug!

A little more advanced..

If you have IPv6 enabled networks and want to make best use of v6-enabled services, tell the recursor to look up AAAA records too (it’s not on by default, as it’s a little slower):


You can also send queries out over IPv6 using the query-local-address6 directive eg:


If you’re security conscious and don’t want any bogus records coming from g/TLDs that isn’t glue/delegations, use the delegation-only directive:



Keeping it simple isn’t stupid

GeoIP / IPv6

As an IPv6 enthusiast/proponent/fanboy I was really happy to see Maxmind finally put up a free-as-in-beer IPv6 GeoIP database[1]. Now to find some applications that will make good use of this data… 🙂

(Don’t worry, I’ll pull it into EPEL/Fedora GeoIP packages either way. It’s not a huge file)

Keeping it simple isn’t stupid, no sir!

I’ve worked with configurations in a variety of baroque formats, not limited to but including your common XML formats, Perl scripts (thank you cleanfeed/amavisd), python (ta maradns!), something that may be Erlang (ejabberd), lua (prosody) and have developed a fondess for the simple simpicity of a key = value pair config.

This is especially useful when you’re in a bind with a relatively unfamiliar piece of software, as I was this morning. The last thing you want to be faced with when you’re under the gun and need something working Right Now is some app developer’s bizarro idea of a sane config file, so keeping them simple and sensible is a huge plus – app developers take note, resist the urge to be too clever 😀

Say what you will about the old Windows .ini file, at least you know what you had to do with it

(The less said about the prank-gone-wrong that is registry hives the better and I’m glad UNIX vendors never took that particular drug :-))

PS. The application was qpidd from the AMQP stack, for reference and both it’s manuals – and Red Hat’s MRG Guides – helped immensely. Microbrews all around!


How polite!

I’m testing the Prosody XMPP server package I’d mentioned some time back (I’ve worked with jabberd/jabberd2 and ejabberd, figured a new tinker toy was on the cards) and got this when trying to check the service status – as a non-privileged serf/peon

mfleming@pong ~]$ /etc/init.d/prosody status
Prosody is not running

You will also see this if prosodyctl is not running under
the same user account as Prosody. Try running as root (e.g.
with ‘sudo’ in front) to gain access to Prosody’s real status.
[mfleming@pong ~]$ su –
[root@pong ~]# /etc/init.d/prosody status
Prosody is running with PID 17701

If only more software packages were that polite and helpful 🙂 (looks in the direction of some larger software concerns that shall remain nameless)

Just Plain Bad Service

I almost lost two of my domains this week – a registrar / hosting company that will not be named decided to close them off without any notice – no email, letter or even a phone call.

15 minutes in a phone queue as the “next to be answered” and a debate with the tech on the other end finally got me the registry keys for both – I certainly won’t be renewing with them or asking for the accounts to be reopened, I’m already transferring them to my GoDaddy account.

Given their focus on “customer service” (even their technical support staff are “e-business consultants”) this is a MAJOR blot and I won’t tolerate it – no more business from me! It’s bad enough that they’re double the price of most offshore competitors but to close down accounts without notice is just plain intolerable.

The irony of the situation is that I worked at that hosting company for years and on realising this the tech at the other end started to agree with my assessment of the situation 🙂 Additionally I still know their systems better than some of the folks there now! (I talked him through the process of getting the keys)

I was quite pleased to see a WordPress 2.8.1 upgrade hit EPEL quite quickly – new features and a security rollup all in one and before I felt the desire to roll one up myself. Cheers Adrian! This release was rock solid and the upgrade the most painless on record.

My own packaging is slowing up a little due to so little out there that is a) not already packaged or b) not really ready. One I have given a test run is the Miredo client for IPv6/Teredo tunnelling. IPv6 tunnelling is widely available – many places will even have native connectivity – but for those cases where you are behind a restrictive NAT (or just want to tinker like I do) it’s an interesting alternative. Fedora packages from my usual repositories.

Revisting Nagios for monitoring – I can’t just walk over and look at a server several thousand miles away –  brought back a few memories, it took a bit more dusting off of the old knowledge  (I use Zabbix at work, they’re dragging me into using Xymon which I don’t like as much) but it’s working quite well and the EPEL/Fedora packages are kept well up to date. With puppet / func it’s even simpler to roll out the NRPE checks (which were always tedious when done by hand)

The SmokePing package has also been a boon in the workplace – I used it to successfully convince our network administrator that the packet loss we’d identified wasn’t our imagination and lo, it got fixed as we could track performance. Again it wasn’t hard to install and the configuration makes sense after a pint or two 🙂

My girlfriend and I have been getting back into yoga – “hot” yoga, which is the standard fare, just in a 35+ degree heat – and the timing is perfect; It’s been freezing here in Brisbane (10-20C on average, which is brisk for this part of the world) and I needed something to keep my mind in check. To my surprise I even held my form despite being away from it for a month or more 🙂

New hosting arrangements

If you’re reading this (and not a “Page Cannot Be Displayed” or “Internal Server Error”) then I’ve successfully moved my site(s) to the new server.

After years of hosting off my own gear, either at datacentres or literally in-house I’ve moved to a Xen VPS at Slicehost, running CentOS 5. Setup and migration was relatively painless (fat-fingered a DNS record however, I won’t be doing that again..) and their default bare-bones install is an absolute boon for keeping an instance clean – I hate cluttered installs full of cruft I don’t need/want.

I’ve even got low-ping IPv6 to most of the world now via Hurricane Electric (just as I remember it – simple and do it yourself, which also meant pinging the service to keep the tunnel up :-D)

The only nit I’ve found has been the lack of SELinux support on their guests – I’ve had it in Enforcing mode on my servers for as long as I can remember it being offered – it just feels weird and wrong not having it now 🙂

I’ve also found that there’s a few Fedora packages not available in EPEL that I just can’t live without (postgrey and linux_logo!). I’m not sure why they’ve not been picked up but I’m tempted to do so myself if there’s been no takers – give EPEL some love, it’s nice and stable and won’t do the dirty on you 🙂

Things I thought I’d never see

If I hadn’t been present for these, I’d probably call myself a bullspit artist. But I swear on a stack of $documents I was there and my experiences are 100% true.

Firstly, I thought I’d never ever see a web framework’s object relationship model used as a database stress tester (cheers Django!). I don’t think it was intended as such but some of the queries it’s generating are just hideous and driving load averages to insane levels.

Secondly – I have two USB keys in front of me. Nothing really unusual about that, except they have “Windows + PHP: Platform of Choice” on them. Yes, I raised an eyebrow too, and I’ve actually done Windows on PHP before at an old workplace.

Choice, PHP and Windows Server from experience boils down to this, in my experience: a) Run as FastCGI or b) Endure a world of suffering (the ISAPI is awful and suphp doesn’t exist).

I’ll of course advocate c) Run it on a Linux box as a DSO / suphp / under mod_fcgid process as your needs dictate – at least they’ll all actually work as you’d expect. 🙂

I got the keys from a presentation / open session run by a good friend and former work colleague Jorke Odolphi, now working for Microsoft (but still genuinely interested in F/OSS, I’ve trained him fairly well) and “Professional Geek” Nick Hodge (also a pleasant and well versed chap).

Microsoft having an open session on OSS was not something my curiousity would let me miss, so off I went. I applaud Nick and Jorke for having a go – the turnout wasn’t huge but the session was very lively and definitely worthwhile.

Writing up an auto-installer for FOSS web apps under Win/IIS was a nice touch, and now you know how difficult it is to implement a depsolver 🙂 IronPython / IronRuby are interesting ideas (the Parrot project / Perl6 peeps are doing almost the same thing and Java has been doing it for ages) but your usage case might be a bit of a corner one. We are all aware in the OSS world how charmingly limited the PHP database drivers can be (hence PECL alternatives and native drivers) so you’re part of a large-ish crowd there 🙂

But chaps you’re going to be pushing harder than Sisyphus to get some real FOSS cred for your bosses, if they genuinely want them.

I’m not going to delve deeply the licensing side of things (for good reason, there’s a post of it’s own) but a BSD-like license isn’t likely to get you the community you might want – it’s not really Free and there’s nary a nod to those making code / docs or other contributions (contributor doesn’t mean distributor by the way, if anyone from MS is reading)

The existing culture and mindset and that of Microsoft, it’s partners and some users is fatal to any “open source” initiative in my view (this doesn’t mean I’m unhappy to see an attempt, it shows the strength and relevance of communities like Fedora’s – or any other distribution’s for that matter)

They’re a cathedral; the faithful are handed tools and protocols blessed by the Powers That Be, with such tools/protocols invariably created because they help maintain the status quo (commercially advantageous to the “church”, closely coupled together to ensure/encourage adherence / lock in etc.) – many don’t know of or even see anything outside of the cathedral! Compare to the marketplace that’s Free Software – don’t like a tool/protocol? There’s other stalls with alternatives. Pick up a few and you’re building your own in no time and trying out ideas they may not even had considered before.

That’s the worrying part for the “high priests” – if the “faithful” are choosing their own tools for their own purposes (not just building using blessed tools for those protocols deemed desirable or “holy”) what’s to stop them straying from the rest of the flock? If the market allows you to build your own stall and offer your own wares, why go to the cathedral and be told how what to use?

I’m sure Microsoft’s upper management are happy (even if their outward demeanor may not show it) to see the projects around the Codeplex – still centred almost solely around their own technologies – .NET / IIS / Sharepoint / Windows Live Auth. Nothing really innovative and widely interoperable (and decoupled from other MS tech) like XMPP or memcached / OpenID / Laconica..

The real test will be projects that tie to *genuinely* open tech, a completely non-MS solution. I suspect the “high priests” may quietly sideline such “heresy”, even if the frontline preachers do not. 😛

Lastly: To answer a question posed to me by James Morris via I asked about the IE8 and $10K giveaway: Technically it was mind bogglingly simple: IE8 has a little “feature” where it will display these “web slices” on certain pages like a favourite / bookmark. Find the “splice” with the winning content (on an MS partner site of course) and the 10K is yours.

The splice itself is just a boring standard DIV element with a “hslice” class.
But the hack worked for Microsoft Australia marketing, so I suppose it serves it’s purposes.

Of course there were Chrome and Firefox extensions to support this behaviour almost immediately 🙂

The post is bought to you by lekhonee v0.4

More of Michael’s Not So Quick Tips


I migrated my filesystems to ext4 (I’ve come from an all0ext3 F10 to F11 upgrade, FWIW) and just like to mention that if you’re migrating your root filesystem you might want to regenerate your initrd via mkinitrd; the stock one I had didn’t seem to like ext4 (complained about unsupported filesystem options at boot time and failed dismally) but a quick initrd rebuild in rescue mode had me up and running in minutes.

I didn’t see this mentioned anywhere – perhaps I’m a corner case – but I thought I’d record it for posterity in case someone else finds it useful – or desperately needs it!


If you’re looking for a simple, cron-capable no-fuss delta/differential backup solution (looks over in Peter Gordon’s direction) I’d consider the “rdiff-backup” package, which has been in Fedora for quite some time, does rsync-like backups of reversed diffs – the actual data transfer is small, you can do “point in the past” backups with little fuss plus if you want just the last copy, you can just grab it from the backup tree as-is, without a need to invoke rdiff-backup’s restore (rdiff-backup -r) option. All you need is SSH set up between two hosts, ideally pubkey auth or GSSAPI (ie passwordless or pre-authenticated)

To back up a home dir (~fred) to a remote server, barney:

rdiff-backup /home/fred barney::/home/fred.backup

To restore tmp/wilma from last week (7 days)

rdiff-backup -r 7D barney::/home/fred.backup/tmp/wilma /home/fred/tmp/wilma

How simple is that?

Hacks for database admins

This is an ugly idea, but it’s functional and insanely simple:

Much of my day-to-day systems admin work is with web developers and frameworks. Some frameworks in my experience, whilst being insanely great and powerful for a developer have an unfortunate tendency to hide things under the hood – one example is where the framework defines the database schema (via ORM et. al) but keeps it fairly opaque to the developer – leaving it to the DBA / system admin to work out what may have changed.

This can be a PITA for both developer and systems mangler alike – the developer isn’t always sure if it’s introducing a regression and the DBA/sysadmin wondering about the performance difference.

Both PostgreSQL (pg_dump -s  or –schema) and MySQL (mysqldump -d or –no-data) allow you to save schema-only dumps of databases (I’m not sure about Firebird or others, I’ve not tried them out lately)

I take this at regular intervals and check it in to version control – I can then see changes via standard VC diff commands. It’s a simple hack but it’s functional and requires no extra tools.

This came about because I need to maintain a script that purges old data from a pgsql database, including foreign keys (manually as the schema doesn’t grok DELETE CASCADE) and a change in the FK relations means that my script broke – if I can follow the schema changes it becomes trivial to add in the FK changes needed. 🙂

I’m IPv6 ready – are you?

Firstly for those wondering about Courier-IMAP / authlib / maildrop+authlib packages for Leonidas:  I’ve built them successfully – only a minor adjustment needed after all that –  and it’s available in the usual place. Enjoy, and let me know if there’s any bugs / issues.

(For a change I managed to get them out the door before someone emailed me asking where they were. Miracles happen! :-P)

I was quite surprised – and pleased – to check my website stats and find that my most frequent visitor is an IPv6 address:

IPv6 in AWStats for ThatFlemingGent
IPv6 in AWStats for ThatFlemingGent

(If only the GeoIP database had an idea about IPv6 netblock ownership…)

A good friend of mine is a network administrator for a fair size network – two AS’ under his control and a network covering the Australian eastern seaboard. He’s often tasked with finding additional IPv4 address space

Because IPv4 addressing is becoming scarce the registrars in many locales (APNIC in his and my case) set a high bar for new allocations to network service providers (must use 80% of existing allocation, justify new allocations for a max of a /22 last I heard) – and rightfully so. They’re not toffees and they are indeed becoming quite scarce, moreso with increasing takeup of internet-enabled mobile devices and broader broadband availability.

Yes, there are other options such as NAT (Network Address Translation) and name-based virtual hosting to mitigate many issues – but not all applications play nice behind NAT, Voice apps and some games being good examples – and port forwarding isn’t simple for the novice user.

IPv6, step up to the plate! Support in Linux has been around for aeons and it’s rock solid. If you’re already IPv6-enabled, you’re likely talking to me over it now[1]. It’s even on by default with “link-local” fe80:: class addressing ubiquitous on new installs (even if there’s a lot of frankly ordinary advice on turning it off!)

For Fedora, there’s a number of options for public IPv6 – the documents for the “initscripts” package show the basics of IPv6 quasi-native tunnelling and “6to4” tunnelling and are a good starting point

The latter is easier and a good option if you don’t have a nearby tunnel broker / point of presence like SiXXS, Hurricane Electric or a provider offering a Hexago-like service.

(Australia is a good example – the AARNet educational network offers such a service, as does Internode for it’s customers; Telstra may still do so but that’s it, with Hurricane Electric a higher-latency option down here. Other points of presence are just too distant to be useful)

Wolfgang Rupprecht has a Fedora-specific howto, which applies just as well for F11 or even RHEL/CentOS.

The aiccu package is in the Everything repository if you’re eyeing off a SiXXS tunnel connection.

The “go6” client from Hexago is another that hasn’t been packaged yet (to my knowledge and while I use it due to my provider’s use of their broker software I’m not really a fan)

HE.NET (Hurricane Electric) lets you use the standard tools, no extra apps needed (bless ’em!)

The simplest method? 6to4. It’s not as fast as full tunnelling or “native” direct IPv6, but it will get you “on the road” so to speak. Unfortunately NetworkManager currently gets in the way,  going from my testing, but on a headless gateway not using NM it works a charm:

  1. Make sure IPv6 is on in your network config: (NETWORKING_IPV6=”yes” in /etc/sysconfig/network)
  2. Tell the network the default IPv6 interface to use (set “IPV6_DEFAULTDEV=tun6to4” in the above file)
  3. Add the following lines to your network interface:
    • IPV6INIT=yes
    • IPV6TO4INIT=yes
  4. That’s about it – restart the network service and you should be rollin’.

It will use anycast to (default anycast prefix host for 6to4) to find the nearest 6to4 broker and use it as the endpoint. Test by going to a site like (if you see an animated turtle, it’s working) and enjoy.

I’m moving servers next week (a Xen VPS with a fatter pipe) and rest assured it will be IPv6-aware!


[mfleming@qbert ~]$ host -t AAAA has IPv6 address 2001:44b8:62:1b0::1

Some updates, and a little “open sermon”

  • I’m in the unusual position of being inside, yet rugged up with a jacket and scarf with what feels like a cold. I can’t say I’m happy about it.
  • Upgraded WordPress here to 2.7.1 (Fedora 11 RPM rebuilt for F10) without too many hassles. It kept retrying the database upgrade(!) but disabling the plugins and a little bit of tinkering (clean cookies, rebuild config, even restarted memcached / set SELinux to Permissive) got it working. I suspect clearing memcached was the trick.
  • Sorry to my Planet readers for some cruft in the RSS, the Related Posts plugin needed to be *ahem* disciplined. It should be fixed now.
  • Loving the Leonidas release – the adventure during upgrade (died partway, restarted, left behind most of F10 for some reason) seems to have cleared out a lot of stuff I didn’t need or use and the rest is a definite improvement. Well done to all concerned!
  • Disappointed that our local news sources are running the front pages with soft non-news pap, when there’s rioting and apparent vote fraud in Iran. Their people are suffering and getting shafted, and you’re running crap about actresses doing nude scenes to get ahead…

To our dear Fedora users:  Please don’t attribute  malicious intent where it’s not warranted. I’ve had one comment here and just responded to a thread on fedora-list from users making some frankly melodramatic claims around how / why decisions are made and features disabled/changed/not kept up to date.

A large chunk of us are not on the Red Hat payroll,  we’re volunteers. Why do we do this? Because we enjoy what we do and are passionate about it. These users should remember that we use it too (“eating our own dogfood”) and want to deliver a top quality distribution.

However you can’t please all of the people all of the time – but just because a feature / change doesn’t suit you, doesn’t mean that the developer / packager  is out to get you

The distinct advantage of an open community is just that: it’s an open community.

If you don’t like a feature, suggest/contribute changes and/or send a patch. If the documentation is lacking, why not write up a how-to and publish it, help update the wiki or the distro documentation? Likewise if the art isn’t to your taste,  I’m sure the Art team welcome volunteers. If you’re fairly knowledgeable, share it with other users on the lists / IRC / forums.

A “This is broken, you guys suck and out to get us” attitude is not helpful, please let such attitudes die off.


A user, packager, infrastructure hacker and occasional developer (since Red Hat 5.1)

That Fleming Gent Meets Leonidas

My long-running repository at ThatFlemingGent (or “Enlartenment” for those who haven’t caught up) is live and ready for use.

As releases have progressed the list I offer has become smaller, as many have been pulled into Fedora proper (either by me or other Fedora contributors), died upstream or just dropped due to lack of my/visitor interest

There’s only 58 packages this time – long gone are the old (pre-)Extras days when I had 140+ 🙂

Highlights: well, there’s um… the GNOME Internode Applet![1]. yet another webserver in Hiawatha! (Think lighttpd with a security focus) – the AIM and MSN Python Jabber/XMPP Transports… A bunch of WordPress plugins (oh how I’m jonesing for a 2.7/2.8 package for F10!) and other small but useful tools, especially for random hackers and systems people.

There’s a couple of non-starters that I’m still working on – namely the Courier suite (authlib won’t build at the moment, I’m trying to work out why) and the MySQL-memcache UDF functions (memcached_functions_mysql in F9/F10). They’ll be added as soon as they build correctly.

Enjoy, and feel free to drop me a line if you have a suggestion / problem 🙂

[1] Internode is my ISP, one of the highest regarded in Australia and for good reason, they’re stable -and it’s owner/CEO Simon isn’t averse to being pranked either: