I had a thought about what's on my wish-list for tech tasks that I'd like to complete this year, and then it struck me that there's a lot going on to update my blog with.

Read more…

I've been having good natured arguments at work about whether it's the End of the World that we are at last switching away from Scientific Linux 6 and it's System V style init scripts, to CentOS 7, which uses systemd.

My own opinion is that systemd is pretty cuspy. It's not perfect, but nor is it some great hulking monolithic monster come to destroy the Unix Way in the Linux world. It offers many worthwhile improvements and I've enjoyed using it in openSUSE for years now. I look forward to switching away from the hair-ball of wet SysV init scripts with clumsy precedent semantics and manual service recovery.

Now, I don't want to throw my hat into the ring on the pro's and con's of systemd having replaced the start-up infrastructure (and a lot of other systems besides) on Linux-based operating systems. Enough has been said already on that front, by many more experienced than I, and further argument is pointless: whichever camp you're in, you won't be convinced of the other sides point of view by now.

However there is one argument against systemd that I'm not so sure about: journald and it's past issues:

• alleged buffering of logs, making diagnostics and debugging on time-critical services difficult or impossible
• binary log files which can be corrupted, and then not useful thereafter (because they're binary)
• volatile storage, so that your logs are gone when you want them the most: after an unplanned reboot

I'll be spending a few days experimenting with journald in these areas, to see if it's as bad now as it was five years ago when concerns like these were being raised.

In this post I want to look at the journald daemon / journalctl log viewer a bit, from the point of view of buffering output, whether and where it could be occurring, and what the implications might be as a web sysop.

This is a medium-long post, with about 23 minutes of terminal output recordings (in text, using asciinema) and is about 2⅓MB to download. It's also about half-an-hour's read on top of that.

Read more…

Here is the fifth part of my blog series expanding on my 4-bit rules of computing.

In this post: rules 7, 8, and 9; which discuss testing and debugging. They are all related in a way: having to do with making good-quality craft work. Because, as much as computer people like to believe that we're "engineers" or that this is "computer science", we're not really. We're crafts people, in a profession that's still very young and finding its roots and methods in order to be consistently successful.

I'm definitely not trying to pretend I'm an "engineer". For real rigour, there is much more required than a few simple rules. But these are some realistic and humble rules in the area of testing that I aim to stick to.

Read more…