Python Virtualenv wrapper functions

A while ago I noted that I should write about my shell functions for creating and managing Python Virtual Environments. Recently I was helping my boss set up some python virtual environments for some different projects, and I couldn't remember how to use virtualenv directly.

I really wanted to just share my wrappers with him, and I found that they weren't quite ready to share because I hadn't finished documenting them.

This post is to remedy that.

Read more…

ssh-pass

I've been using pass for a while now and I really like it. But I don't like putting passwords or passphrases into my system clipboard if I can help it: writing code to sniff the clipboard is child's play, so I'd like to avoid that attack vector if possible.

One place that I thought would be simple would be the SSH utility to add keys to your SSH Agent: ssh-add. It should be possible to do something like this:

$ pass github/sinewalker|head -1|ssh-add github/sinewalker

Unfortunately this isn't so: ssh-add doesn't accept your passphrase from STDIN when piped like this.

But, there is a way to do it.

Read more…

Converting latin-1 To utf-8 with Python

Tonight I finally converted all the Glossary pages in my mirror of the Jargon File into Unicode (utf-8 encoding) so that they will transmit and display properly from GitHub Pages (or any other modern web server). It was a fairly trivial thing to do in the end, but I am likely to need to repeat this for other things at work, so I'm blogging it.

The Jargon File was converted into XML-Dockbook and Unicode for version 4.4.0, but ESR only converted the front- and back-matter, not the Glossary entries (i.e. the actual lexicon). Those are still latin-1 (ISO-8859-1). And although the HTML rendition begins with the correct header declaring this:

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>

The pages are actually served from catb.org as Unicode (utf-8). For instance, compare /dev/null on catb.org with my mirror of /dev/null.

Read more…

Journald experiments - Testing systemd's logger: speed and buffering

I've been having good natured arguments at work about whether it's the End of the World that we are at last switching away from Scientific Linux 6 and it's System V style init scripts, to CentOS 7, which uses systemd.

My own opinion is that systemd is pretty cuspy. It's not perfect, but nor is it some great hulking monolithic monster come to destroy the Unix Way in the Linux world. It offers many worthwhile improvements and I've enjoyed using it in openSUSE for years now. I look forward to switching away from the hair-ball of wet SysV init scripts with clumsy precedent semantics and manual service recovery.

Now, I don't want to throw my hat into the ring on the pro's and con's of systemd having replaced the start-up infrastructure (and a lot of other systems besides) on Linux-based operating systems. Enough has been said already on that front, by many more experienced than I, and further argument is pointless: whichever camp you're in, you won't be convinced of the other sides point of view by now.

However there is one argument against systemd that I'm not so sure about: journald and it's past issues:

  • alleged buffering of logs, making diagnostics and debugging on time-critical services difficult or impossible
  • binary log files which can be corrupted, and then not useful thereafter (because they're binary)
  • volatile storage, so that your logs are gone when you want them the most: after an unplanned reboot

I'll be spending a few days experimenting with journald in these areas, to see if it's as bad now as it was five years ago when concerns like these were being raised.

In this post I want to look at the journald daemon / journalctl log viewer a bit, from the point of view of buffering output, whether and where it could be occurring, and what the implications might be as a web sysop.

This is a medium-long post, with about 23 minutes of terminal output recordings (in text, using asciinema) and is about 2⅓MB to download. It's also about half-an-hour's read on top of that.

Read more…

Password databases: setting up password-store on a Unix computer

Having covered what pass is, why I'm using it, and the required supporting tools gnupg, git, ssh and a private git remote, it's time to go over how to put the system together.

Setting it all up on a Unix computer is fairly straight-forward. Getting it onto an Android is a bit different. So in this post I'll cover how the pieces of the system fit together, and then walk through setting it up on Unix.

Synchronising your local password-store git repository with your remote store is done a bit differently depending if this is the first time you're setting up the remote, or if you already have a remote and you wish to merge it into your new local. I'll cover that too.

Read more…