Happiness is Family Shaped (by sarah)

Spring Cleaning (by alaric)

I've spent more time building infrastructure than using it, I suspect. I love building infrastructure, so I've often built it because I can; however, with everything that's happened in the past six years, I've ended up struggling to maintain the infrastructure I already had. So I've had to change tack and become much more pragmatic about my infrastructure astronautics, such as getting rid of my limited company and migrating from a tightly-bound cluster to a single box for my hosting platform.

This has given me some time to tidy up and simplify the infrastructure I want to keep.

So this weekend, I got around to rebuilding the Kitten Technologies web site. This is where I publish my open-source creations; they were all version-controlled in Subversion, and I had a PHP site with some static pages, then a dynamically generated project browser that would pull out files called VERSION.txt and README.txt from the SVN repositories to build a description page, offered up tarballs of all released versions for downloading, and linked to an SVN web interface for browsing the repo. I wanted to get around to implementing ticket tracking at some point so folks could submit tickets.

However, for a while I've ached to migrate to Fossil for version control, mainly because it has integral ticket tracking and a wiki for each project, along with integral repository browsing; it provides a fully-featured project Web site, and it's a distributed VCS to boot, which is also useful. However, I wanted it to still all look like a nice integrated site for all my projects. So what I've done is to write a Fossil skin stylesheet that has my new look in it, then to build the wrapper site using the same CSS (eg, by using Fossil's names for div classes and overall page structure), based on Hyde; the CSS is actually generated from an scss master file that Hyde processes to generate the CSS as part of the static site, which the Fossil repos just refer to. My deployment script rolls the skin out to all of my repositories whenever it's updated, so they are all kept magically in sync. It still has a few rough edges (I want to improve the navigation with a consistent site-wide nav bar above the Fossil menu bar, that has the current project highlighted; this will be slightly more complex as I'll need to make the script modify the skin for each project to highlight the correct one) and I am still incapable of making non-ugly CSS, but it means that Kitten Technologies is now live on Fossil. I've a lot of projects still to migrate, but after I've done the "fiddly" ones that need some level of manual tweaking, I hope to produce a script to automate handling all the rest.

Secondly, I've been tidying up the home fileserver. It was down for some time for various reasons, which means that I've ended up with a new archive of photos, music, and PDFs forming on my laptop. I pulled our music collection out of the backups onto my laptop, too, which meant that I then had a diverging fork of that (as there was some new music on the file server since the last backup, which I later retrieved from the disks), so the re-unification of all those tens of gigabytes of files has been fiddly. But, it's now largely done, which is great; and there's now precisely one master copy of everything, and the home wiki is back up to date and pruned of outdated TODO items from several years ago.

However, this has increased my desire to implement Ugarit's archival mode. Rather than manually curating directory structures to organise my stuff (and the pain of merging changes to them), I'd love to just be able to pour files into a Ugarit library and tag them with metadata (maybe some time after the original import, if I'm in a hurry at the time), then create virtual filesystem views on that which reflect things like "All my music, organised by artist/album/title" or "All my photos, organised by who is in them, year, then event title". Combine that with the proposed Ugarit replication backend, and it will even manage replicas on my laptop as well as the home fileserver, all kept seamlessly in sync; having a home fileserver was easy when I worked from home on a desktop machine so I could just permanently mount the filesystem from the server, but it's a bit trickier with a modern laptop-based lifestyle.

Also, as the archive is already backed up into Ugarit, migrating it into a Ugarit "library" will be fast and efficient - Ugarit will automatically recognise that it already has the content of the files, and just need to upload the metadata!

I think with that and my workshop sorted out, I'm done with spring cleaning - my urgent tasks are now sorting out paperwork for my Cub pack, fixing an offline external disk on my fileserver, getting Ethernet to the workshop so I can do useful computer work in there (and move the home fileserver out of poking range of the baby, who loves to turn it off), resurrecting my salmonella install, hacking Ugarit, ring casting, and getting the foundry working so I can cast bronze, and wearable computer work! Not to mention endless minor DIY things in the house - we've got pictures to put up, dents in the plasterboard walls to fill, a flu to install for the fire, walls to repaint, ...

Server upgrade (by alaric)

I host a heap of web sites (including this blog), email domains, source control repositories, mailing lists, and various other things (such as one of the official Chicken Scheme egg mirrors, a Jabber server, and an IRC server with bots). I do this with a combination of dedicated server hardware which I hire space, power, and connectivity for in London for the primary stuff, and a virtual private server in California for backup services and rapid DNS lookups from the USA.

This is a costly hobby, but it gives us a platform upon which to do interesting things, and lets me help other people out with free hosting; as I need to put in the time and money to run the infrastructure anyway, the spare capacity on it is essentially free.

The most demanding part is server upgrades. Periodically, I buy a new physical server, install it with all the software it will need, put it alongside the current hardware in the data centre, and transfer the data and settings across and configure everything that needs configuring on the new server until it works just like the old, then switch them over. I do this when the current hardware is getting full or overloaded or unreliable or just plain out of date, as I don't trust in-place updates of the core system software - it's too easy to end up with NOTHING working.

However, this has been overdue for several years. I bought the new hardware (this time, with a contribution from my biggest user of disk space!) nearly two years ago, and installed it in the rack nearly a year ago, but only yesterday did I get the chance to spend a day sitting next to it in London coaxing it into readiness then doing the final switch over...

It didn't go entirely to plan, of course. I'd previously written a script that used rsync to copy all the user data over; the first time I ran it it copied everything, then subsequent runs only had to copy the differences. The idea was that I would have less down time while I copied the data from the old server to the new (which has to happen with both servers offline, so that nothing can change during the copying process) if there was only the final changes to copy. However, I realised that the accounts of my biggest user of disk space weren't covered by my script as they had been slightly hacked to accomodate their growth.

And the whole process of moving the software configuration was made more complex by the fact that I had previously been running two servers in a kind of symbiotic cluster, in order to meet the load with the hardware of the time. Nowadays 64-bit multi-core behemoths with gigabytes of RAM are cheaply available and well supported by NetBSD, so everything can be done on one box. This is a much simpler setup, but it means that I had to undo the complexity of the previous setup when transferring everything across!

I ran into a few other unexpected problems, too; I noticed that the clock on the new server was terribly wrong, despite it running NTP. I did a manual ntpdate, and then just in case, another to check that it was now only a few millisecond out - but it was already half a second out again! It quickly became apparent that the clock was ticking about one second in every two seconds of real time...

Looking in the output of sysctl -a, it became apparent that I had a choice of time counter sources: it was using the TSC, but I also had an HPET, a clock interrupt, an APIC clock, and the good old 8254; my machine was brimming with alternate clocks. I tried switching to the HPET with sysctl -w kern.timecounter.hardware=hpet0 and suddenly time was running as expected. I popped that in /etc/sysctl.conf so it would come back on reboots, resynched the clocks, and everything's been fine since. I can only presume that the kernel was reading the CPU clock speed wrong, or some kind of dynamic clock scaling is happening, so that the (CPU-based) TSC wasn't having its ticks converted to seconds properly.

I had a big setback with the email setup, as NetBSD comes with Postfix as part of the base system but I wanted a more recent version from packages, but I ended up getting tangled with what version was being run in various situations and what configuration file was being used, which took a while to sort out. And then of course there's Mailman, the mailing list server software, which is complicated by needing write access to its filesystem-based state when run from the mail system (for incoming mail) or the Web server (for the web interface), so uses lots of setgid binaries and group-writable files and the like, and so always takes a lot of fiddling to get working properly.

But... I did it. And so, having completed my tax returns earlier this year (which is what freed up the time to prepare for and do this mission), I have now gotten rid of all the major obligations that have been hanging over me for the past few years.

I still need to visit London again - I've left the old servers running alongside the new in case I missed any files that need to be transferred; I'll give people a chance to check I've not missed any of their stuff before remotely powering them down (to save electricity, which I pay for) and coming in to take them (to free up the space). But that's relatively easy!

A week off (by alaric)

I took last week off of work, to recuperate from the house moving and to spend some time setting up my workshop. That sounds like more house moving, but it's without deadlines, and it's taking time to sort out my own space the way I like it. So it's relaxing and settling! I did a few other things, too. I chose the Half Term week for my holiday, as no work AND no school runs would mean I didn't need to get up in the mornings, and so I could spend more time with our children. Sarah's teenaged (well, 12 and 14) cousins visited for half term, too, and were eager to help out with interesting DIY projects!

• We laid concrete in the fireplace, to bring the base level with the rest of the floor so we can tile it to make a nice hearth. We mixed the concrete by hand, and leveled and tamped it into the space, then checked back on it over the next few days until it was hard.

• We laid mortar along a messy and dirt-accumulating crevice in the workshop floor, to level it. This was fairly similar to the laying of concrete, except using sand rather than all-in ballast as the aggregrate.

• Painted most of the workshop floor with the special concrete floor paint I bought from the excellent and helpful Bailey Paints. We can't paint over the bit we laid mortar on until it's cured sufficiently, which will take a few months, so that bit can wait.

• Constructed and arranged the furniture in the workshop. Shelving had to be assembled, and my famous double-deck electronics desk put back together!

• Mounted my tool board on the workshop wall. It's happy in its new home.

• Mounted and wired in the Caffreys sign. Years and years ago Sarah and I, with my friend Matthew, witnessed a pub being redecorated, and the illuminated Caffreys sign was being chucked into a skip. We asked if we could take it, and it's followed us ever since then, being used as a rather unweildy novelty lamp; now we have a place we can mount it properly to the wall, so it's mounted on the outside of my workshop, with the cable run properly through the wall so it can be plugged in inside.

• Mounted and wired in the router, power strip, switch and UPS. We have a nice new cupboard under the stairs, built for us by Sarah's brother David. I've mounted the "core network" devices directly to the inside wall of the cupboard, and will mount one of my patch panels there when I run CAT5 to sockets around the house (and a trunk to the workshop, where my second patch panel will be installed in the comms cabinet - when it's bolted to the wall; we bought the bolts for that, but didn't get around to it).

• Gone climbing (with Jean and Sarah's cousins) at The Warehouse in Gloucester, which has excellent facilities for children and young folks. Everyone had a good time.

• Worked more on my scripts to migrate the massive amount of data from my current hosting setup, love.warhead.org.uk, to the new hardware. Love currently runs on a pair of servers, fear and infatuation, whose responsibilities will be taken over by one, just called love. This will be a simpler and more reliable setup, which will be easier to migrate in future, and will (touch wood) crash less. Oh, and it gives us much more disk space.

The love migration scripts were about the only stuff I did with computers all week. I'd have liked to have done more (I have some Ugarit, R7RS, and Chicken Scheme TODOs), but the presence of teenagers who would get bored if they didn't have exciting DIY tasks to do meant I focussed on things I could do with them. This isn't a problem, as in only one week I couldn't do ALL my projects; even focussing on DIY, we didn't get it all done 🙂

However, I think I need to take time off to relax like this more often. Mainly because, despite not needing to be up in the morning, I kept waking up at around 6am and not being able to get back to sleep. And once I woke up from a nightmare that I was neglecting all my responsibilities and everyone who depended upon me was being let down. These are not healthy signs...

Tax returns (by alaric)

Many years ago, around 2000, I formed a limited company. A bunch of us wanted to rent a rack in a data centre and host our servers there, and it seemed wise to have a separate legal entity to sign the contract with the ISP for the rack.

This cost me some money here and there in fees, and I had to take the time to keep books and file annual reports and accounts, but it was bearable.

Then a few years later, I became a freelance software engineer, and as the company already existed, it made sense to operate through that. Making a profit rather than being propped up by cash loans from me meant the company's tax returns became more complex, so I paid an accountant to do those, and that was fine as the saved money in using the company paid for his fees.

So all was fine until 2007, when a flood destroyed our home (and my placed of work).

I had to keep working to support my family, so I went to London to borrow space to work and sleep in, and did what I could to keep cash flowing. But this meant I wasn't spending much time at home where my paperwork was, and had little time to deal with book keeping, and my post was arriving into a building site back home.

A year later we were able to move back in, to try and pick up the pieces. But life was harder than it had been, and Sarah was sicker, and it's taken me until 2012 to catch up.

But last week I submitted my last personal tax returns. Towards the end of last year, my last corporation tax returns. Earlier that year, my last VAT returns. And the good news is, I seem to be eligible for some tax back. I'm happy about that as a friend leant me a thousand pounds in 2007, and I've still yet to pay him back; my tax repayment will cover that, plus some interest and a little left over... I'll pop it into the savings account and wait a few months in case HMRC change their mind, however.

But having all this done is a huge weight off my mind, one that had hung over me for about for years. Right now I seem to be in a phase of shedding burdens and finishing things; the only big thing I have left hanging over me now is migrating my servers, then I can start relaxing a bit and get on with my projects for the year!