Towards the Family Mainframe (by alaric)

Last September, I posted progress on the construction of our domestic mainframe. To recap, the intent is to build a dedicated home server that's as awesome as possible - meaning it's reliable, safe, and easy to maintain. That rules out "desktop tower PC in a cupboard" (accumulates dust bunnies, gets too hot, easily stolen, prone to children poking it); "put a 19" rack somewhere in your house" is better, but consumes a lot of floor footprint and doesn't fix the dust bunny problem. So I've made my own custom steel chassis; fed cold air at pressure via a filter, incorporating a dedicated battery backup system, locked and anchored to the wall, and with lots of room inside for expansion and maintenance.

Since that blog post, I've finished the metalwork, painted it with automotive paint using a spray gun (which was a massive job in itself!), fixed it to the wall, and fitted nearly all of the electronics into it.

A significant delay was caused by the motherboard not working. I sent it back to the shop, and they said it was fine; so I sent the CPU back, and they said THAT was fine; so I sent both back together and it turned out that the two of them weren't compatible in some way that was solved by the motherboard manufacturer re-flashing my BIOS. That's now up and running; I was able to use the HDMI and USB ports on the outside of the chassis to connect up and install NetBSD from a USB stick, then connected it to the network and installed Xen so I can run all my services in virtual machines. It's now running fine and everything else can be done via SSH, but the HDMI and USB ports are there so I can do console administration in future without having to open the case (unless I need to press the reset button, which is inside).

The one thing it's lacking is the management microprocessor. I've prototype this thing on a breadboard and written the software, but need to finish off the PCB and cabling: but it will have an AVR controlling three 10mm RGB LEDs on the front panel, and three temperature/humidity sensors in the inlet and outlet air (and one spare for more advanced air management in future). But the idea is that the three LEDs on the front panel will display useful system status, and the environment sensor data will be logged.

Here's what it looks like from the outside; note the air inlet hose at the top left:

The socket panel on the left hand side worked out pretty well - 240v inlet at the bottom, then on the aluminium panel, three Ethernets, HDMI, and USB (my console cable is still plugged into the HDMI and USB in the photo, which won't usually be the case):

And here's the inside, with lots of space for more disks or other extra hardware; the big black box at the bottom is the battery backup system:

Now I have Xen installed, I'm working on a means of building VMs from scripts, so any VM's disk image can be rebuilt on demand. This will make it easy for me to upgrade; any data that needs keeping will be mounted from a separate disk partition, so the boot disk images of the VMs themselves are "disposable" and entirely created by the script (the one slightly tricky thing being the password file in /etc/). This will make upgrades safe and easy - I can tinker with a build script for a new version of a VM, testing it out and destroying the VMs when I'm done, and then when it's good, remount the live data partition onto it and then point the relevant IP address at it. If the upgrade goes bad, I can roll it back by resurrecting the old VM, which I'll only delete when I'm happy with its replacement. This is the kind of thing NixOS does; but that's for Linux rather than NetBSD, so I'm rolling my own that's a little more basic (in that it builds entire VM filesystems from a script, rather than individual packages, with all the complexities of coupling them together nicely).

I'm using NetBSD's excellent logical volume manager to make it easy to manage those partitions across the four disks. There are two volume groups, each containing two physical disks, so I can arrange for important data to be mirrored across different physical disks (not in the RAID sense, which the LVM can do for me, but in the sense of having a live nightly snapshot of things on separate disks, ready to be hot-swapped in if required). I still have SATA ports and physical bays free for more disks, and the LVM will allow me to add them to the volume groups as required, so I can expand the disk space without major downtime.

So for now it's just a matter of making VMs and migrating existing services onto them, then I can take down the noisy, struggling, cranky old servers in the lounge! This project has been a lot of work - but when I ssh into it from inside the house (over the cabling I put in between the house and the workshop) and see all that disk space free in the LVM and all the RAM waiting to be assigned to domU VMs that I can migrate my current services to, it's all worth it!

Learning things (by alaric)

I love learning new things. I'm usually struggling to find new things to learn; the last fun bit of computer science I learnt about was Bloom filters, and they didn't hold my attention for long. The last really fun bit of computer science I learnt about would be content-addressed storage, and I'm still having fun with that, but I can't find any more to learn about it. I'm having to make stuff up myself, which is rewarding in its own way, but much harder work.

Of course, this past week I've been learning TIG welding, which has been awesome. It's been a while since a whole new field of things to learn has opened up to me, and it's nice to work on a new class of physical skills. My routine physical learning is my weekly Krav Maga training, but I crave variety. My lust for learning benefits more from intensive two-week courses than an hour a week for years. I'd love to go and take a proper welding course at a college, but I can't spare the time; I have to practice when I can in the evenings and weekends. I'm getting good at horizontal/flat welds, but I'd like to master vertical and overhead (because if I work on anything large, such as the festival trolley, in my cramped workshop, I often can't rotate everything around to be nice and flat). Also, suspecting that the trouble I was hitting with stick welding is at least partly to do with the limitations of my old cheap AC welder, I want to use my new welder's capability to do nicely regulated DC stick welding and see if I can learn to do good stick welds. And I'd like to get some practice in welding aluminium and stainless steel, as I have applications for both of those.

So that's physical skills sorted, for now. Mentally, I've been learning how antennas work. There's no particular reason for this; it's something that's always puzzled me somewhat, but what's triggered the recent interest was a birthday gift from an old friend, of the ARRL handbook, which goes into some detail; and then meeting an interesting guy at a Cheltenham Hackspace Open Evening who turns out to design broadcast transmission antennas for a living, who answered a load of questions left open by the books. I still want to get a better intuitive grasp of the quantities involved - how many volts and amps appear on an antenna feeder line? What field strengths in volts per meter would you expect to find at what distance from an antenna? Does that relate to a corresponding magnetic field strength in tesla?

I've also been learning Morse code. This is quite interesting. I thought I'd memorise that table of dots and dashes and that'd be that, but it turns out that this technique tends to maroon people at slow morse speeds, as they mentally record a sequence of dots and dashes then mentally look it up in the table to find the letter. To get fast Morse skills, which tend to tend towards one or two characters per second, you need to learn to recognise the sound - "di di dah" - as a letter directly. So I've been using a combination of the Farnsworth and Koch methods to learn Morse; growing my "vocabulary" a letter at a time, and using an enlarged inter-character spacing. The latter is because I was tending to find that I'd hear a character, and write it down, but while I was writing it another would have been and gone, which was confusing. I want to reduce that inter-character gap, but I might wait until I've learnt the entire alphabet via the Koch method, so I can mentally be writing entire words rather than concentrating on a letter at a time - with a reduced alphabet, Morse training tends to involve writing down random gibberish (so far, I know M, K, R, U, S and O; at least the old Nokia SMS beep now makes sense to me... di-di-dit dah-dah di-di-dit!). Again, I have no particular reason to learn Morse - I learnt it as a child, but then forgot it through not using it, which had always faintly irritated me. I've often wondered about using it as a crude interface to tiny embedded computers, although it'd be frustratingly slow for most uses. The usual reason to learn Morse is to do CW amateur radio; that's an idea I've toyed with in the past, but being able to talk to random people over radio holds little appeal (I can talk to random people over the Internet much more cheaply and easily). However, I'd be interested in getting an amateur radio licence as a mental challenge, or as a means to some other project that requires radio communications capabilities, so I might go for a course if one comes up at a good time. I'd like to be able to operate a radio transmitter in an emergency situation, too.

I love to learn things, but I feel sad about not using the skills I pick up. Ok, I don't want to use my Krav skills - they tend to involve hurting people, and are only useful when already under danger of harm coming to me or people I'm protecting, which is nothing to be happy about. But I practice welding because want to make metal things. But by no means do I only learn things when I have a need for them; I learn stuff because it looks interesting and the opportunity arises, then I try and find applications. I've already been excitedly thinking about how aluminium welding will simplify the construction of one of my old back-burner projects - making a hiking staff out of aluminium tubing, that has a stack of lithium-ion batteries inside it at the base, a computer with a keyer in the middle so I can interact with it, and a high-brightness LED lantern on the top so I can have a variety of illumination options (white all around, white forwards only, red all around, etc). I had been working out complex systems of brackets and bolts to hold it all together, but TIG welding it would be much easier, neater, lighter, and stronger. Now, I had considered having a button that could be used to strobe the lights for Morse emergency signalling - and the logical next step was to include a co-axial semiconductor laser in the top that could shine a bright beam for signalling in Morse (and, at a pinch, be used to light fires; you can get 2W laser modules off the shelf these days, and all those lithium ion batteries are going to be able to source a lot of power...). So perhaps I should get a ham licence, and make the staff in sections joined together with insulators, and make it be a two-meter band dipole antenna (which is one meter long) with a CW transceiver inside, so it can also send and receive Morse by radio? That might be fun, and not much extra hardware as it'll already have a decent ARM microprocessor inside.

For now, though, I'd better focus on finishing off my server chassis (which I'm building my welding skills up towards), and make a new welding bench (mine is curved, and wobbles because the floor is uneven and the legs are too weak), and do some metalwork that's needed around the house... I'd like to do some more focussed Lojban study, too; right now I'm just picking up vocabulary by looking for words for things I don't know yet when I need them, but re-reading the reference grammar to remind myself of bits I rarely use would be good!

DNS issues today (by alaric)

Gah! This morning, my alerting system texted me to say that love (the primary server) can't talk to ihibehu (the backup server in the USA). A quick looked confirmed that we seemed to have some kind of routing loop in level3's network, which was therefore returning "TTL exceeded" to pings. I could connect to ihibehu OK from another network, confirming that it was just a local routing spat of some kind. I shrugged and moved on with life.

However, people started complaining they couldn't resolve DNS for stuff I host, so I had another look. love and ihibehu are both DNS servers (they go by the name of ns0.warhead.org.uk and ns1.warhead.org.uk in that role), and if one is unreachable, then the other should be contacted, so all should have been fine. However, it turned out that the IP address for ns0.warhead.org.uk was still pointing to its old location (and love don't live there anymore), so ns0.warhead.org.uk wasn't "working"; and so for the people whose route to ihibehu went via the routing problem, ns1.warhead.org.uk wasn't working as well.

Oops! One tricky aspect of distributed fault-tolerant systems is that sometimes part of them fails and you don't realise because all the user-visible stuff silently fails over. Therefore, you need to test things below the failover layer to make sure they work individually. Although I check both DNS servers are up, I wasn't checking that the "glue records" mapping the nameserver names to IPs pointed to the right place...

But I clearly remembered sending in the request to the registrar to change the glue record for ns0.warhead.org.uk when I moved it, didn't I? I checked my emails and, yes, I'd send that request, but with all the other stuff I was dealing with in the migration, I never chased it up. And lo, nestled among my spam emails was a response from the registrar, reminding me that I still had access to the interface to do it myself (The registrar used to be me, but I passed that mantle on to somebody else), and suggesting I do so. So it had never gotten done.

"No time like the present, then," I thought, and set out to send in the request, only to find that I don't still have access to the interface, because it also needs a password which I removed from my password databases when I passed control of the registry interface over. Doh!

So I've re-requested that the registrar does it for me. Thankfully, the routing loop has healed up and all is working again while I wait for that to happen. And I'm going to write a test for my glue records being correct into my monitoring system, because that was just sloppy!

Ugarit 2.0 released (by alaric)

Unless I messed up the release process, Ugarit version 2.0 is now available from the Chicken Scheme egg repository.

What does this mean to you, dear reader? Not a huge amount; you can go and read the release notes at the bottom of the Ugarit documentation page for the fine detail. But, to summarise:

• The beginnings of archival mode! As well as storing chains of snapshots of a filesystem, as Ugarit has always done (generally to be used as a versioned backup system), Ugarit vaults can now also store "archives", which are groups of files or directory trees identified by arbitary metadata, such as "This is the song 'Ooofarno' by 'Bobby and the Beaters', which is track 11 out of 12 from the 'Fishes In The Sea' album", or "This is a photo of Aunt Mavis taken at 13:58 on the 3rd of August, 2020, at Uncle Bob's 100th birthday party", or "This is a PDF of a paper by Donald Knuth on ternary numbers, called 'Simplified Arithmetic in Base Three'". You can then find things by searching on this metadata, which is much, much, nicer than creating trees of directories to organise all your stuff into. The user interface for getting things in and out of archives is still a bit minimal - but I have plans to fix that.

• The way tags are stored has changed. Ugarit 2.0 will read vaults created by prior versions happily, but when it writes to a vault, it'll write new-format tags (which have type information as well as a pointer to something), which old versions of Ugarit won't be very pleased to find.

• We now store a "metadata block" in every vault, pointed to by a hidden tag (we didn't used to be able to hide tags, so old versions of Ugarit will show you a funny tag, and complain if you try and do anything with it, as it's a new-format tag). This stores a vault format version number, so we can better handle incompatible changes to the vault format going forward; and as it's hashed and encrypted like any other block, it means you'll get an instant error if you try and connect to a vault using the wrong hashing and encryption settings, rather than bizarre errors further down the line.

• We've made it possible to store large logs in the vault. When we do a snapshot or an archive import or something, we keep a log of warnings and recoverable errors that cropped up while doing so. This is stored as a file attached directly to the snapshot or import object, so it can be arbitrarily large.

• Added log.sexpr and properties.sexpr files in the explore-mode interface, inside every snapshot or archive import object, which let you access the log and the metadata. These are files you can extract, or you can look at them with the new cat command.

• Added a cat command in the explore-mode user interface to dump a file to the screen for viewing.

• Added a client-side cache of snapshots and imports, which significantly speeds up the exploration of backup histories and archive metadata. Optionally, you can make the cache persist between sessions, otherwise it's made afresh for every explore session.

• As well as the existing ability to fork a tag into two tags that share the same history (applicable both to snapshot tags and archive tags), added the ability to merge two tags into one, melding the two histories into one. This includes some exciting logic to combine those histories for display in explore mode!

• Added a new sqlite backend, which provides a storage inside a single file, managed by sqlite. I wrote it to make testing easier, but it's a useful storage backend in its own right!

• Tidied up the Ugarit internals, splitting the core up into a load of separate modules. This makes development easier for me, and means nothing to users.

So what's next? I want to improve the usability of archive mode - right now, it's quite easy to import a bunch of files, but you have to hand-edit a text file to provide metadata beyond what it can automatically extract (currently just basic file information, plus whatever it can extract from ID3 tags and Ogg metadata); and then you can explore the history of the archive (as a series of imports) through the explore interface, or use a command-line tool to search for files, and then extract them or stream them to standard output.

What I want is:

• A shiny (web-based?) UI for searching the archive, seeing thumbnails of images, and the ability to download files with a single click or to perform bulk editing operations on metadata with ease and panache.

• A music player, that lets me pick music from an archive to queue, or to be given an arbitrary search criteria to find music to random-play, playing direct from the archive.

• A way to pick photos and assemble them into galleries, which are then publicly visible through a Web interface. Sarah wants to be able to put sequences of photos together, as well as individual photos that don't form parts of sequences, into multiple albums, for her blog publishing stuff (which is often quite image-heavy). The current image publishing framework I threw together for her years ago is a bit limiting now, and quite clunky to use.

• A mountable filesystem that lets me access archives, either in a generic manner (with auto-generated directories for every property, and every value of that property, containing all files with that value of that property) or with customised directory layouts (such as presenting my music collection as /music/ARTIST/ALBUM/NUMBER:TITLE.EXTENSION, with all the capitalised bits generated from the metadata). I'd like to do this by adding this to the explore mode virtual filesystem, and then having that mountable.

Ugarit performance (by alaric)

Ugarit was once renowned for its poor import performance, and rightly so. However, it's a lot faster these days - not, sadly, due to amazing optimisation work on my part, but because Thomas Hintz made write-u8vector! faster. It's not released yet, but will be in Chicken 4.10.

There's still work to be done, though. In my experiments with archival mode, I imported 9GiB into an archive in:

real    24m14.822s
user    17m51.485s
sys     1m59.920s

Writing an uncompressed tarball of the same 9GiB took:

real    8m49.931s
user    0m1.076s
sys     1m1.315s

That's a factor of 3. Ugarit spends four minutes waiting for I/O while tar spent eight minutes, which is puzzling, but Ugarit spent seventeen minutes of CPU time while tar spent one second; this will be down to the fact that Ugarit still copies each byte of the file several times between reading it in and writing it out, and I know how to fix that!

There will be some unavoidable architectural cost in the fact that Ugarit will always use at least two processes - a frontend and a backend, with the data sent over a pipe between them - but I think there's a lot I can reduce first. Onwards and downwards!