Ugarit performance (by alaric)

Ugarit was once renowned for its poor import performance, and rightly so. However, it's a lot faster these days - not, sadly, due to amazing optimisation work on my part, but because Thomas Hintz made write-u8vector! faster. It's not released yet, but will be in Chicken 4.10.

There's still work to be done, though. In my experiments with archival mode, I imported 9GiB into an archive in:

real    24m14.822s
user    17m51.485s
sys     1m59.920s

Writing an uncompressed tarball of the same 9GiB took:

real    8m49.931s
user    0m1.076s
sys     1m1.315s

That's a factor of 3. Ugarit spends four minutes waiting for I/O while tar spent eight minutes, which is puzzling, but Ugarit spent seventeen minutes of CPU time while tar spent one second; this will be down to the fact that Ugarit still copies each byte of the file several times between reading it in and writing it out, and I know how to fix that!

There will be some unavoidable architectural cost in the fact that Ugarit will always use at least two processes - a frontend and a backend, with the data sent over a pipe between them - but I think there's a lot I can reduce first. Onwards and downwards!

Ugarit archive mode manifest maker (by alaric)

When I last wrote about Ugarit progress, I had developed archive mode to the point where one could import a list of files with metadata from a "manifest file", and then search for files based on the metadata from the manifest and stream out chosen files. I gave an example of using this to play MP3s matching a search pattern:

[alaric@ahusai ugarit]$ for i in `ugarit search test.conf music '(= ($ artist) "UNKLE")' keys`;
do ugarit archive-stream test.conf music $i | mpg123 -;
done

Well, that was all based on hand-written manifest files, which are no fun to produce (our music collection is large). As such, I've been working on a "manifest maker" that takes a list of files and directories and makes a manifest file from them, recursing down through directories to list all the files. And for each, it automatically extracts metadata into the manifest file, which can then be hand-edited if required, and then used to import from.

The idea is that the manifest maker will have support for a number of file types it knows how to extract additional metadata from, and the first one I've implemented is ID3 tag extraction from MP3s. I've implemented the ID3 V2.2 and ID3 V2.3 specs, as those were the two that I found present in the subset of my MP3 collection I'm testing against!

For example, here's the output it produced for one of my MP3s:

(object "./test-data/THE HOLLIES - He Ain't Heavy, He's My Brother.mp3"
  (filename = "THE HOLLIES - He Ain't Heavy, He's My Brother.mp3")
  (mime-type = "audio/mpeg")

   ;; Unknown ID3 tag "COMM"="engiTunNORM\x00 00000402 00000000 00001B59 00000000 00004E65 00000000 000040EC 00000000 00015FD5 00000000"
  (keyword = "Pop")
  (name = "He Ain't Heavy, He's My Brother")
  (creator = "THE HOLLIES")
  (creation-date = "2002")
  #;(featuring = "")
  (collection-name = "Legends CD2")
  #;(collection-volume = "")
  #;(collection-volumes = "")
  (volume-index = 16)
  (volume-size = 18)

  (mtime = 1428948696.0)
  (ctime = 1428948696.0)
  (size = 4063360))

It prints out unknown ID3 tags as comments, in case a human can glean some useful information from them to put into the metadata, and it suggests the names of metadata tags I might be able to provide by hand that it hasn't found (in this case, a tag for other people featured in the music, and two for indicating that this album is part of a set. As it happens, it is, as the "CD2" in the name suggests, but it wasn't indicated in the ID3 so I'll have to hand-edit it; likewise, the date from the MP3 of 2002 is clearly for the production of the album, not that classic track... ID3 metadata is often a bit shabby!). Also included are file mtime, ctime, and size in bytes.

I hope to add Ogg Vorbis metadata next; I'd like to add EXIF support to parse information out of the JPEGs in our vast family photo library, but it looks much harder, and I'm not sure how useful it will actually be!

Experiments in Food: Soylent and Joylent (by alaric)

I was interested to hear about Soylent - the meal replacement, not Soylent Green - when it came onto the scene. I lead a busy life, which includes cooking for two kids and a wife (one of the children has an intolerance to cow milk, and my wife has an intolerance to gluten as well as milder issues with cow milk and soya). I often find myself in the situation of being quite hungry myself, at the start of needing to cook a complicated meal from fresh ingredients. Also, I don't like eating breakfast until an hour after getting up, so I tend to eat in the office on weekdays; due to a shortage of filling breakfast options that will last long enough for me to finish them off on two days a week, I often end up skipping breakfast at weekends or just grazing on raw root veg from the fridge, then feeling woozy come lunch time (about when I need to start organising lunch for the family). So the idea of a powder I can store for a long period and then turn into a balanced meal replacement with near-zero effort, and cheaply at that, certainly has some appeal.

Sadly, Soylent decided they can't ship to the UK (and muttered something about refunding my contribution to their crowdfunding effort on that basis, but not until November 2015 - and I funded them in July 2013!), so I gave up on the idea of giving it a go.

But thankfully, they have published their recipe online, which has prompted a Dutch company to set up shop making it and shipping it from the EU! They're called Joylent, and as the name suggests, are taking a rather light-hearted approach to producing basically the same stuff. So I gleefully ordered some, and have started experimenting with it.

I don't plan on living on the stuff, although some have - I just want an easy, filling, meal replacement for when circumstances require it.

So far, I've had two "meals" of it, and the results have already been somewhat interesting.

The first one was a weekend breakfast replacement. I started with the vanilla flavour; it was tasty and 600ml of the stuff went down easily, leaving me feeling satiated. I found I felt full, and with plenty of energy, but I was craving crispy and strong-tasting food; I think my mind didn't quite believe that a soupy liquid with a gentle vanilla flavour could have actually fed me. Although I was craving salty fries and pickles, I didn't actually want to eat anything; I just wanted those flavours, and would probably have been happy to just nibble a tiny amount or something.

Come 1:30pm, four and a half hours later, I felt a sudden pang of hunger, but it passed quickly. I still didn't feel light-heated and ill as I often do when I can't easily eat. In this particular case, I was on a long drive, so we didn't get to stop for lunch until 3pm. I was feeling... "peckish" by then; I fancied the idea of eating, but wasn't suffering from hunger, which was unusual for such a late lunch. I ate a paneer tikka wrap and the leftovers of Sarah's nachos, so plenty of exciting textures and strong tastes, which was exactly what I wanted! I didn't have to eat very much to feel full and satiated, and had a light (and nutritionally meagre) dinner of chips and ketchup that evening (due to lack of alternatives, being a vegetarian in a place that focussed on the eating of sausages), and went to bed not feeling malnourished at all.

The next morning, I had a whole grapefruit for breakfast, but was feeling pretty hungry come lunchtime at 1pm. Sadly, the place we ate was focussed on the eating of roasted animals, so all I had for lunch was a small plate of roasted potatoes and steamed vegetables, which was tasty but not very nutritionally diverse (I'd eaten little protein since lunch the day before). So before setting off on the drive home, somewhere in the middle of the afternoon, I had a second vanilla Joylent. It was pleasing that I'd been able to chuck my nice Joylent mixing bottle and the open pack in my bag for the trip; I bought a bottle of water in a shop to mix it up, but if I'd brought my own bottle of water I'd have been ready to throw together a "meal" wherever and whenever I wanted.

I was once again satiated, although a little less so; I think I put less powder in (judging a third of a pack of powder is tricky, although I think I'll soon be able to work out how big a third of a pack looks in the mixing bottle and get it right in future), as it came out a lot more watery this time, despite not being full quite to the 600ml mark. Once again, I was quickly craving crunchy strong tastes, so at about 8pm, I ate some salt and vinegar crisps at a motorway service station. This quickly led to me feeling I'd eaten too much salt; I felt a bit dehydrated and had a nasty salty taste lingering in my mouth. I suspect the morale of that story is that I've become accustomed to eating too much salt; I need to train my mind to realise that I don't need to have tasted savoury salty flavours to have eaten a meal!

As I lie in bed typing this at 11:15pm, I'm feeling a bit hungry, but not uncomfortably so (I didn't have dinner or anything else to eat).

The Joylent flavours are banana, chocolate, strawberry and vanilla; I think it would probably be a good idea for me to train myself out of craving salty tastes in a meal, but it's quite interesting that I've not found anything else I've eaten lately as filling as the Joylent, nor able to keep me "going" as long. I could see myself living happily on the stuff, but I would really miss food tastes and textures. However, it's made me more aware of how nutritionally limited a lot of foods are. I like the thought of using something like this as "fuel" and then having small quantities of spicy crunchy foods for the taste!

Also, it would be interesting to try and make a curry flavoured Joylent. Either get some without any flavourings added, or start with a mild-tasting one and blend in a nice mix of spices. I may have to perform some experiments in that area!

Cabling my workshop (by alaric)

As a child, I read a novel called Triplanetary, which featured a character called Gray Roger. Roger was the evil master of a space-born flying citadel, and usually sat in an office, at a massive control desk. Through this desk, he controlled the massive machinery driving the citadel and its tremendous weapons.

This image (and many others like it) meant a lot to me; for as long as I remember, I have wanted a Secret Lair with Control Panels.

When we bought our house, one of the big selling points for me was a small building at the end of the garden, which I turned into my workshop. The far end has my computer desk and my electronics desk, and the end by the doors is a messy area for metalworking.

However, the years since have been blighted by problems with the roof leaking, which has damaged furniture and tools, and caused horrible mold to grow on everything. Last summer, with the help of a friend, I properly felted the roof, which seems to have solved the leaking. There is still work to be done cleaning the interior and re-painting stained floors, but the inside has now dried out.

However, even aside from the roof problems, I had no Internet connection to the workshop. Wifi couldn't reach from the house, and wouldn't be sufficient for my needs. So even with the place drying out, I've not had a place I can sit and work with a computer at home; everything was done with laptop on lap on the sofa, or sat at the kitchen table, or similar - hardly comfortable, while I have a proper desk and office chair sitting out there!

So I have long wanted to run Ethernet cables down to the workshop, and haven't had a chance to organise it while I'm focused on trying to fix the roof.

But the colocation deal I use to host my Internet server is coming to an end, and rather than moving to a new one, I'm going to run the server from home. We have reliable, low latency, high-speed fibre broadband from Andrews and Arnold, and a server hosted at home will be one I can attend to for maintenance, upgrades and emergencies; having the current one based in London is somewhat inconvenient!

As there's nowhere good in the house to keep servers, they need to go in the workshop - so I now have a deadline to get Ethernet out there. And over the past few weekends, I have done just that, with much help from Jean (a lot of cabling work cannot be done single-handed; long bits of trunking need to be held up at both ends before being stuck down, and pulling cable through conduit requires somebody pulling at one end while somebody else feeds the cable in carefully at the other).

We had to use no less than three different cabling techniques, as the cables pass through several different regions.

It all starts in the cupboard under the stairs, where the PPPoE termination of the fibre broadband from the phone network is. I've since mounted the patch panel onto the wall and tidied up the cabling, but here's how it looked earlier:

The cables go through the wall (which was once the external wall of the house before it was extended, so required drilling with a massive diamond core drill) into the library, where I've put a neat white box over the hole:

The trunking goes off to the kitchen wall. The hole through that wall is smaller, so can be hidden behind the trunking at the left. In the kitchen I used smaller trunking that can't cover the hole, so another white box covers the hole on this side:

Across the garden, we had to switch from trunking to waterproof conduit, all sealed together with funky glue, and access boxes with waterproof gaskets:

Then inside the workshop, I used suspended cable trays. I love suspended cable trays - I can just lay extra cables up there whenever I need, which is of course a great feature for a Lair full of Controls:

Finally, the cables come down from the tray system into the comms cabinet:

Last night, I tested all four cables with a cable tester, and all pairs are OK. I've put the desktop PC I built (but never got to use) in the workshop and hooked it up to one of them, and it's working fine at Ethernet frequencies, too!

Having a desktop machine is a great boon, even if I'm sat in the house on my laptop - I bought a cheap, light, laptop with a view to using it as a remote terminal to the much beefier desktop machine where possible.

And, of course, no Lair is complete without a way of disposing of enemies. Having had enough of water around the place already, I opted against the traditional shark pits, and instead went for a cage full of dinosaurs instead.

And so now, I can sit in my bed with my laptop, and make the computer do things in the cold, dark, workshop at the end of the garden. Muhahah.

Oh, UKIP… (by alaric)

This came up on Twitter:

Have you seen who @UKIP candidate @glenntingle follows on Twitter? https://t.co/FH6wojKzKY (Warning possibly NSFW!)

— Dave Cross (@davorg) March 31, 2015

But Glenn Tingle quickly shut down his account, so the world could no longer see what his interests are.

Thankfully, I'd nabbed a screenshot:

@Zoonie @davorg I have a screenshot. There's pages and pages of near-naked women, this just shows the first two! pic.twitter.com/PVyXPiISAr

— Alaric Snell-Pym (@alaric) March 31, 2015

Oh dear. Oh dear oh dear.