Ugarit performance (by alaric)

Ugarit was once renowned for its poor import performance, and rightly so. However, it's a lot faster these days - not, sadly, due to amazing optimisation work on my part, but because Thomas Hintz made write-u8vector! faster. It's not released yet, but will be in Chicken 4.10.

There's still work to be done, though. In my experiments with archival mode, I imported 9GiB into an archive in:

real    24m14.822s
user    17m51.485s
sys     1m59.920s

Writing an uncompressed tarball of the same 9GiB took:

real    8m49.931s
user    0m1.076s
sys     1m1.315s

That's a factor of 3. Ugarit spends four minutes waiting for I/O while tar spent eight minutes, which is puzzling, but Ugarit spent seventeen minutes of CPU time while tar spent one second; this will be down to the fact that Ugarit still copies each byte of the file several times between reading it in and writing it out, and I know how to fix that!

There will be some unavoidable architectural cost in the fact that Ugarit will always use at least two processes - a frontend and a backend, with the data sent over a pipe between them - but I think there's a lot I can reduce first. Onwards and downwards!

Ugarit archive mode manifest maker (by alaric)

When I last wrote about Ugarit progress, I had developed archive mode to the point where one could import a list of files with metadata from a "manifest file", and then search for files based on the metadata from the manifest and stream out chosen files. I gave an example of using this to play MP3s matching a search pattern:

[alaric@ahusai ugarit]$ for i in `ugarit search test.conf music '(= ($ artist) "UNKLE")' keys`;
do ugarit archive-stream test.conf music $i | mpg123 -;
done

Well, that was all based on hand-written manifest files, which are no fun to produce (our music collection is large). As such, I've been working on a "manifest maker" that takes a list of files and directories and makes a manifest file from them, recursing down through directories to list all the files. And for each, it automatically extracts metadata into the manifest file, which can then be hand-edited if required, and then used to import from.

The idea is that the manifest maker will have support for a number of file types it knows how to extract additional metadata from, and the first one I've implemented is ID3 tag extraction from MP3s. I've implemented the ID3 V2.2 and ID3 V2.3 specs, as those were the two that I found present in the subset of my MP3 collection I'm testing against!

For example, here's the output it produced for one of my MP3s:

(object "./test-data/THE HOLLIES - He Ain't Heavy, He's My Brother.mp3"
  (filename = "THE HOLLIES - He Ain't Heavy, He's My Brother.mp3")
  (mime-type = "audio/mpeg")

   ;; Unknown ID3 tag "COMM"="engiTunNORM\x00 00000402 00000000 00001B59 00000000 00004E65 00000000 000040EC 00000000 00015FD5 00000000"
  (keyword = "Pop")
  (name = "He Ain't Heavy, He's My Brother")
  (creator = "THE HOLLIES")
  (creation-date = "2002")
  #;(featuring = "")
  (collection-name = "Legends CD2")
  #;(collection-volume = "")
  #;(collection-volumes = "")
  (volume-index = 16)
  (volume-size = 18)

  (mtime = 1428948696.0)
  (ctime = 1428948696.0)
  (size = 4063360))

It prints out unknown ID3 tags as comments, in case a human can glean some useful information from them to put into the metadata, and it suggests the names of metadata tags I might be able to provide by hand that it hasn't found (in this case, a tag for other people featured in the music, and two for indicating that this album is part of a set. As it happens, it is, as the "CD2" in the name suggests, but it wasn't indicated in the ID3 so I'll have to hand-edit it; likewise, the date from the MP3 of 2002 is clearly for the production of the album, not that classic track... ID3 metadata is often a bit shabby!). Also included are file mtime, ctime, and size in bytes.

I hope to add Ogg Vorbis metadata next; I'd like to add EXIF support to parse information out of the JPEGs in our vast family photo library, but it looks much harder, and I'm not sure how useful it will actually be!

Oh, UKIP… (by alaric)

This came up on Twitter:

Have you seen who @UKIP candidate @glenntingle follows on Twitter? https://t.co/FH6wojKzKY (Warning possibly NSFW!)

— Dave Cross (@davorg) March 31, 2015

But Glenn Tingle quickly shut down his account, so the world could no longer see what his interests are.

Thankfully, I'd nabbed a screenshot:

@Zoonie @davorg I have a screenshot. There's pages and pages of near-naked women, this just shows the first two! pic.twitter.com/PVyXPiISAr

— Alaric Snell-Pym (@alaric) March 31, 2015

Oh dear. Oh dear oh dear.

R.I.P. Terry Pratchett (by sarah)

I wanted to blog about Sir Terry Pratchett the fantasy author last week but could not. I found out via his social media feeds and it struck to the marrow in away that I wasn't expecting nor prepared for. My facebook status read "Oh no frown emoticon Damn you upload tech you are coming to slow - we've lost another..." with a link to the authors page. With Leonard Nimoy's death as well I was hurting.

I'd known Terry's demise was coming, I've know since we went to his talk at the Cheltenham Lit Festival in 2007. There are sadly no photos due to the camera getting busted during the house flooding and there is no signed book either because I'd spent all my money on getting the tickets to the event and paying for parking and fuel.

It was an amazing experience and formative experience for me, but he announced he was having little strokes/bleeds and we knew what that meant. Watching over the years as he appeared at other things saying he could no longer read, or do many things was heart breaking. But he kept writing, he kept being inspirational and though opinions amongst my friends varies about the quality of those stories I've liked a lot of them.

I felt slightly awful as one of my first thoughts was... "I hope he finished the book for the autumn." It is also weird that a man I met/saw in the flesh once, just once in my life could upset me so with his death - I always used to think that people who morned for celebrates were stupid - I was wrong.

His books have been a big thing in my life. It all started with my GCSE Chemistry teacher who wrote in my leaving book to try his books out, she thought they would be perfect for me and she was right. I did not however go straight out and grab one, that had to wait until I was struggling with my A'levels, by this point I was doing a creative writing course at lunch times at the college and I was kind of getting sick of the accusations of plagerism especially on stuff I worked on jointly with my dad. Plus revision was coming up and I needed some distraction.

I used to go to Hornchurch library to procrastinate, I could be found "doing my homework" on the table by the geology etc... books not many of the wall of books around me would be anything to do with my homework. I rationed myself on the fiction books though as otherwise I would stay up all night, I had just written a story about falling off of a dragons back and puncturing reality and again been told it sounded like Pratchett. I went over to the sci-fi and fantasy stand (they did not have a book case - horror just about managed and was nearly all Stephen King), they had just put out some new books, one of which had a cartoony cover... I flicked it open having noted the name on the cover. It was Watchmen, it was about a dragon and sure enough it was like listening to me and my dad making up stories together.

I did not put it down, I finished it in three days - I am still a slow reader and was in the middle of A'levels - four A'levels when the normal was three plus lots and lots of extra stuff like the Geology Society, Creative Writing courses, Guiding and Thriftwood Campsite not to mention jobs and unsuitable boyfriends.

I hoovered up the libraries grand total of 3 discworld novels, Dad found me another two in a second hand shop in Brentwood. They however turned out not to be discworld but The Dark Side of the Sun and Carpet People. Throughout my A'levels I lent heavily on Terry to pace me through essays and all nighters, being slightly problematic in that there were rarely chapter breaks so I would have to remember to stop at ten page marks. Dad ended up reading them - he ended up a huge huge fan! I had to make him keep writing, assuring him that he is different enough (one day I will get his work out there!).

I learnt a lot during my A'levels and alot of the really important stuff like how to be myself and people can be dicks, came from his books. Of course they were not the only fiction I was reading but they were the only ones that I was saving up to buy models from - not that I ever managed that one 🙁

During the last leg of my A'levels I had a triple exam day, I had a clash! Not of two exams but of three and one of them was a long everything exam for Classical Civilizations - apparently it is not normally taken with with sciences. I wasn't allowed to talk to anyone and I was pushing the boundary of having to stay over night somewhere in isolation. I had to be escorted by teachers - I am in truth a shy person and found this highly worrying and embarrassing, I wasn't even able to go and buy food so had packed dinners. I also had Small Gods - I am not a last minute crammer, the dyslexia rules that out as an exam passing strategy, mainly I need to relax and not have panic attacks between exams. I read the whole book between the exams and felt like a complete zombie with brain over load - the book kept me sane, the book pointed out certain things that I was beginning to suspect myself already. I think bits of it may have accidently sneaked into my Clas Civ exam essay.

The summer between college (I was at a sixth form college and not at school for my A'levels) I worked at Thriftwood Campsite as an outdoors pursuits instructor, by the end of the summer I was the Duty Warden due to illness sweeping through our ranks. On of the boys (the guy who should have been in charge) lent me his huge, huge stack of discworld novels and a minidisc player. So when I suddenly found myself in charge (there was the actual Warden and Sub-Warden but they lived off site though not a thousand miles away and the next summer I was a Duty Warden from the start), the discworld helped me unwind.

The books were a mirror and not as distorted as people seem to think. I got to go to uni - I was sick, I was ill, Harry Potter and Vimes got me through. I had to repeat my first year of uni and when I had to go to the meeting for them to decide what was best, I had role models that didn't exist in normal lit. I had the witches which fitted me and my friends and my family, the Watchmen and the softest spot I reserve for Rincewind. Not to mention Imperial College is a rambling misfit of buildings with cellars that are linked and tunnels to various other bits like the museums etc. There was also the Queens Tower - I was blatently in a discworld novel.

I remember the thrill of The Fifth Elephant posters going up in the tunnel that goes to South Kensigton Station. And when I returned for my second and rather more successful bash at university I had scanned the book covers and printed them out A4 to be posters for my wall, this was how I then found the sci-fi, horror and fantasy library as they pretty much came and claimed me once told about my wall.

I was never competely well after that though and when stuck in bed, when unable to run and jump I turn to three book groups - Discworld Novels, Harry Potter and Philip Pullman. These got me through weeks of oxygen monitoring and bed rest during Jean's pregnancy, they got me through nights of breast feeds. They have been read so many times I am on second and even third copies of some of them.

For someone who couldn't read until I was 12 - books are increadibly important to me and none more so than Terry Pratchett's books. The stories curl around themselves so that I do not have stories in my head anymore but worlds, I feel such a sense of loss that the mind that produced these works is gone. They started as satire but became something more.

Poor Jeany has only just started reading his work and I have censored a little bit - she is 9. She went as Tiffany Aching for world book day - one of his characters. I gave her the book about the 9 yr old witch because she is struggling as we all do with growing up and being different and I knew that though I was never like Tiffany - Jean is. This was someone who could be a fictional role model for her but realistic, that make mistakes, some really awful ones and how you get through that. Jean loved the book. There was a horror on her face when I told her he had just died, "but I'll run out of his books now" - it was a genuine sorrow.

She made Nac Mac Feegles, and had a frying pan. She also declared that she NEEDED book 2 in the Bromiliad series (Books of Gnomes).

Part of the great ache is that though the world is missing him, no one had the right to ask him to stay, he was suffering and loosing him self a fragment at a time. The tweets on his twitter feed are loving and sad and that deep humour that is something more than comedy or tragedy.

And of course there is something else, Terry Pratchett always read to me like my dad's stories - the ones I grew up with and going to see him speak made this more apparent. It was like watching my Dad's ulter ego, the one who got on with writing the stories down. So it's also a stark reminder for me that my Dad is not immortal and worse, I've always bought him the latest Pratchett book when he's been ill etc... mortality what a fragile thing you are.

Which leads me on to the last bit. To conquer my own sadness I drew some pictures and I posted one of these to social media, it is not brilliant, it is just a sketch and it's not very big and it was done for me. But people have been asking to buy it. It is not for sale, I'm afraid, as my Dad has asked for it. However, when I said this, some of you have been asking for prints - there will be prints, it might take a while as I need to clean the image up and I'll also do some of my other pictures, there will also be a page with high res versions on for download. But I can not/will not take money for these, instead if you request one or download one then maybe think about giving money to a research or care charity. There is already a Just Giving page for Terry Pratchett supporting the charity NICE which is the one his daughter has asked people to give to.

And for anyone who knows me in the non-internet world, I am still doing a tribute party, I just need to find a free weekend which is being tricky! (the cake pic is from my dad's 65th birthday).

And for anyone wanting to see good tribute art I suggest you check out the Multiverse board I've started on Pinterest.

Last few bits - that's I've just remembered I wanted to mention:

The xkcd comic made me cry.

I felt a huge sense of relief when I first read about L-space and all libraries being connected through out time and space, this occured just after I'd started to discover all the book/libraries/museum burnings in our history.

When I was in labour with Jean everything was going wrong, dangerously so and I couldn't speak and was basically dying. I was very lucky but at the time death was no longer scary - taking my baby with me however was. Within me I knew death and it was a combination of the Aztec woman, mother, warrior, birth and death and Death from the discworld. A sort of cross between Terry Pratchett's Death and Neil Gaiman's I suppose.

It sounds hippy and I'm not saying it's not but I clung to life by thinking of the glow of universal power in deaths eye sockets, by remembering I was as much part of existence as everything else. I know doubt would have found some other imagery but it is always locked in my mind. And because I am me this appears in a poem and the poem has been on the radio and stuff, I always wondered if anybody would spot the connection.

This picture is one I started painting after I read my first discworld novel - I stopped when someone pointed out the elephants were supposed to be the other way around. I am now considering finishing it anyway.

And finally - my thoughts go out to Terry's family and friends because what I am feeling is but a shadow of their pain and remembrance. RIP sir.

Anti-abortion protestors in Gloucester (by alaric)

I nipped into the center of Gloucester at lunchtime today. Doing so takes me past Gloucester Royal Hospital, and on my way in, I saw two men outside the sexual health clinic clinic (Hope House), with a big banner on the railings reading "Pray to end abortion". Looks like it's made the local news: Women using Hope House reassured that an anti-abortion protest will not affect their treatment.

On the way back, standing in front of them was a woman holding up a bit of cardboard upon which had been written "YOUR BODY, YOUR CHOICE". I gave her a grin as I passed; one of the men was muttering something inaudible to her, and I heard her start to respond with "It's been shown in studies that...".

Is this going to get worse? Are we going to need a clinic escort system, like they have in America? I work just opposite the hospital, and am not afraid to get shouted at or attacked while protecting people's rights; so if anybody does feel intimidated about visiting Hope House, I'll happily walk you in/out if I've got any break time free. You can ask me on Twitter.