Ugarit 2.0 released (by alaric)
Unless I messed up the release process, Ugarit version 2.0 is now available from the Chicken Scheme egg repository.
What does this mean to you, dear reader? Not a huge amount; you can go and read the release notes at the bottom of the Ugarit documentation page for the fine detail. But, to summarise:
The beginnings of archival mode! As well as storing chains of snapshots of a filesystem, as Ugarit has always done (generally to be used as a versioned backup system), Ugarit vaults can now also store "archives", which are groups of files or directory trees identified by arbitary metadata, such as "This is the song 'Ooofarno' by 'Bobby and the Beaters', which is track 11 out of 12 from the 'Fishes In The Sea' album", or "This is a photo of Aunt Mavis taken at 13:58 on the 3rd of August, 2020, at Uncle Bob's 100th birthday party", or "This is a PDF of a paper by Donald Knuth on ternary numbers, called 'Simplified Arithmetic in Base Three'". You can then find things by searching on this metadata, which is much, much, nicer than creating trees of directories to organise all your stuff into. The user interface for getting things in and out of archives is still a bit minimal - but I have plans to fix that.
The way tags are stored has changed. Ugarit 2.0 will read vaults created by prior versions happily, but when it writes to a vault, it'll write new-format tags (which have type information as well as a pointer to something), which old versions of Ugarit won't be very pleased to find.
We now store a "metadata block" in every vault, pointed to by a hidden tag (we didn't used to be able to hide tags, so old versions of Ugarit will show you a funny tag, and complain if you try and do anything with it, as it's a new-format tag). This stores a vault format version number, so we can better handle incompatible changes to the vault format going forward; and as it's hashed and encrypted like any other block, it means you'll get an instant error if you try and connect to a vault using the wrong hashing and encryption settings, rather than bizarre errors further down the line.
We've made it possible to store large logs in the vault. When we do a snapshot or an archive import or something, we keep a log of warnings and recoverable errors that cropped up while doing so. This is stored as a file attached directly to the snapshot or import object, so it can be arbitrarily large.
Added
log.sexprandproperties.sexprfiles in the explore-mode interface, inside every snapshot or archive import object, which let you access the log and the metadata. These are files you can extract, or you can look at them with the newcatcommand.Added a
catcommand in the explore-mode user interface to dump a file to the screen for viewing.Added a client-side cache of snapshots and imports, which significantly speeds up the exploration of backup histories and archive metadata. Optionally, you can make the cache persist between sessions, otherwise it's made afresh for every explore session.
As well as the existing ability to fork a tag into two tags that share the same history (applicable both to snapshot tags and archive tags), added the ability to merge two tags into one, melding the two histories into one. This includes some exciting logic to combine those histories for display in explore mode!
Added a new sqlite backend, which provides a storage inside a single file, managed by sqlite. I wrote it to make testing easier, but it's a useful storage backend in its own right!
Tidied up the Ugarit internals, splitting the core up into a load of separate modules. This makes development easier for me, and means nothing to users.
So what's next? I want to improve the usability of archive mode - right now, it's quite easy to import a bunch of files, but you have to hand-edit a text file to provide metadata beyond what it can automatically extract (currently just basic file information, plus whatever it can extract from ID3 tags and Ogg metadata); and then you can explore the history of the archive (as a series of imports) through the explore interface, or use a command-line tool to search for files, and then extract them or stream them to standard output.
What I want is:
A shiny (web-based?) UI for searching the archive, seeing thumbnails of images, and the ability to download files with a single click or to perform bulk editing operations on metadata with ease and panache.
A music player, that lets me pick music from an archive to queue, or to be given an arbitrary search criteria to find music to random-play, playing direct from the archive.
A way to pick photos and assemble them into galleries, which are then publicly visible through a Web interface. Sarah wants to be able to put sequences of photos together, as well as individual photos that don't form parts of sequences, into multiple albums, for her blog publishing stuff (which is often quite image-heavy). The current image publishing framework I threw together for her years ago is a bit limiting now, and quite clunky to use.
A mountable filesystem that lets me access archives, either in a generic manner (with auto-generated directories for every property, and every value of that property, containing all files with that value of that property) or with customised directory layouts (such as presenting my music collection as
/music/ARTIST/ALBUM/NUMBER:TITLE.EXTENSION, with all the capitalised bits generated from the metadata). I'd like to do this by adding this to the explore mode virtual filesystem, and then having that mountable.
Sat 13th Jun 2015 1:02 pm 
Subscribe
By @ndy, Sat 13th Jun 2015 @ 4:01 pm
Good work! 🙂
By sidhu1f, Wed 17th Jun 2015 @ 5:31 pm
Cool! Any performance comparison with backup programs based on similar ideas (but written in python) like obnam and attic? Also, any idea of the effort required to port ugarit to guile?