Further progress on Ugarit archival mode (by alaric)

Further to my last post on the matter, I've been working on the basic user interface to accessing archive metadata.

As before, let's do an import to an archive tag in a vault. I've made a manifest file with three MP3s in - all data that could be extract from ID3 tags, and I plan to write a tool to automate the generation of manifests by examining their contents in exactly that manner, but for now I had to hand-write one:

[alaric@ahusai ugarit]$ cat test.manifest
(object "/home/alaric/archive/sorted-music/UNKLE/Psyence Fiction/13 Be There.mp3"
        (title = "Be There")
        (track = 13)
        (artist = "UNKLE")
        (album = "Psyence Fiction"))

(object "/home/alaric/archive/sorted-music/UNKLE/Psyence Fiction/11 Rabbit in Your Headlights.mp3"
        (title = "Rabbit in Your Headlights")
        (track = 11)
        (artist = "UNKLE")
        (album = "Psyence Fiction"))

(object "/home/alaric/archive/sorted-music/Led Zeppelin/Remasters/1-09 Celebration Day.mp3"
        (title = "Celebration Day")
        (track = 9)
        (volume = 1)
        (artist = "Led Zeppelin")
        (album = "Remasters"))

As before, I import it, loading the files into the content-addressible storage of the vault, automatically deduplicating, and possibly storing the data on a cluster of remote servers (although in this case, I'm just using a local vault). This was done with Ugarit revision [80b324f3af]:

[alaric@ahusai ugarit]$ ugarit import test.conf music test.manifest
Loading manifest file test.manifest...
Importing from test.manifest to tag music...
Importing /home/alaric/archive/sorted-music/Led Zeppelin/Remasters/1-09 Celebration Day.mp3...
...imported with key 4d64e4650333741cb56c3e6a785b6de4d23324cb1055e529
Importing /home/alaric/archive/sorted-music/UNKLE/Psyence Fiction/11 Rabbit in Your Headlights.mp3...
...imported with key 370bee7debb458357a2b879014d4abbeb409215ed269c1c6
Importing /home/alaric/archive/sorted-music/UNKLE/Psyence Fiction/13 Be There.mp3...
...imported with key 39df8bafd530a66614ad60ab323033b1385cdd842528dbd2
Committing import...
Imported successfully to tag music with import key ac26354ccfb0530109932c1aaddd414b59d4394d44ec43cd
Written 16MiB to the vault in 24 blocks, and reused 0B in 1 blocks (before compression)

But now it's in, we can query the metadata. Firstly, let's see what properties are available - a combination of the ones we wrote in the manifest, and automatically-generated ones such as a MIME type and the original import path:

[alaric@ahusai ugarit]$ ugarit search-props test.conf music
album
artist
filename
import-path
mime-type
title
track
volume

Let's see what values there are for the "artist" property:

[alaric@ahusai ugarit]$ ugarit search-values test.conf music artist
UNKLE
Led Zeppelin

(they're sorted by popularity, and we have two UNKLE tracks, so that comes first)

Let's see what UNKLE albums we have, by filtering for objects with an artist property of "UNKLE" and asking what values of the "album" property are available:

[alaric@ahusai ugarit]$ ugarit search-values test.conf music '(= ($ artist) "UNKLE")' album
Psyence Fiction

Let's see what we know about music by UNKLE:

[alaric@ahusai ugarit]$ ugarit search test.conf music '(= ($ artist) "UNKLE")'
object 39df8bafd530a66614ad60ab323033b1385cdd842528dbd2
    (album = "Psyence Fiction")
    (artist = "UNKLE")
    (filename = "13 Be There.mp3")
    (import-path = "/home/alaric/archive/sorted-music/UNKLE/Psyence Fiction/13 Be There.mp3")
    (mime-type = "audio/mpeg")
    (title = "Be There")
    (track = 13)
object 370bee7debb458357a2b879014d4abbeb409215ed269c1c6
    (album = "Psyence Fiction")
    (artist = "UNKLE")
    (filename = "11 Rabbit in Your Headlights.mp3")
    (import-path = "/home/alaric/archive/sorted-music/UNKLE/Psyence Fiction/11 Rabbit in Your Headlights.mp3")
    (mime-type = "audio/mpeg")
    (title = "Rabbit in Your Headlights")
    (track = 11)

Ok, let's listen to all our music by UNKLE (the extra "keys" parameter to the search command says to just output the object keys, one per line, and the "archive-stream" command streams the contents of an archived file to standard output):

[alaric@ahusai ugarit]$ for i in `ugarit search test.conf music '(= ($ artist) "UNKLE")' keys`;
do ugarit archive-stream test.conf music $i | mpg123 -;
done

...music by UNKLE plays...

We're slowly moving towards having a usable and useful archival filesystem, backed on a modular content-addressible storage system! Isn't that neat? Of course, it's not amazingly useful as it stands - at first sight, it's like a very crude version of the browser found in any modern music collection management app these days; but this is the seed of something much more interesting. For a start, it can categorise files using any user-defined schema. The backend storage can be encrypted, and accessed remotely over a network (and, in future, replicated over a cluster, or mirrored between your laptop and a home fileserver, and automatically synchronised when they're connected). The same storage can be used to store backup snapshots as well as archives, and if files exist in any combination of archives and snapshots, then only one copy of it will be stored (or need uploading, even); most files in an archive will have started off in a backed-up directory tree, or will be extracted into one.

There are many interesting use cases for Ugarit, but my personal one is to have a fault-tolerant vault of all the data that matters to me, neatly organised so I can find things quickly, and so I can access things from different locations (even when offline). Rather than having files scattered over different disks on different machines, and having to move things around to make space, and remember where they are, I can add more disks to the vault when I need more capacity, and have Ugarit manage everything for me. With the amount of data I manage, that'll be a great weight off my mind!

Configuring replication (by alaric)

Storing all your data on one disk, or even inside one computer, is a risky thing to do. Anything stored in only one, small, physical location is all too easily destroyed by flood, fire, idiots, or deliberate action; and any one electronic device is prone to failure, as its continued functioning depends on the functioning of many tiny components that are not very easily replaced.

So it's sensible to store multiple copies, ideally in physically remote locations.

One way of doing this is by taking backups; this involves taking a copy of the data and putting it into a special storage system, such as compressed files on another disk, magnetic tape, a Ugarit vault, etc.

If the original data is lost, the backed-up data can't generally be used as-is, but has to be restored from the backup storage.

Another way is by replicating the data, which means storing multiple, equivalent, copies. Any of those copies can then be used to read the data, which is useful - there's no special restore process to get the data back, and if you have lots of requests to read the data, you can service those requests from your nearest copy of it (reducing delays and long-distance communication costs). Or you can spread the read workload across multiple copies in order to increase your total throughput.

Replication provides a better quality of service, but it has a downside; as all the copies are equally important, you can't use cheaper, slower, more compact storage methods for your extra copies, as you can with backups onto slower disks or tapes.

And then there's hybrid systems, perhaps were you have a primary copy and replicate onto slower disks as a "backup", while only using the primary copy for day-to-day use; if it fails then you switch to the slower "backup replica", and tolerate slower service until a new primary copy is made.

Traditionally, replicated storage systems such as HDFS require the administrator to specify a "replication factor", either system-wide or on a per-file basis. This is the number of replicas that must be made of the file. Two is the minimum to actually get any replication, but three is popular - if one replica is lost, then you still have two replicas to keep you going while you rebuild the missing replica, meaning you have to be unlucky and have two failures in quick succession before you're down to a single copy of anything.

However, this is a crude and nasty way of controlling replication. Needless to say, I've been considering how to configure replication of blocks within a Ugarit vault, and have designed a much fancier way.

For Ugarit replication, I want to cobble together all sorts of disks to make one large vault. I want to replicate data between disks to protect me against disk failures, and to make it possible to grow the vault by adding more disks, rather than having to transfer a single monolithic vault onto a larger disk when it gets full.

But as I'm a cheapskate, I'll be dealing with disks of varying reliability, capacity, and performance. So how do I control replication in such a complex, heterogeneous, environment?

What I've decided is to give each "shard" of the vault four configurable parameters.

The most interesting one is the "trust". This is a percentage. For a block to be considered sufficiently replicated, then copies of it must exist on enough shards that the sum of the trusts of the shards is more than or equal to 100%.

So a simple system with identical disks, where I want to replicate everything three times, can be had by giving each disk a trust of 34%; any three of them will sum to 102%, so every block will be copied three times.

But disks I trust less could be given a trust of 20%, requiring five copies if a block is stored only on such disks - or some combination of good and less-good disks.

That allows for simple homogeneous configurations, as well as complex heterogeneous ones, with a simple and intuitive configuration parameter. Nice!

The second is "write weighting". This is a dimensionless number, which defaults to 1 (it's not compulsory to specify it). Basically, when the system is given a block to store, it will pick shards at random until it has enough to meet the trust limit of 100%. But the write weighting is used as a weighting when making that random choice - a shard with a write weightinh of 2 will get twice as many blocks written to it as a normal block, on average.

So if I have two disks, one of which has 2TiB free and the other of which has 1TiB free, I can give a write weighting of 2 to the first one, and they'll fill so that they're both full at about the same time.

Of course, if I have disks that are now completely full in my vault, I can set their write weighting to 0 and they'll never be picked for writing new blocks to. They'll still be available for reading all the blocks they already have. If I left the write weighting untouched everything would still work, as the write requests failing would cause another shard to be picked for the write, but setting the weighting to 0 would speed things up by stopping the system from trying the write in the first place.

The third parameter is a read priority, which is also optional and defaults to 1. When a block must be read, the list of shards it's replicated on is looked up, and a shard picked in read priority order. If there are multiple shards with the same read priority, then one is picked at random. If the read fails, we repeat the process (excluding already-tried shards), so the read priority can be used to make sure we consult a fast, nearby, cheap-to-access local disk before trying to use a remote shard, for instance.

By default, all shards have the same read priority, so read requests will be randomly spread across them, sharing the load.

Finally, we have a read weighting, which defaults to 1. When we randomly pick a shard to read from, out of a set of alternatives with the same priority, we weight the random choice with this weighting. So if we have a disk that's twice as fast as another, we can give it twice the weighting, and on a busy system it'll get twice as many reads as the other, spreading the load fairly.

I like this approach, since it can be dumbed down to giving defaults for everything - 33% trust (for a three-way replication), and all the weightings and priorities at 1 (to spread everything evenly).

Or you can fine-tune it based on details of your available storage shards.

Or you can use extreme values for various special cases.

Got a "memcached backend" that offers fast storage, but will forget things? Give it a 0% trust and a high write weighting, so everything gets written there, but also gets properly replicated to stable storage; and give it a high read priority, so it gets checked first. Et voila, it's working as a cache.

Got 100% reliable storage shards, and just want to "stripe" them together to create a single, larger, one? Give them 100% trust, so every block is only written to one, but use read/write weightings to distribute load between them.

Got a read-only shard, perhaps due to its disk being full, or because you've explicitly copied it onto some protected read-only media (eg, optical) for security reasons? Just set the write weighting to 0, and it'll be there for reading.

Got some crazy combination of the above? Go for it!

Also, systems such as HDFS let you specify the replication factor on a per-file basis, requiring more replication for more important files (increasing the number of shard failures required to totally lose them) and to make them more widely avilable in the cluster (increasing the total read throughput available on that file, useful for small-but-widely-required files such as configuration or reference data). We can do that to! By default, every block written needs to be replicated enough to attain 100% trust - but this could be overriden on a per-block basis. Indeed, you could store a block on every shard by setting a trust target of "infinity"; normally, when given a trust target it can't meet (even with every shard), the system would do its best and emit a warning that the system is in danger, but a trust target of "infinity" should probably suppress that warning as it can be taken to mean "every shard".

The trust target of a block should be stored along with it, because the system needs to be able to check that blocks are still sufficiently replicated when shards are removed (or lost), and replicate them to new shards until every block has met its trust target again.

Tell me what you think. I designed this for Ugarit's replicated storage backend and WOLFRAM replicated storage in ARGON, but I think it could be a useful replication control framework in other projects, too.

The only extension I'm considering is having a write priority as well as a write weighting, just as we do with reads - because that would be a better way of enforcing all writes go to a "fast local cache" backend than just giving it a weighting of 99999999 or something, but I'm not sure it's necessary and four numbers is already a lot. What do you think?

Ugarit archive mode progress (by alaric)

Ugarit's archive mode is getting along nicely. I now have importing from a manifest file that specifies properties for the import as a whole, and a list of files to import with their own properties, and basic browsing of the audit trail of an archive in the virtual file system. That includes access to the properties of an import via the virtual "properties.sexpr" file. Note also that lots of import and file properties are automatically added, such as the hostname we import from, the input path for each file, a MIME type deduced from the extension, and so on.

Below the fold is a transcript of it in use, which probably won't mean much to many people...

Read more »

Dads & Lads (by alaric)

popped up in my local news today: Lads and Dads Club to strengthen family unit.

"That sounds a bit... 'last century'", thought I. What activities might require male-only participation? Getting women pregnant probably isn't a task that fathers and sons would traditionally share, so it pretty much had to either be mustache/beard management or targetted urination, right?

But, no, it's a club for doing outdoorsy/adventurous activities. Exactly the kinds of things my wife and eldest daughter enjoy doing, as it happens (our youngest daughter, on the other hand, is scared of mud and trees). So, why the "father/son" branding?

Well, what I suspect they're trying to do is to give fathers motivation and opportunities to spend more time with their children. With the kinds of careers that make the money required to raise a family still much more accessible to men than women, a lot of kids are largely being raised by their mothers, which will indeed give kids a skewed view of the world. For a start, it'll tell them that parenting and domestic chores are for women while going afar and earning money is for men, which sucks. And it'll tell young boys that their destiny is to grow up to be a distant wage-earner, while girls are told that their destiny is to be a carer who's always hanging around the home and spending somebody else's money.

Yep... That sucks. Fathers spending more time with their kids is a good thing, and help in doing that - in telling them that they should, and that it's an OK thing for a man to do, and in giving them ready-made group activities to turn up so they don't need to organise stuff themselves, and to get them in with other similarly-minded people to exchange tips and make friends - sounds awesome to me.

However, I think it's been unfortunately tainted with gender stereotyping. That it's sons that need more time with their fathers, but daughters don't. There's an assumption that the boys need to learn MAN THINGS from their MAN DADS, while the girls are fine learning woman things from their woman mums.

But kids don't have these gender stereotypes as to what activities they should do, unless we force them onto them. And forcing gender roles onto people causes misery.

The Lads and Dads club have a rather defensive-sounding statement on their site saying that mothers and daughters are welcome too, but the rest of the site is full of statements like:

Lads and Dads Club is about creating fun and inspiring male environments. We’ve got great events for grown-up sons and their fathers, from hill walking to sky diving and weekends away. And because not all sons have a dad around, and not all dads have sons, mums and daughters are welcome too.

Boys need to be boys ... At the core of the Lads and Dads Club we are all about supporting fathers in their efforts help their children to grow and develop and have lots of fun along the way. This necessarily means that we are creating a very male oriented environment and we think that this is a good and necessary thing for Fathers, their Sons and often their Daughters too. (and, at least when I fetched it, the bottom of that page had a big ad for "ItsNotForGirls.com: Men's fashion at its finest")

So they've clearly given some thought to gender issues, but seem a bit confused, as no attempt is made to reconcile "Mums and daughters are welcome" with all the hyping of "male environments".

What's going on here? Did somebody complain about the sexism, maybe citing "discrimination", and they felt compelled to quickly wedge in the statements about accepting girls to avoid "trouble"?

You see, I think that organising outdoorsy/adventure activities for kids is great. And encouraging fathers to take a more active role in raising their children is great and particularly needed, because sexist stereotyping tells men they're supposed to be distant and leave the parenting to the mothers. But restricting this parenting to sons, and using it to reinforce the very gender role assumptions that are the root cause of the problem, isn't exactly the best way of dealing with it.

I'm worried that this thing will succeed, and in doing so reinforce the "men work all day, then go to the pub with their mates to avoid going home to their families, and then hide in their sheds all weekend" stereotypes.

And I'm worried that it'll become the target of feminist anger and be destroyed, leaving all the people behind it feeling angry that women came and destroyed their attempts to build a male-only space, leading them to become men's rights activists and try to fight the feminazi conspiracy, rather than trying to help fathers to do cool stuff with their kids (of all genders).

What would be AWESOME is if they realised that the problem they're seeing (fathers aren't getting to do much parenting, at the cost of both themselves and their kids) is a symptom of false traditional ideas of what gender means, and work to solve the problem in ways that break down those gender models, rather than reinforcing them. Now, to market it to fathers who, themselves, will have been brought up with those kinds of gender stereotypes, they'll need to be careful to make it look acceptably "manly" so that the fathers aren't turned away from it; there's a very fine line between "not offending gender stereotypes" and "promoting gender stereotypes", but it can be walked. The organisation's current description of themselves (starting with the unfortunate name...), however, is nowhere near that line. There's no reason to actually use the word "father", rather than "parent"; if the content is all written without touching on gender stereotypes of any kind, it can be "not unfatherly" without being "exclusively fatherly" (and, therefore, "not unmotherly" without being "exclusively motherly" either!).

The United Kingdom is full of "mother and toddler" groups. Even if they're called "parent and toddler" instead, they're invariably female-dominated, which reinforces itself as fathers are left feeling quite out of place at them. It would be great to fix that, and a logical place to start is by making "father and child" groups for various age groups of child, so fathers at least have a comfortable place to share fatherhood with each other. Given a chance to build their confidence in that role, we would be able to break down the barriers and migrate towards everything truly being "parent and toddler"; but as with whenever we apply "positive discrimination" to try and encourage people to do something society historically tells them they shouldn't, this needs to be handled with utmost care and sensitivity, lest it becoming divisive and discriminatory in its own right.

This can happen alongside "adventure activities for kids" groups that encourage mothers and fathers to go and do fun stuff with their sons and daughters - which would start off being dominated by fathers and sons, because of our social expectations; but there's no reason to give into that and assume it's the natural state of affairs.

We're all suffering because of gender stereotyping. There's no reason to force that onto another generation.

Recent Ugarit progress (by alaric)

I had some time to work on Ugarit yesterday, which I made good use of.

I really should have worked on raw byte-stream-level performance issues - I did a large extract recently, and it took a whole week - but, having a restricted time window, I caved in and did something fun instead; I started work on archival mode. As a pre-requisite for this, I added the facility to give tags a "type" so we can distinguish archive tags from snapshot tags - thereby preventing embarrassing accidents that end up with a tag pointing to a mixture of snapshot and archive-import objects...

(Not that I didn't think about the performance issues. I have a plan in mind to rearrange the basic bulk-block-shovelling logic to avoid any allocation whatsoever by using a small number of reusable buffers, which should also avoid the copying required when talking to compression/encryption engines written in C.)

Read more »