Replicated data (by alaric)
Well, as I mentioned before, a node keeps a copy of the timestamp it considers itself up-to-date with for each fascicle it carries a copy of. When it dies and comes back, it requests updates to that fascicle's entities since that timestamp.
Well, while a node is UPDATING, it should be sent changes it's missed in timestamp order. And while in that state, it should be updating its timestamp with the timestamps of the update messages it receives, while also keeping a note of the most recent timestamp it's seen in current write messages that it's been sent. If it dies in mid-UPDATE, it will continue to UPDATE at the point it left off, since the last timestamp it successfully updated is what it has stored. It will end up requesting new copies of changes it's actually seen while in the UPDATING state, but that's just slightly wasteful; at least it can't miss any updates. Only when the update is complete will it take the live writes it's received while updating into account and store the most recent timestamp from them on its disk.
If a node dies during a fascicle merge, it gets a little messier.
The easy answer is for it to sit and wait for the merge to be complete, while still marked as DEAD, until it's a DEAD node in the new fascicle, then proceed to the UPDATING state. It'll have its original fascicle's state up until the point at which it died, and the state of the other fascicle up to however far it had gotten with the merge operation, plus changes to the other fascicle since the merge started. So it could take the timestamp it had gotten up to in the merge and request all updates since then, duplicating some changes - again, wasteful, but better than losing any changes.
However, it's not nice to keep perfectly good nodes in the DEAD state, rather than contributing to society. If the fascicle merge is complete by the time the node comes back from the dead, the above approach will be required, but if it comes back while the merge is still in progress, it could go to UPDATING in its home fascicle, while carrying the merge on from where it left off, but merge changes up to the timestamp where it came back online, so that it gets copies of all the changes to the other fascicle it missed while offline. The merge cannot complete until all non-DEAD nodes have signalled that they're finished, so the merge will not finalise until the node claims to have finished merging - so it shouldn't signal that until its UPDATE in the home fascicle is complete, too.
And what happens if a node dies, and comes back to find itself in a different set of fascicles to what it knows about? If it's just lost some page ranges, then that's not a problem; it can just delete some data and continue. But if some more page ranges have been allocated to it, it has to get up to date copies of the affected entities before it can claim to be LIVE. Therefore it should go to UPDATING, and issue update requests for the page ranges it has from the timestamp it lost contact with the cluster onwards, but also issue requests for fresh copies of all the entities in the page ranges it is assigned doesn't have copies of. This suggests that the 'request changes between two timestamps' and 'request copies' network messages should also contain a list of page number ranges to filter the results by.
There! Now I'll mull that over in my head for a few weeks and see if I can spot any flaws.
Wed 22nd Aug 2007 11:09 am 
Subscribe