Merging BitTorrent and HTTP (by alaric)
I've been kicking an idea around for a while now, so I thought I'd blog it, rather than just sit on it then feel frustrated when somebody else has it and gets RICH and FAMOUS and POPULAR...
Basically, BitTorrent makes publishing large files on the Web much less of a burden on the server than HTTP. If I put a 10MB file up on an HTTP server and give out the URL, everyone who fetches the file will transfer 10MB from my server. The same 10MB, over and over again.
If, however, I run a BitTorrent seed on my 10MB file, connecting to a tracker server, and give people the .torrent file describing my file and naming the tracker, then people with BitTorrent clients can connect to the tracker and find a list of connected clients with parts of that file (initially, just my seed client), and start fetching chunks of the file from them. As soon as a few people are downloading my file at once, they can actually start sharing chunks between themselves - my seed sends a chunk to one client, then my seed and that client are both available to send chunks to more clients. This reduces the load on my server a LOT, and thus reduces the cost of publishing large files.
Lovely stuff.
However, it's complex. Rather than dump a file in a directory on my web server and give out the URL, I have to run a tracker server, create a .torrent file, run a seed client, and distribute the .torrent file (perhaps by copying it to a directory on a web server and giving out the resulting URL).
It strikes me that one could probably write an extension to HTTP, implemented by an Apache module, that:
- If a GET request for a file comes in with a special header stating that the client supports it, then engaging this special behaviour. Otherwise, sending the file as normal. The server may be configured to send the file as normal if its size is below a certain limit, too.
- Have a tracker built into the server. I think the tracker protocol is HTTP anyway?
- If one does not already exist, automatically generating a
.torrentfor the file, naming itself as the server, and sending that as the response body
Then clients/web browsers that support it could then automatically fetch static files using BitTorrent, from servers that support it, while still maintaining perfect backwards compatibility between mixtures of old and new servers and clients, and without needing any extra admin effort (beyond perhaps installing and enabling the Apache module).
As far as I can tell, it'd be better than Web Seeding.
Sun 12th Nov 2006 12:45 am 
Subscribe