Too many projects (by alaric)

I have a curse: I'm too inventive.

Yes, this has many benefits; it's certainly useful in my career. But when my ability to come up with ideas far exceeds my available time to execute them, it's a bit sad.

I've met people who seem to think that ideas are rare things. They guard them jealously, seek patent protection, and hold onto them long after they're obsolete; if one of their ideas turns up somewhere out there in the wild, they are sure somebody has stolen it from them.

But I think that ideas aren't that precious. I'm glad when I see an idea I've had realised, since it saves me the effort of realising it. While thinking about Bayesian spam filtering I realised that it failed to take the ordering of words into account, and wondered if it would be possible to build a Markov chain model of spam and another of ham and then measure a message against those two models to see which they fitted best; so when I discovered CRM114 I was happy - and even happier when I found out that Markov-based spam filtering is in Alex Shinn's hato MTA. If you're having more ideas than you can implement, then it's great if somebody else implements them; it means you can focus your limited implementation efforts on one less idea, or use their implementation as a basis to build your idea (if it's slightly different) rather than starting from scratch.

So, without further ado, here's my current Idea List:

1. Improve my welding skills by making a "hexoctagon", a garden ornament about a metre across made of inch-wide strips of steel organised into three octagons joined about a common axis of intersection and then joined with further strips, so that it's a hexagon from above and an octagon from the side. Cheap material requirements, easy cutting to shape, and then lots of welds that should be within my ability, so good practice.
2. Rebuild my furnace, and get back into aluminium casting.
3. Work on my wearable-computer projects. I've been experimenting with casting electronic circuits into blocks of transparent resin, which makes them durable and waterproof while not being as bulky as a box, an also allows one to see LEDs and LCDs embedded in the resin with them. The interesting part is getting connectors and tactile inputs like pushbuttons to the surface, but I have some ideas I've been experimenting with in those areas. I want to develop these to build a head-mounted display with an LED torch built in, a wrist-mounted display with a chord keyer inside, and a modular central unit to go in an inside pocket with central processing, storage, power supply, and interfaces in. Wearable computers are fun.
4. Extend the structured cabling around my home into adjacent buildings. Challenges include thick stone walls and a driveway to get across. I'll buy a metre-long 2cm-thick drill bit when I have some money, as my 30cm one didn't make it, and I'm researching stringing a catenary over the driveway.
5. Finish implementing my SFTP client library for Chicken Scheme, then write an SFTP storage backend for Ugarit using it. Also, various other improvements to Ugarit, such as caching the hashes of files by their mtime, so we can easily tell if a file has not been changed without re-hashing it, for faster snapshots.
6. Tinker with hato, adding local delivery to virtual users defined in a database, and an IMAP server that can serve from both real and virtual users, so I don't need to create full UNIX users for every IMAP mailbox. Perhaps adding LMTP support, too.
7. Tinker with Chicken Scheme internals; in particular, I want to improve the blob support, to eliminate unnecessary copying between the heap and normal malloc()ed memory, and within the heap between blob and SRFI-4 interfaces to the same region of memory.
8. Ongoing programming language research, extending the work of my final year project at University on metaprogramming, with some refinements I have in mind to support hygienic macros.
9. Ongoing development of ARGON, which is a vast sea of subprojects in itself
10. Work out the economics of distribution and storage of liquid nitrogen. Is it practical to use LN2 as a means of cooling buildings, rather than refrigeration-based air conditioning units that suffer from inefficiencies due to their small scale, and the fact that they pump the heat just outside the building, where it can easily leak back in again? Heat pumps are particularly bad in built up areas where everyone's doing it. LN2-based aircons could even be portable pocket-sized devices, very welcome to motorcyclists in slow traffic on hot days, and commuters.
11. Make test strips for common dietary intolerances. As a vegetarian, I often wonder if a given dish at a buffet is "safe" or not, and it's worse for people with serious allergies. Is it possible to make cheap and reliable test strips that change colour in the presence of (eg) animal fats, or whatever it is in peanuts that kills people? The colouring in the food itself may prove an obstacle, but diluting it in water and having a control on the strip to compare the colour against should be able to mitigate that, I suspect.
12. Make a better satnav system. I don't really like the traditional "At the roundabout, turn left" systems. I much prefer looking at a map of my route and seeing what I'm driving past, and knowing where I am in relation to major landmarks. So I'd really like to just see a big map of my entire journey, and a small map of my local environment, both with a marker on showing where I am. But let's generalise this a bit. How about making a generic map display system, able to take a base map and then apply any number of layers, each provided by a software module, possibly over a network, with streaming updates. Then the GPS can just produce a layer with a single point that moves about, while a map server just produces a static map, or streams map data from OpenStreetMap on demand. A database of places is a layer that exposes a set of points. A route planning system can provide a layer that shows a fat blue line over your route. The UI should let you lock the display to a given map feature, such as the moving point that comes from the GPS unit. This is a bit complex for a simple satnav unit, but when you're managing a fleet of vehicles, you can pull in a map layer plus the GPS layer from every vehicle (via GPRS etc) and then see the all on the same map, without needing any special extra software. It's a construction kit for map-based information display. But why stop there? Generalise a little further. This map information is all just a source of data sets defining points, lines, and polygons in lat/long space, with streaming and real-time updates, and a viewer that knows how to display these things in lat/long space. Why not generalise the coordinate system to an arbitrary set of time, space, and other unit axes? The system can know about special types of axis - such as time and spherical coordinates - and then have generic mappings of the other ones into spacial axes, and allow analysis of arbitrary data sets, overlapping them if they have axes in common. So if you have a data set that's the spread of pollution with time, you can overlay the lat/long axes onto your map (or an augmented reality view), and have a slider to choose a position on the time axis; or pull it into 3D by adding the time axis and look at the spread of the pollution as a cone. Your email history can be a set of points aligned on a time axis, and your appointments can be points and regions on the time axis, sometimes with lat/long added (if you took a GPS reading of your position in the past, or if you know where you're supposed to be in future), that you can compare to the spread of pollution (did it coincide with you starting to get sick?), and plot on the same map you use for navigation (so you can work out the best route to get to all the places you need to get to in future), or just fold away the space axes and look at a timeline of your life that you can zoom into. Many applications can be fulfilled in one integrated "data viewer", and many network effect benefits reaped by having them integrated seamlessly.
13. Audio jukebox appliance. At its simplest, a small box with an Ethernet port, a power inlet, and an audio out port. It finds your music from one or more network servers, or has an internal hard disk you can put music on, or combines both. It will play songs from a queue, and if the queue is empty, will either be silent or will randomly pick songs that match a search query. But that's the boring part. The fun part is that it also has a message queue, and supports a simple network protocol to send it messages. A message causes the music to pause with a quick fade out, then a jingle (depending on the message type) played, then the message text to be read out via a text-to-speech engine such as Festival. When there's no messages in the queue, the music resumes. For extra coolness, add the ability to drive remote slave units over the network; the units are organised into a logical tree, and by default, every node inherits music and messages from its parent. So you set the global playlist at the root node, then on a child node, you can queue some songs and have that child (and its subtree, if there is one) then play different music, reverting to the global music (with a fade) when it ends. And you can send a message to any node in the tree, and that message will be text-to-speeched from that node and all its children. This is something I want about the house, and in my van (see next point), and would be great for secret underground bases, which definitely need computer voices making announcements.
14. Jazz up my van with a vehicle computer. An MP3 jukebox and navigation system. And a separate battery, so running them all (and the inverter) can't make the engine-starting battery go flat, but when the engine's running the auxiliary battery charges from the alternator.
15. Virtual networking. See blog post.
16. Write a "Practical Programming in Scheme" book. Base it on Chicken Scheme, as it's aimed at real-world tasks rather than education. Cover all the stuff that rocks in Scheme, but that basic introductions don't cover as they're trying to teach programming to new programmers: parameters, coroutines, fold, really making the best use of closures, that sort of thing.
17. Finish up some novels I have in various (early) stages of development.
18. Design CPUs. I think there's some fun to be had with unusual architectures. In particular, I want to develop tiny MISC-based microcontrollers, and I also have some interesting ideas on how to build a highly asynchronous move-based architecture that should make good use of instruction-level parallelism, while having a very simple control unit that lets you fit lots of them on a chip, sharing execution units between them.
19. Build a giant distributed (around the house and grounds) replicated storage array for storing my backups, as a single giant Ugarit archive, using a distributed-archive backend for Ugarit that I've designed, which will allow fine-grained control of a replication policy in a very fail-safe fault-tolerant way, yet being simple to implement.
20. Implement my old protocol agnostic middleware idea. "Middleware" is a bit of a dirty word these days, so perhaps "framework" would be better. I was excited that somebody seemed to have gone ahead and done this, but it's in Python 3 which is a bit limiting; I could put together an implementation in Scheme without too much fuss by layering existing interfaces to protocols such as HTTP, organising them into a common request-routing infrastructure.
21. Re-copyedit some role playing games (of the pencil-and-paper rather than computer variety) I wrote as a teenager, and set up http://www.point-defect.co.uk/ as my games publishing arm.
22. Write a Web-based MMORPG set in a procedurally generated universe I have designed, with certain interesting properties. Ideally turn it into a profitable online business, through advertising and subscriptions.

I think it's important not to try and have any illusion that I'll finish any of my projects. It's all too easy to sit and think about how cool each one would be, and to be sure I'm going to finish it because it would be awesome, and talk enthusiastically about how I'm going to do X, Y, or Z. When I was younger, I fell into this trap, and then felt bad about myself when I didn't.

Now I remind myself: these are my hobbies. I say I would like to do X, Y, and Z. But I make no promises, and set no deadlines. When I get some free time, I will work on whichever one I feel like, so progress is hard to predict.

I do feel sad, though, that I have interests in so many fields. I am jealous of people who have just one passion, and devote all their energy to it, and therefore take it a long way. Some days I'm not in the mood to mess with computers. Some days I'm not in the mood to mess with metal. It varies.

20091002: Updated with some more detail on some of the ideas.

Why are networks so hard to build? (by alaric)

One of my sidelines is network management.

Often, the problem is this: you have a bunch of sites, each with zero or more external connections out to the wider Internet (or to people who you provide an Internet connection to), and each with zero or more computers that need some level of network connection (be they servers or workstations). Each computer needs to be able to talk to some subset of the other computers, and maybe able to talk to computers out on the Internet or some other external network, and maybe computers on the Internet or some other external network are able to talk to it. And computers may be on public IP addresses, or on a private IP address; in the latter case, if it can talk to other external networks there needs to have a public IP address (possibly shared with others) that its connections are NATed from, and if incoming connections are allowed, there must be a public IP to which those connections are sent to be "forwarded" into the private IP. We can think of those NAT/forwarding public IPs as "virtual IPs", which don't correspond to a physical computer, but seem to by way of some form of port/address translation.

Also, each computer or external network connection needs some level of reliability. Some have low requirements, and we can happily tolerate perhaps up to a day of outage per year; that's mere 99.7% uptime. The fabled "five nines uptime", 99.999%, equates to a maximum of about 30 minutes of downtime a year. And that downtime isn't just used up by equipment failures; if your network's requirements grow and you need to upgrade things to provide more capacity, you might need some downtime to replace and reconfigure things.

In other words, the problem domain is already complex. But the fun's just starting.

Read more »

R7RS (by alaric)

There has been recent discussion on r6rs-discuss about the r7rs draft charters, most of it arguing from various camps.

I want a Scheme that lets me apply advanced programming language techniques - lightweight Higher-order functions and Hygienic macros rather than Boilerplate code, Continuations rather than a fixed set of predefined Control flow mechanisms, symbols rather than Enumerated types, Functional programming rather than getting tangled with too much state, dynamically-scoped parameters rather than God objects - to my day-to-day tasks. I'm a professional programmer; for a living, I've written code in Java, C, C++, PHP, Perl, Python, Ruby, SQL, AWK, shell and JavaScript, and I'd love to have been able to use Scheme for all of the above. I'm limited more by the usual commercial pressure than by any technical issues with Scheme or the qualities of my favourite implementation, Chicken, so my wishes for R7RS are relatively minor in terms of changing the semantics of the language. What I really want is a Scheme report that will unit the Scheme community, so we can continue to have a wide array of innovative implementations that all have their own strengths and weaknesses - but with much better portability of libraries between them, so they really do start to feel like one language with multiple implementations rather than separate languages.

So I feel that things like module systems and access to networking needs to be standardised, so each implementation doesn't gratuitously have their own syntax for doing the same thing. But these things need to be optional, so implementations are not constrained to be large in order to earn the name "R7RS Scheme".

So I thought I'd step up and propose a solution.

Read more »

Why C++ is not my favourite language (by alaric)

Yesterday, I had a discussion with somebody who's a big fan of C++. He spoke of C++ as letting you write high-level code (with smart pointers, exceptions, destructors and templates conspiring to smartly remove lots of the normal drudgework of C programming), while still being able to perform low-level memory access like C if you want to, and getting highly optimal code. He admits that C++ is complex and takes a lot of time to master, but that you can produce highly elegant and compact code once you've mastered it.

But I think this is broken.

Read more »

n2n revisited (by alaric)

I have spoken before about n2n, the peer-to-peer VPN tool that makes it easy to create efficient virtual networks.

Normal VPN products are really more of a "virtual private cable" than a "virtual private network" - they just establish a point-to-point link over the Internet, requiring a login to set it up and encrypting the traffic. This means you can have a virtual connection to a real private network somewhere; and if a few people connect into that network via VPN links, then there really is a virtual private network between you all, but all going through a central point where all your links meet.

While with n2n, everyone connects to a shared "supernode" that keeps a list of who is connected to the VPN, and from where; then when you want to connect to somebody else, you use the list from the supernode to establish a direct encrypted connection between yourself and them, rather than going through any central point. So it's an actual virtual network out of the box. You can even have more than one supernode running, so that any one can fail; all the supernode does is to provide the directory service.

Also, you don't need to maintain a database of user logins; a supernode can carry any number of virtual networks. When you connect to the supernode, you just tell it the name of the community you want to join, and it will share your connection details with anybody else in the same community - you can make communities up on the fly rather than needing to maintain a central list. Access control is handled by the simple fact that you need to know the correct encryption key for the community you want to join, or your messages will be received garbled by everyone else, and ignored.

Anyway, for a long time, I wanted to get into n2n, but I couldn't as it didn't compile out of the box on NetBSD; but a desire for a better VPN solution at work has led to me getting it working. It wasn't that much work, in the end, as the existing FreeBSD support already had a BSD approach to things.

n2n is distributed via Subversion, so they don't have version tarballs - this is a problem for my NetBSD port. So I decided to mirror it into git with git svn, then forked it as "Kitten n2n", made my NetBSD port, tagged a release, pushed it to github, uploaded a tarball from that tag, and then made a NetBSD package of net/kitten-n2n.

I'll tinker with it for a few more days, then I'll submit it to the NetBSD folks for consideration.

I'll keep pulling in from the official n2n Subversion repo, to pull down patches, and I'll see if they'd like my patches pushed up - as well as NetBSD support, there's a few things I'd like to fix as well (I've spotted passing an integer through a void* by casting, which is slightly dodgy practice and produces warnings on my 64-bit machine, but is easily fixed by passing a pointer to a heap-allocated copy of the integer!)