The implementation of Web applications (by alaric)
I never did implement the thing, but that's just as well, since some years later I started to come across deeper problems.
Yeah, I know the PHP programming model sucks because it doesn't really encourage you to separate logic from presentation - personally, I enforce the distinction myself through self discipline, but it'd be much better if the framework made it easier; but that's easily fixed - just have a simple page template language and put page templates in your document root, but let them reference files that contain back-end logic written in a full programming language. Sort of like my practice of having PHP files that generate no output but define lots of functions, and PHP files that output stuff but contain no more code than that required to decide what to output, which include the former PHP files; but made formal. But that's not what I'm worried about. That's a trivial detail.
What I mean is that the handling of forms still needs to be higher-level than letting the application developer spit out the HTML for a form then, when it's submitted, telling them what came in the POST or GET variables.
Now, all the nice modern Web app frameworks (Django, Ruby on Rails, etc) really fail to do all that much more than PHP. If you added an object/relational mapping library to PHP and happened to structure your PHP pages in the same way that Rails or Django do (eg, have the target of a GET or POST be a PHP script that does any processing required and sets up the results in global variables [playing the part of controller] then includes a PHP script that just outputs HTML using the contents of those variables [playing the part of the template]), then all the frameworks give you is a load of useful time-saving shortcuts for common patterns.
Which is great, but I think a lot more can be done.
Take, for example, widget libraries. Developers of desktop GUIs are usually given, by the GUI toolkit, a library of form field types; just like Web app developers, they get text fields, buttons, radio buttons, checkboxes, and so on. However, they can also usually make up their own controls (either by directly drawing to canvasses and responding to input events, or by combining existing controls: three sliders can make an RGB colour selector, for example), and package them up so they can be used as easily as any inbuilt control.
Few Web application frameworks, however, offer this functionality. They let you use the standard controls supplied by HTML, but if you want to share libraries of widgets, you have to roll your own framework for doing so; for example, provide a function to output HTML for your widget with a given initial value and field name (if it uses more than one underlying HTML form field, then that name can be used as a prefix), and a function that, given a set of HTTP POST vars and the field name returns the value of the widget.
Which isn't so bad - until you start to want funky AJAX controls.
Say you have a Web app that involves editing a list of things. You want to show the user the list as a scrollable table, with clickable rows. Clicking a row brings up a form beneath the table letting you edit that row, with Ok (submit changes), Cancel (discard changes), and Delete (zap this row) buttons. Also, there is an Add button which, if clicked, brings up a blank edit form with Add and Cancel buttons. Plus, attempting to Add a new row or Ok changes to a row may result in a validation failure.
Now, in a desktop GUI, you'd probably create (if there wasn't one already) a ListEditor widget composed of a scrollable view control containing a table control, plus an Add button control, with some magic to let the user of the ListEditor provide their own row display code and row edit form layout.
In the Web app, we want sexy AJAX while not penalising people who can't run Javascript, so we need all those clickable rows and buttons have javascript event handlers attached that cause AJAXy actions to happen, but if the JS is disabled, the links still click and the buttons still submit the form, allowing the server to process the action and send back a new page.
This is a headache to implement. Even more so if you want it to be a reusable, generic, list editor control that can easily be dropped into place in applications.
However, there is a way to do it; again, if we change the structure of our applications somewhat.
Take a look at this page. It's a very basic Web app that does a few basic arithmetic operations.
Take a look at the source. All of the included files can be seen here - .phps symlinks show you the source rather than invoking the PHP.
The general structure is that we use AHAH rather than AJAX per se; the PHP code to generate a part of the page can then be triggered from the Javascript and send back its result via HTTP or, if JS is not enabled and the fallback behaviour of the button is invoked (posting to the server), it is invoked server-side and its result spliced directly into the response.
This trick is then used to build up a widget library. All of the basic HTML form fields are embedded in widget classes, which provide for automatically keeping the state of the field when the page is submitted back to the server and the same form redisplayed; and in the case of submit buttons, also invoking server-side behaviour (defined in test-logic.php in this case).
An interesting one is the "Sheet" widget, used for an options popup in the example. This widget actually outputs four bits of HTML into the page; one is a button that, when clicked, brings the sheet up. The next is a hidden div wrapper that goes in front of the content for the sheet, then another is the closing element of the div wrapper; and the last is a button to go inside the sheet content, which will close it.
The two buttons are generated with attached JS actions that just alter the DOM to show and hide the sheet, but they also have names so that if JS is disabled and the form is actually submitted, they toggle the value of a hidden form field which also controls the visibility of the sheet - which is used by the code that generates the start of the div wrapper to decide whether to make the sheet hidden or visible by default.
Voila - the sheet works with JS enabled or disabled, and the actual application page doesn't need to care. But to do this, it relies on all the other widgets in the page transparently preserving their values if the form is posted.
And that's the interesting part of this widget system; any given form can be assembled from any number of entirely independent widgets. They don't need to know about each other, since they have a clearly defined responsibility to, if they've not been touched, recreate themselves identically after a form post submission.
But my proof-of-concept isn't complete, because the structure of the code mandates a static widget structure. What about our list editor? Each list entry might be composed from widgets, and the number of them is dynamic. The best solution for this I can think of is to rely on some level of object/relational mapping logic existing somewhere, and to provide the list widget with two pieces of code: one that queries the database to get a list of object IDs that should appear in the initial state of the list when first viewed, and one that maps an object ID to the object's value, which can then be passed to each instance of the "list row" widget supplied to it.
It can then, when the page is viewed and the list editor widget is instantiated, ask the database for all the row IDs, then dynamically create the required number of instances of the list row widget, named by taking the name of the list widget, and appending the object ID.
If the form is then posted back, it'll need to repeat the process, so that if (for example) a button in a list row was pressed, when the widgets are all asked if they'd like to claim the form submission, that list row exists to claim the press (unless the object the list row represents has disappeared from the database - in which case the event will go unclaimed and nothing will happen; perhaps it might be nice to figure out a way to let the list editor widget realise the event was something to do with one of its children that no longer exists and offer an explanation to the user).
It can also be given the name of a custom list row editor widget, which it may or may not instantiate to provide the add/edit form, given appropriate initial state.
Sun 17th Dec 2006 2:37 am 
Subscribe
By Ben, Sun 17th Dec 2006 @ 10:25 am
Continuations have already been used in web app frameworks. This article...
http://www-128.ibm.com/developerworks/java/library/j-cb03216/index.html
... has a nice list of frameworks at the bottom. I think Jetty is the most famous continuation based web app framework.
I disagree that web app frameworks have got things the wrong way round. You are writing an application which happens to be delivered over a web browser, not a web site which happens to have lots of code.
I suspect your view is formed by the nature of a lot of your work, where a bit of code has to be added to a bunch of marketting visuals and text. This is not what the frameworks were designed for -- the assumption was that there would be more code than fluff.
By alaric, Sun 17th Dec 2006 @ 1:45 pm
The way Jetty have added continuations, however, is a bit clunky to say the least... when your code hits the suspend point, IIRC, it throws a special exception to terminate the handler but keep the HTTP response pending; when the event comes in to resume the request, the handler is invoked again but is informed that it is dealing with a request already in progress. As I understand it, it was designed for the particular case of AJAX event getters, those GET requests which return a pending event if there is one, or else wait without replying until one comes. A central event dispatcher can find all requests 'waiting' on that event, put it in their local queues, and awaken them.
As for the code vs. fluff dispute - well, the particular case I had in mind was UpMyStreet.com.
UMS (for those who don't know it) was (and kind of still is, but it's nothing like it used to be) a site where one could enter a location - a postcode, part of a postcode (just SW7, for example) or a town name or a region name (London or Camden, for example), and see lots of information about that area: council tax rates, crime rates, property prices, the socio-economic grouping used by insurance companies and credit agencies, etc.
Originally, this was implemented with a common include file which defined the Location class, an instance of which was fed the location string supplied in the URL and then proceeded to compute assorted general bits of metadata. Each information page then contained code which would use the tools in the Location class to extract the appropriate data from the database and slot the results into the HTML.
However, the structure of the site was changing towards having more, smaller, pages. And in order to deal with the fact that the programmers were having a lot of simple work to do with rearranging the pages for design reasons, we moved to an architecture where every page included a common header file that set up the Location class and automatically created an instance of it based upon the URL parameters, then neatly wrapped the data-getters into a suite of easy to use library functions. We then taught the HTML designers enough PHP to:
A dictionary of magic words that could be used to extract and format information. For example, one might want to get the name of the local council covering the location, and the band D tax rate. To do this, one needs to consult the council module (which gets a whole lot of information, since it's all in the same row of the table) and then extract the desired data, and format it appropriately, something like:
< ?php $council = load_council (); ?> ...Your local council is < ?php echo $council->name ?> and the band D tax rate is £< ?php number($council->tax_d, 2) ?> per annum...
We had a few simple functions like
numberwhich would format a number nicely with commas and the given number of decimal places. Theload_*functions would access the global location instance, and spit out an object with fields that could easily be accessed (with strings already html entity escaped, for instance) from template code.Having done this, the programmers didn't need to care what went on what page, generally. The HTML designers could make any page extract whatever data they wanted, and display it as they wanted. They could add and remove pages willy-nilly, which under the controller-based models of rails et al would need a programmer.
With a decent object store, many of the 'data getter functions' wouldn't need writing, either, if the HTML developer was just given read access to the store - most of the UMS ones really just encapsulated an SQL query, appropriately setting it up then processing the results.
Perhaps one area in which controller-based structure is the way to go is when you have form posts that do something; the above code is all idempotent data extraction. However, I have a theory that having HTTP POSTs return content is a bad design pattern. For a start, it results in a page that has no URL - if you have a page that came from a POST in your browser, you can't bookmark the URL and expect to get that page back if you go back to the URL. What I think should be done instead is to have POSTs go to handlers that are entirely programmer-written code to perform the action, which then return with a redirect to a URL that, when GETed, shows the appropriate result page.
In that case, the POST handler might be considered as a kind of controller (and it's far closer to the original meaning of 'controller' in the MVC model, IMHO). But it'd only be in the loop for the particular case of a POST form, and even then, only loosely bound (by returning a URL to GET) to the resulting view.
Now, what are most sites like? I think the average public Web site is mainly pages that display information, which I think are most easily handled by giving the HTML developers a template language that can access a library of data accessors rather than bothering programmers with. The POSTs tend to be fewer and further between; adding something to your basket, submitting a message, checking out, or the aforementioned awkward case of a 'dialogue' to perform some complex interactive process. All of which, I surmise, can be neatly handled within an overall site structure that remains in the hands of the HTML designers, with the programmers providing either simple POST handler 'controllers' that redirect to a results page URL afterwards, or POSTing or GET-linking to the start pages of dialogue processes.
By alaric, Sun 17th Dec 2006 @ 5:24 pm
I think Seaside (from Ben's link) is the closest to how I'd handle my 'dialogues' - but not normal idempotent view pages with links between them. They're best done statelessly. I'd like seamless integration between stateful dialogues, view pages with GET params, simple POST forms with controllers that return redirects, AJAXy edit pages with non-JS fallback POST handling, and any other models that are applicable to parts of a site. In other words, a general architecture that brings them all together; and I think this can be done with Apache modules that handle 'special files' in a public_html directory, be those files page templates that can access library modules, dialogue 'scripts', 'controller scripts' that accept a POST and return a redirect (or an error...), static resources, etc.
The one thing that sucks about that, of course, is that your URLs expose your technology: how many URLs are of the form "view.php?id=123"? I'd tweak the Apache filename matching slightly so that if the file named in the URL does not exist, but a file with the same name but with an extension appended matching a script or template handler DOES exist, then that file is used. So if one had a template language called "bob" one would store HTML templates in files ending ".html.bob" and access them with URLs ending in ".html". Or in files ending with ".bob" and access them with URLs with no extension on. These decisions should be cached by the Web server, to avoid having to do lots of stat()ing on every request.
Since POST controller scripts return a redirect, they could use absolutely any template language you had installed on your system to generate their result, which nicely decouples things. And the sending of pages for the dialogue system could be implemented by creating something like an Apache virtual request to generate the output page, nominating a page template in any language and just passing make-believe GET parameters to it.
That way, you'd have a toolkit of different structures for different parts of a site that can be connected together freely and openly.
I'd wondered about serialising continuations and putting them in hidden form fields rather than burdening the server with sleeping processes, which would make the server properly stateless (albeit opening some cans of worms regarding state external to the application itself, so not encapsulated in the continuation - external databases, for example) - but I felt that a lot of continuations could get rather large, putting a big strain on the network and client, and there's be mobility issues with changing class versions etc. on the server. Seaside, to my surprise, stores the state of the handler when it's suspended and gives the client an ID referring to that state, but when it's resumed again keeps the handler state while starting off with a copy of it, so one can use the Back button to return to previous states! Although this means it acheives that without sending huge continuations across the network, it does mean that every time you send an HTTP request, a new continuation is created on the server and kept until it times out, which will take up an awful lot of RAM 🙁 My own research into using continuations, rather than just suspending a process on the server, revolved around lightweight continuations; a lot of the environment in each continuation will be identical (eg, the innards of the request handling parts of the framework will be at the top of every stack), so representing continuations as "diffs" against a standard template, making sure the dispatchy bits of the framework use tail calls (which don't take up stack space, being a JUMP rather than a CALL), or having the framework start off the user application handler in a 'clean thread' with a very minimal stack, so the framework's state is not part of the continuation.
By alaric, Sun 17th Dec 2006 @ 5:35 pm
Ooh, a vaguely related point I might as well stick in here (just showing that this is a 'stream of ideas' post rather than one I've carefully thought out in advance) is that pages (even the idempotent ones) should really have a header in the template or script or whatever that defines the parameters, and gives them types.
This has a few uses.
This will all sound familiar to Ben - he had lovely declared page parameters in his very own Web app framework, and they worked well.
By Ben, Sun 17th Dec 2006 @ 8:48 pm
Yes, URLs need need explicit typed parameters. I dislike RoR's style; it starts off well with /controller/action/id but then after that it just uses key=value parameters. And nothing is typed or parsed automatically for you. I may end up rewriting the 'routing' to do something about it.
On continuation, it doesn't necessarily have to be implemented as a proper continuation. Another model is a state machine with data. This could be handled by some funky pre-processor or compiler support.
By alaric, Wed 24th Jan 2007 @ 11:48 am
I've since found out just how terribly badly most web application frameworks handle HTTP cache control headers, so have written a post about that which gives more ideas on what a decent framework needs: http://snell-pym.org.uk/archives/2007/01/24/http-caching/