The following post is intended for a very technical audience. Consultant supervision is advised.
I wrote a thingee that lets you mount “The Web” as a filesystem. So far it works under Mac OS X, but I intend to port it to Linux. It uses MacFUSE for the filesystem interface – makes it much easier to write without kernel muckery. It’s all written in C. It’s available on GitHub. It’s called CREST fs – Cached REST, or Cached Representational State Transfer.
The thinking is that The Web, as a whole, is a set of resources, which I should be able to access like files from a file system. So as opposed to doing something like trying to curl http://samplesamp.sa/apage/something, then trying to run the code that lives on that page, you could instead mount the web under some directory, then cd to samplesamp.sa and execute apage/something directly. The first access to ‘something‘ would require it be fetched from across the internet, but with a caching scheme built in, subsequent accesses should be from the local hard-drive caching system.
I thought it would be clever to be able to mount it under a folder called /http:/ – so you could say:
ls -al /http:/www.google.com/someurl/something
See? I think that’s clever. You could probably put two slashes there and it wouldnt’ mess up anything. And if you were sitting in the / directory you could skip the first slash. But that’s starting to get too clever just for cleverness’s sake.
There’s stuff like that Out There already, but I wanted to build something that was extremely aggressive about caching, and very primitive (low-level, usable in Early boot environments). I expect it to notbe remotely coherent, but I do want it to be fast. So far, so good.
Sidenote: it’s my first Git project. Git is nice. Hosting it on GitHub is interesting, too, but less interesting than the fact that it’s on Git.
Writing code in C is painful. Allocating memory is not fun. Troubleshooting subtle memory leaks is not fun. Doing your own string manipulation by hand is not fun. But there is a certain feeling you get from being this close to the bare metal of the hardware…that’s really pretty exciting.
This is the second time I’ve written this – the first version was lost in the Great Hard Drive crash of ’08. Writing all this low-level crap is not that cool, but writing it for the second time is even less cool.
The intent behind all of this is to tie it in with Braydix somehow – to allow you to boot a “minimal” Braydix image from CD or USB key, and have it pull the rest from Teh Intarwubs. I have a new client who does a lot of work within Amazon’s EC2 environment, so I’ve had to study how that works. I think this FS might be interesting for that, too. As soon as I can find an excuse to put something up over there, I definitely will mess around with that too. Once I’ve done that, just think of it – they’ll be Braydix both client and server versions.