Lightdesktop tweaks

The console font wasn’t fixed-width so using the console was driving me crazy. Fixed. I changed the filesystem to point to the new domain (big pain in the ass). Tweaked the installer and filesystem so the /boot directory is fully under the control of crestfs. New parallel version of crestfs. Fewer pauses, much good. Added make and gcc and lots of stuff so you can now compile things (still not self-hosting though). Put in an ACPI daemon so you can close your laptop lid and the system might go to sleep (doesn’t work perfectly yet). /etc/ is the next directory to get taken over by crestfs, but will be a bit of a challenge because some things like to write in there, and there are a couple of very odd symlinks that point to /proc or /tmp, and crestfs won’t let you make symlinks like that.

Ran into some huge disappointments with davfs2 though – write performance isn’t very good, It won’t let you connect if you don’t have an internet connection, and you apparently can’t make symlinks, and all of that really really sucks. I’m going to replace it with something that I figure I will eventually merge back into crestfs. I figured I was going to have to do this eventually, but it came sooner than I had hoped. I already have code for doing HTTP GET requests, and directory listings and so on – basically the ‘read’ side of the equation – so I don’t think it will be too horrible to get the write side going with PUT and POST (what I’ll be using for symlinks) and DELETE…

Braydix is now LightDesktop

So I have new downloads of the Braydix thing – I’ve been able to shave boot times way down with the new 2.6.30.x series kernel. I’ve also simplified and stripped down the bootstrap system to help speed the boot times further.

While I was tearing everything up and smooshing it back together, I also took the opportunity to start renaming it to the name I think I’ll be using going forward: LightDesktop. I made a website too (well, I repurposed a botched previous website I made at least…). I actually enjoyed making the text on the website because I went with a far more informal tone for fellow nerds – the only ones likely to ever look at or use the thing.

I’m still unhappy with how it works on my crappy little Asus netbook, but it’s pretty solid on Craptop (my Toshiba). I think there’s something funny with the wireless drivers for the ath5k, but I’m too exhausted to look into it much.

Downloads are up: ISO and FAT fs image for USB memory sticks.

OS X Server Permissions Problems – Fix

I’ve seen plenty of people complaining about similar things – in 10.5.3 or so, the help files for OS X Server Admin don’t match what you see in the control panels. Here’s how to make it so that files are shared within your group by default when you save them on the server in 10.5.3 or greater (I was using 10.5.6). (I think this applies only to AFP):

Go to Server Administrator. Click on the name of the server in the left-hand side (not any particular service, just the name of the server). Click on (Shit, I’m doing this from memory…) Uh, I think it’s “shares?” And then click on your share in question. the bottom half of the pane should have some little tab headers you can click, I think one says Permissions? Add an ACL that permits Read & Write for your group-in-question, and forces inheritance for all contained files and folders. Hit the Save button on the bottom. THEN hit the little gear icon thingee near the bottom leftish and do ‘propagate permissions’. The last step is to make all contained folders have that new ACL, the first step is so that all future created files and folders have the read-write permissions set OK. The ordering of those last two steps probably doesn’t matter.

It’s not particularly complicated, but it’s not intuitive, and it’s not what the help files were saying. Nor what people in various forums were saying. I’m doing my trick of documenting it here so I don’t have to look it up next time šŸ˜›

Braydix…ease of use?!?!!?

My Macbook pro went in the shop again, and instead of spending $100 to rent another one, I spent $300 to buy a cheap, crappy Toshiba laptop at BestBuy. It was not a fun experience, but the laptop worked as well as one could expect. Vista is actually as bad as they say, but mostly only in the networking – of which I do alot, so I am a bit biased.

So once I got my craptop what would be the first thing that I would do! But try to install Braydix. Alas, ’twas not so easy…

First off, the CDROM detection code wasn’t being ‘patient’ enough to let the drive spin up. I worked around this by manually walking through the boot code and typing it line-by-line…all 30 or 40 lines of it. Not fun. It would then crash in some other spectacular manner, later on. But I did notice the terminal window coming up very quickly šŸ™‚

So I knew I had some real work to do once I got my Macbook back (upon which my development environment for Braydix lives).

First, I made the CDROM detection more patient – well, instead of making it patient I made it insistent. It keeps looping around and around until it finds what it needs to boot from. Which may be forever, but if so, too bad. Poor computer. Turns out that Craptop would end up finding the CD around the third pass or so through the loop. OK, fine.

Having burnt around…oh, 2 or 3 CD’s for the purposes of booting from, I was now thinking that it might be nice to boot from a handy-dandy USB memory stick I have hanging around, so I don’t keep wasting CD’s. Especially since I have the image around 32 MB, with some crunching and caution, I bet I could get it to fit on a 32MB USB memory stick. So that required some rejiggering, and I made my development environment better in the process – so everybody wins. Now we can boot off of USB, and Boot from FAT filesystems. (edit – just found some stuff to yank – 31MB FTW…edit 2 – oh, and I had to add more stuff, so now 34 FTL šŸ™ )

In the course of doing my boottime diagnostic, I couldn’t remember what the various boot functions I had created were for. I had to go read the config file to figure out. That sucked, so I also made a little menu (only shows up if you hold down Shift while the system is booting. This is diagnostic Wizard stuff, not even power-user stuff for the most part).

Two important things – this boots to a terminal window, not the browser. This is a placeholder for eventually making some kind of launcher-thing.

To finally get the craptop to actually work with this stuff for WiFi, I had to dig around to figure out the WiFi story. Apparently, it’s WiFi chip is a USB based one…whoops! So I had to add all kinds of terrible and crazy USB device detection, etc, etc. Not at all fun.

I rewrote a huge chunk of CREST-fs, the internet filesystem upon which the bulk of the rootdisk lives. It performs much better and is more aggressive about pre-fetching things that will help it later on. I enjoyed deleting lots and lots of complicated code, best feeling as a programmer, when you’ve out-eleganted yourself at something.

I created a ‘config’ directory scheme that persists across reboots (living on your hard disk, actually) for things like WiFi passwords. Who wants to keep typing those in?! I eventually imagine that I might store video display resolutions and other such ‘little, trivial, machine-specific’ things in there. Maybe the root URI of your home directory (*NOT* your password! Too dangerous!)

I think I’ve come up with a new name which I’m not going to share yet, but I’m still simmering on.

ISO: Infinix4.iso

FAT filesystem…blob thing: fatimage.fat.

Notes:

I don’t know an easy way to make a USB bootable flash disk. This is the method that works for me, on a mac. I attach my USB-key thingamadojie. I go into Disk Utility. I tell it to unmount (Not eject!) the partition. I go into terminal and say dd if=whatevermydiskimgis of=/dev/diskBLAHsBLAH (My Mac chose /dev/disk1s1, but Disk Utility should tell you for certain). For some freaky reason, even though the partition sizes do not match, this seems to kinda work. Don’t ask me. Your USB key must have an MBR partition scheme (pretty much standard), ‘normal’ MBR boot code (also kinda standard?), and EXACTLY ONE partition marked as ‘active’ (fiddle with fdisk -e to make this so, should show up with an asterisk). Apparently, according to what I’ve read, this is how USB key flash diskey things come from the store, but YMMV. This blows out your entire partition. This can fit into 35 MB or so, but it will make the math look funny – e.g. “256MB drive, 34MB in use, 1MB available”. That’s expected. There are other ways to do this with DOS exe’s and other such crap that I really don’t want to mess with, so suck it up, too bad!

Braydix vs. Google Chrome OS

So I won’t lie – when reports started coming out about Google’s new Chrome OS, I was completely devastated. I liked to pretend I was making Braydix ‘just for fun’, but I think I had always secretly hoped that I might someday make a product of some sort out of it. Google’s announcement basically means the chances of that are nearly completely shot. That did not make me happy, to say the least.

But it’s been some time, and I’ve cogitated on it and pondered and moped, and I think I’m going to keep messing with what I’ve got. There are a few possible outcomes

  1. Google claims their software will be open source. So there’s nothing preventing me from taking whatever I want from that and integrating it with whatever I’ve got. I’ve got a pretty nifty filesystem, and a pretty tight high-level architecture for stuff.
  2. Or, if they really do a great job (very likely), I could just switch to it, completely. Whatever services or things I make available to my ‘thing’ can probably be made to be just as available to the Googlers.
  3. Or if their licensing terms are horrible, or their product is Big Brother-ey, I become the ‘friendlier’ alternative.
  4. Or maybe Braydix just becomes irrelevant.

The likelihood is that Chrome OS will suck whatever amount of developer-juice and hobbyist-interest away from me, and to them. I don’t foresee that changing.

But whatever the GOOG does, they cannot take away from me what I’ve learned, or what I’ve built, and if they make something that I don’t like, I can always make sure that I am making something I do like.

I do think the amount of time I spend on Braydix will decrease. It’s natural – when I have something that isn’t so unique and wonderful and clever as I thought it was, the impetus to work on it decreases. And when Google is advancing the same thing as I am, their bandwagon is a touch bigger than mine.

All that being said, though, here’s a nifty new version of Braydix. It does lots and lots of clever things, but to the end-user just looks like it boots up into a browser, and that’s it. When there are software updates to be had, there’s no ‘updating’ process that has to be run, the new version is just ‘there’. There’s even room for some way of running third-party apps. As a neat little aside, it will even work completely disconnected from the internet…though since all it does is browse the web, there’s not much of a point. Yet. Anyways seems pretty simple, but the truth is far more complicated…

********** WARNING. GEEKS ONLY BELOW THIS LINE ***********

Like any Linux distro, Braydix uses its own initramfs as the tiny bare-minimum memory-based filesystem from which to mount the ‘real’ root filesystem. The Braydix initrd looks for a ‘cache’ hard drive that it can boot from, and boots from it if it can. If it doesn’t find one, it uses a special “bootstrap file” to prepare a minimal set of files that so that it can boot up and launch a browser (and network config utilities and disk config utilities, etc). Once I got the files down to a size where they would fit, I used a ‘CRAMFS’ file to jam them into (it’s a special compressed filesystem).

Once the initramfs has loaded, it passes control on to the cache HD, or the minimal filesystem from the bootstrap. That’s the piece that manages to overmount crestfs over the current ‘web cache’ (whether it be a disk or temp files). CREST-fs is the crazy thing that I built that allows you to mount the whole of the Web as a filesystem. The ‘overmounting’ trick is tougher than it seems – the very files you’re running off of are the ones that are being replaced by a new copy! Gotta be careful – and debugging is a bitch. The cache that it’s overmounted-over is treated as cache, and if the contents are ‘current’, newer versions won’t have to be grabbed from the internet.Then it launches the browser.

The result? a 30 meg ISO. And I haven’t really gotten deep into trying to squeeeeeze and scrunch the files in there down (but I don’t think it will help much). and that 30 meg ISO could have give you access to a huge and complex back-end system – as I am expecting to do. It’s really fun putting things *not* on the bootstrap file, but on the server, and watching them show up on my test VM. That’s probably what’s happening next – write support (somehow), C compiler, and getting it all to be ‘self-hosting’ – instead of using a Redhat install as my ‘workstation’ to develop this, I will use instead a Braydix install to develop it. That will be fun!

Oh, and F google. F ’em in the eye. They’re starting to really freak me out.

The second-to-last release of Braydix…

So I’ve definitely made some Braydix progress – here’s the latest. I need to eventually come up with some kind of name that’s catchier than “Braydix” – that has the syllable for “dicks” in it, which…isn’t that nice. It’s still pretty huge – 140MB right now – but I have things being compiled with debugging information ‘on’ (which bloats the sizes out) and some things that I should have working as libraries are actually compiled-in. I think I may also make some stuff compressed, which very rough back-of-the-envelope math says should cut the size to a third (!).

So what’s in it, and why is it the second to last release? Well, I’ve tied in the CREST-fs ‘internetty’ filesystem pretty early into the boot sequence, so the system in essence can ‘boot off the web’. The idea is, when it’s integrated perfectly, that you would be able to do some kind of initial installation thing, and never have to do any upgrades or anything. Just reboot, and what you need is ‘there’. But right now it’s not quite integrated perfectly – I’m not sure how I do kernel or initramfs updates, or how they’d get picked up by the client. So once I’ve got *that* figured out, then *that* release becomes my ‘final’ release.

I’ve also built a very very simple installer, which you can launch using one of the virtual terminals (Ctrl-Alt-F1 is the main browser window, Ctrl-Alt-F2 through F4 gives you a terminal. Type ‘installer’). I wouldn’t run it on any piece of hardware you cared about though – it could format the wrong disk, or install to the wrong partition, or do any number of other, awful things.

The clever bit – well, there are many clever bits, but the ‘novel’ clever bit – is that I have a special ‘bootstrap’ image to prime the CREST-fs backing cache. So you can, in fact, boot up off this disk image with no network connection at all. And you’ll at least get the browser launched and the GUI up and running. From there, you can possibly configure wireless or something and continue to use the system from there.

The idea – and I don’t think I’ve articulated this very well – is that you should never have to install anything on this. Whenever new things show up on the internet, they’re just ‘there’ automatically. You never have to update anything – reboot and you should have the latest. If I end up doing a third-party application thing, just chuck your binaries and libs and stuff on a website somewhere, and it’s available to the systems – the CREST-fs mounting can talk to any webserver just as easily as any other, mine isn’t particularly special (except for the odd symlink that points to it, or a PATH entry).

Some notes:

  • Libraries SUCK
  • Compiling things ‘static’ is really hard
  • Booting is really complicated
  • The Linux Initramfs is really brilliant. You can do all kinds of ridiculously crazy shit with this. Very cool stuff.
  • Busybox is still awesome
  • “Virtualbox” as a development environment under OS X is (relatively speaking) an extremely pleasant place to develop in

Nerd Alert!

The following post is intended for a very technical audience. Consultant supervision is advised.

I wrote a thingee that lets you mount “The Web” as a filesystem. So far it works under Mac OS X, but I intend to port it to Linux. It uses MacFUSE for the filesystem interface – makes it much easier to write without kernel muckery. It’s all written in C. It’s available on GitHub. It’s called CREST fs – Cached REST, or Cached Representational State Transfer.

The thinking is that The Web, as a whole, is a set of resources, which I should be able to access like files from a file system. So as opposed to doing something like trying to curl http://samplesamp.sa/apage/something, then trying to run the code that lives on that page, you could instead mount the web under some directory, then cd to samplesamp.sa and execute apage/something directly. The first access to ‘something‘ would require it be fetched from across the internet, but with a caching scheme built in, subsequent accesses should be from the local hard-drive caching system.

I thought it would be clever to be able to mount it under a folder called /http:/ – so you could say:
ls -al /http:/www.google.com/someurl/something
See? I think that’s clever. You could probably put two slashes there and it wouldnt’ mess up anything. And if you were sitting in the / directory you could skip the first slash. But that’s starting to get too clever just for cleverness’s sake.

There’s stuff like that Out There already, but I wanted to build something that was extremely aggressive about caching, and very primitive (low-level, usable in Early boot environments). I expect it to notbe remotely coherent, but I do want it to be fast. So far, so good.

Sidenote: it’s my first Git project. Git is nice. Hosting it on GitHub is interesting, too, but less interesting than the fact that it’s on Git.

Writing code in C is painful. Allocating memory is not fun. Troubleshooting subtle memory leaks is not fun. Doing your own string manipulation by hand is not fun. But there is a certain feeling you get from being this close to the bare metal of the hardware…that’s really pretty exciting.

This is the second time I’ve written this – the first version was lost in the Great Hard Drive crash of ’08. Writing all this low-level crap is not that cool, but writing it for the second time is even less cool.

The intent behind all of this is to tie it in with Braydix somehow – to allow you to boot a “minimal” Braydix image from CD or USB key, and have it pull the rest from Teh Intarwubs. I have a new client who does a lot of work within Amazon’s EC2 environment, so I’ve had to study how that works. I think this FS might be interesting for that, too. As soon as I can find an excuse to put something up over there, I definitely will mess around with that too. Once I’ve done that, just think of it – they’ll be Braydix both client and server versions.

More Spam

Ugh.

So my clever hack about RSET apparently triggers problems in feeble, horrible, nasty mail clients like Eudora – which one of my client’s clients actually uses. So I had to back out my change. It was funny to hear someone read me my ‘garbage’ message right back to me, though.

So in the process of poking around, I found that there was already a feature in the qmail chkuser patch which allows you to set a number of bad recipients before which you are over your limit. So I enabled that. And it did not at all stem the flood, because it simply just rejected all subsequent attempts with 400-series messages – not disconnecting the sender.

So once again, I jumped in to the code. And I made it so that it actually disconnects you instead of just marking subsequent connection attempts as automatically-failing.

This seems like it’s working. I have 6500 IP’s in my self-written blacklist, and the smtp server-load has dropped to half. It’s still there, though, so I’ll have to keep an eye on it.

All in all, not a fun day…