Thursday, June 19, 2008

Ubuntu + LTSP + DrQueue = Render cluster

The latest version of Ubuntu incorporates the venerable LTSP project in an interesting way -- any chroot environment can be configured as an environment to be PXEBooted. Since PXEBoot has been built into every consumer machine for five years, many new things are possible.

LTSP is designed to be a way that ancient desktops and modern thin clients can be configured to save money on the point-of-access. This new facility means that much more can be done with it. Understanding how requires a bit of explaining.

A chroot environment is a configuration in Linux where the user can (CH)ange the (ROOT) directory to some subdirectory of the current computer. It's used in services to isolate a particular service or user's environment so that they can't access things they're not supposed to. It's like a limited virtual machine. It can be configured the same way as a normal environment would -- with local applications, events, all the usual stuff.

LTSP extends this by building the chroot environment into an image file that a booting machine can use as its own real environment. By controlling the chroot environments issued to various machines based on MAC address (an address unique to the machine or network card) one can assign a specific chroot environment to a particular machine. This allows the LTSP to issue a thin client linux to ancient computers that deliver a modern experience using the server's greater computation power. It also allows the system to send special environments based on the client's architecture. PowerPC Macintosh computers require a special one, as do some others. You can even PXE boot a virtual machine -- so as to leverage virtualization technologies and server consolidation dynamically. A controller process can be configured to monitor loads on your network at dynamically launch virtual machines to handle the loads as the need requires.

It has been possible for some time to build a redundant architecture for every common service that uses various network and software methods to assign work for one service to multiple servers. By leveraging this PXE boot, specific environments for specific services, and assigning machines to service tasks via MAC addresses it's possible to create a redundant architecture to provide all of these services that scales to any size.

This changes a great deal in infrastructure design. Every server can round-robin to whatever server is available. When services are slow: add another server to the list that receive the image for that service and boot it. It will automagically configure itself to receive a share of the load and serve clients. Need more power in your render cluster? Buy as many render nodes as you need and PXE Boot them -- no touch configuration. A node fails? That's fine. It's all redundant. Swap it out and move on. Even the LTSP servers themselves can be made redundant in this way, so that as long as one persists the architecture will survive.

What I think is cool about this: You can build the most powerful render cluster in the world without writing even a single line of code. That's right - the programmer-free cluster. It's all off the shelf hardware and software.

Over the next few weeks I'll be building a render cluster using cheap equipment. Watch this space to see what I can do with it.


Just Articles said...

Love the Blog!

Anonymous said...

Grate stuff, where is the report on what have you been doing? :)

symbolset said...


It went fine. I configured LTSP to boot to a menu with options for some images:

Ubuntu Linux (several flavors)
imaging with clonezilla
Remote desktop thin client (several flavors)
Several other flavors of linux

I have several servers to provide desktops because I'm interested in trying the different flavors. These are either selectable at boot time or configurable in DHCP, depending - you can configure DHCP to feed a particular server to a specific MAC address. I could probably support a few hundred users with the servers in my basement, but the power bill would be off the hook and there's not that much room in my basement anyway. Maybe someday I'll set up a service where anonymous users can remote desk to my servers over the Internet for a fee - obviously I'd have to put some servers in a more understanding country for that, but that's ok. There's plenty of understanding countries with bandwidth and I could probably support a cost effective number of users this way in one rack.

The default is a thin client with a desktop on my Ubuntu server. This boots to a desktop in under 30 seconds from power on. This is great for guests - I just hit the POST key for boot to network, let it default and they're on. They have flash and audio and all the usual tools. They can even download to a flash drive if they have one. The only thing they can't do is catch a virus. They don't have software installation rights of course, and there are no linux viruses in the wild anyway.

Imaging with clonezilla is cool. I configured the defaults on that option to download an image from my Samba share, and select an image to restore from a menu. All my favorite backups are there, and I don't have to take the defaults - I can point-in-time image any PC in the network in 30 minutes, and restore in less. I took out some of the options I won't use because too many options can be a bad thing. Clonezilla makes malware recovery on the kids' XP boxes a lot more fun. Since clonezilla boots from pen I could do that, but this was an interesting config. I can and have done this with Ghost too, but it's not free.

What I don't have is time to post about how this is done. I've moved on to other stuff. We're doing networking now. Sorry. There's lots of help available though.