Thursday, June 19, 2008

Ubuntu + LTSP + DrQueue = Render cluster

The latest version of Ubuntu incorporates the venerable LTSP project in an interesting way -- any chroot environment can be configured as an environment to be PXEBooted. Since PXEBoot has been built into every consumer machine for five years, many new things are possible.

LTSP is designed to be a way that ancient desktops and modern thin clients can be configured to save money on the point-of-access. This new facility means that much more can be done with it. Understanding how requires a bit of explaining.

A chroot environment is a configuration in Linux where the user can (CH)ange the (ROOT) directory to some subdirectory of the current computer. It's used in services to isolate a particular service or user's environment so that they can't access things they're not supposed to. It's like a limited virtual machine. It can be configured the same way as a normal environment would -- with local applications, events, all the usual stuff.

LTSP extends this by building the chroot environment into an image file that a booting machine can use as its own real environment. By controlling the chroot environments issued to various machines based on MAC address (an address unique to the machine or network card) one can assign a specific chroot environment to a particular machine. This allows the LTSP to issue a thin client linux to ancient computers that deliver a modern experience using the server's greater computation power. It also allows the system to send special environments based on the client's architecture. PowerPC Macintosh computers require a special one, as do some others. You can even PXE boot a virtual machine -- so as to leverage virtualization technologies and server consolidation dynamically. A controller process can be configured to monitor loads on your network at dynamically launch virtual machines to handle the loads as the need requires.

It has been possible for some time to build a redundant architecture for every common service that uses various network and software methods to assign work for one service to multiple servers. By leveraging this PXE boot, specific environments for specific services, and assigning machines to service tasks via MAC addresses it's possible to create a redundant architecture to provide all of these services that scales to any size.

This changes a great deal in infrastructure design. Every server can round-robin to whatever server is available. When services are slow: add another server to the list that receive the image for that service and boot it. It will automagically configure itself to receive a share of the load and serve clients. Need more power in your render cluster? Buy as many render nodes as you need and PXE Boot them -- no touch configuration. A node fails? That's fine. It's all redundant. Swap it out and move on. Even the LTSP servers themselves can be made redundant in this way, so that as long as one persists the architecture will survive.

What I think is cool about this: You can build the most powerful render cluster in the world without writing even a single line of code. That's right - the programmer-free cluster. It's all off the shelf hardware and software.

Over the next few weeks I'll be building a render cluster using cheap equipment. Watch this space to see what I can do with it.

Saturday, March 29, 2008

Networking education resources

If you want to get a good basic understanding of how basic networking works, you could do worse than to take the ProCurve Networking Primer. It offers the fundamentals in an easy to understand self-paced course. It doesn't have a lot of vendor bias in it.

HP offers a great deal of training in fundamentals for free. Some of it is specific to their products and some of it is not. On the ProCurve Training page you will find some materials to study if you are interested in these things. It's accessible to the public.

A lot of the HP training is available free to the public but it's hidden behind a membership page so people can't find it easily. This is silly because the training itself is hosted on a public FTP server. For example the exam preparation guides are in the epgs directory. There's quite a lot of interesting stuff on ftp.hp.com and it's wide open for browsing.

IBM also has a good deal of online training available here.

Naturally MIT's Open CourseWare covers networking as well

Friday, March 28, 2008

Nettop, netbook, Mobile Internet Device, Blah

Yeah, there's lots of vapor in the air regarding "thin is in" low power, small performance mini notebooks and portable PC components. It would easy to grouse about how we've heard this before and when the air cleared the thing cost $2500 if you could buy it at all and was lame until you dropped it, at which point it was worthless.

The thing is, that story's over. Flash storage as a medium has matured and become much cheaper. You can get small LCD (or newer tech) monitors at ridiculously low prices because of economies of scale. The small LCD in the eee PC for example is used in point of sale equipment, digital photo frames, kiosks, and a number of other devices. With a low power processor that's also cheap the Bill of Materials on this equipment starts getting interesting.

At IDF in a few days the NDA's for lots of companies building platforms on Intel's Diamondville and Silverthorne (nee Atom) processors expire and we're going to see what kind of device the major manufacturers can build with a 0.5W - 2.5W processor that is very cheap, runs IA32 architecture and clocks at reasonable (1.8 GHz?) speeds. I think there will be more than a few surprises in store.

I'm going to speculate there are more than a few that are a decent laptop computer that costs about what consumers are currently paying for an MP3 player like the Zune or the IPOD. That's going to drive a lot of market in the third world. It's going to change a lot of things about the bottom end of the laptop market. Some of these things are not going to be computers at all, but they also will be really cool.

One thing's for sure though: If any of them run Vista, they won't do it well.

Saturday, March 08, 2008

Will Intel's Atom be a smash?

The buzz has begun on Intel's Atom processor. Formerly known as Silverthorne and Diamondville, this disruptive technology is set to sweep the world by summer.

What is it? Atom is a processor. It consumes between 0.6 and 2.5 watts running full out depending on the clock speed, and as little as .03 watts in sleep mode. It is tiny- 25 square millimeters or roughly 3 millimeters by 9. It's x86 compatible, as it's derived from Intel's Core architechture. Clock speeds for it are currently estimated at 1.8GHz at the top end, 500MHz at the bottom. The technology is capable of either hyperthreading or dual core.

What's the big deal? This is huge. Look at the requirements for Windows XP. 300MHz Pentium class processor with 128MB of RAM. This thing easily clears even the recommended requirements at the minimum 0.6 watts power level. With an Intel Solid State hard drive and 1GB of RAM this thing is a whole PC that fits inside a tin of Altoids and runs on AA batteries or can be embedded inside a 22" monitor for about the cost of the cardboard box it comes in. In silicon small = inexpensive and this is tiny. This moves a real PC into the realm of affordability for a huge segment of the world's population that was previously not served.

Mini-ITX is a popular platform, and VIA gets up to $300 for their 1.5GHz platforms in this form factor. "This complete platform is expected to be priced at no more than $50-60 in retail." Wow. Just wow. The implications for car PC and embedded media player applications is enormous.

What else? There's a Centrino Atom chipset aimed at the Eee, OLPC and Classmate class of cheap notebooks with wireless, fair video, and all the usual goodies that stays at low power. Over 50 subnotebooks that Intel is now classing as "Mobile Internet Devices" or MIDs are launching right away. Phone applications are obvious. Perhaps less obvious are the implications for home routers, thin clients, toys, home robotics, gumstix, Network Attached Storage, Wireless mesh networks, military applications, POE webcams, supercomputer applications and workstations.

The downside: Although some vendors will claim the "Vista Capable" label, we all know what that means. It means that the PC is incapable of giving a good experience when loaded with Vista. A version of Ubuntu is available for it already, though, that runs Open Office just fine so you should be able to open those PowerPoint presentations in Impress without any trouble on your Mobile Internet Device.

The interesting question really is "what would you do with it?" Really. Pretend for a moment you're a platform engineer and tell me what you would do with this thing.

Sunday, December 30, 2007

Zscanner 800


The Zscanner 800 looks like a nice gift for the budding 3d developer. It's a handheld trinocular camera with laser LED projector that interpolates points in 3d, in real time.

Just the right size for scanning the faces of your whole family so you can update their secondlife avatars. Output is .stl files.

At $50k for this version getting models from the real world into the virtual world is getting cheaper all the time. The intellectual property ramifications of this technology coming within the reach of common citizens must be astounding.

This is still out of reach for me, but I would not mind renting one for a few days.

Thursday, December 27, 2007

Switching to linux

Lots of people are thinking about switching to linux these days. There's a website called Groklaw dedicated to documenting legal events, and they have quite a useful page on the subject. Rather than working up my own, I'll point you there.

Monday, May 28, 2007

Video on phone update

Recently I had to replace my video-capable phone with a Blackberry, and as luck would have it I got mine only days before the Blackberry that does video and audio came out. It was a tradeoff. My thumb hurts because the BB scroll wheel is a poor substitute for a touch screen. On the other hand, mail arrives quickly and I can use it as a wireless modem for my laptop.

Since my blackberry doesn't do video any more, I chose the Centon moVox 1GB for that. It's $50 delivered, plays mp4 videos and mp3 audio, and stores a gigabyte of info. You format it and drop files into it like a pen drive, and they play just fine. It charges from USB too. I've tried the methods previously described here for converting video, and they work just fine. Now I can store four two hour movies on it and a couple albums, and reserve my phone for other stuff.

You can't tell from looking at the Newegg description, but the thing is very small. It's about 1cm thick, 3cm wide and 4.5 cm tall. The buttons are not intuitive -- you have to play with them to figure out that ff+play = enter on the menus. The manual is no help -- it's an amusing example of engrish, and lacks even an identification of the buttons.

There is no output for external video -- this thing is strictly a microscreen player. If you're interested in a little recreational video in a pinch, though, it will do the trick. If you get one, don't forget to format it FAT. You can't store any media on it until you do. It comes in more expensive versions up to 8GB. I have no idea what battery life is, but after I've used it a while I'll update this post. As always, Newegg delivered promptly and as advertised. It's a lot more fun to just fire off an online order in the middle of the night than to burn a precious day and five gallons of gas wandering from (hopefully open!) store to store searching for something that might or might not be good.

Realtime satellite photos of Australia

You can see cool weather photos of Australia from BSCH which is the Brisbane Storm Chasers homepage.

Saturday, December 30, 2006

Play that DVD on your phone

Sure, you paid for that movie on DVD. You didn't care what format it was in, you just wanted the movie. Now you're busy like me and you want to watch it on your cell phone or handheld video player. This article tells you how it's possible, without buying any software.

Before I get into that, though, I have to warn you. The movie studios don't want you to have that content available in the format you desire. You're going to break the laws of several nations on your way to convenient video. Such is life. When you get busted, I didn't tell you to do it, I only told you how. BTW, this guide is for media shifting and legitimate backups only. Don't go using this information for sharing purposes. If you didn't buy access to the content, you've got no right to it. If you're not sure you have a legal right to make backups in your local area, consult the services of a legal professional before continuing.

On with the show. This isn't the only way to do this, but it's a convenient way if you have the necessary equipment. You'll need:

A windows computer
A linux computer
A device to play your video on that plays MPEG2.

The first thing you need to do is to get it off of the DVD and onto the hard drive of your windows PC in an unencrypted format. For this you can use the program DVD Decrypter. Just install it from the downloads section of http://www.doom9.org/ and insert the DVD. The default settings almost always work. You'll want to do one movie first, just to be sure you have it right. After that if you're converting bunches it might be best to decrypt several at a time.

The next thing is to get it out of the DVD file format and into something a little more portable. For this you can use Auto Gordian Knot (AutoGK). It's available from the same source. In this step choose encoding as AVI with Xvid video codec and MP3 VBR for the audio. For output settings choose 100 percent target quality, rather than size or CD's based output. That way your output will come all in one file and it will only require one pass (it's faster). Also, multiple sessions of lossy compression introduce unwanted artifacts in the output. Always keep the maximum quality until the final rendering step. Don't worry -- the file will be small enough to fit in your player when you're done. If you're doing a batch, choose Add Job and then select a different disk folder and output file and add it to the batch. Click start.

AutoGK uses some external programs to convert your file according to the settings you gave it. They're installed with AutoGK, though and you don't have to think about them much. The first time you use them though, at various steps you'll be prompted to accept their license terms. AutoGK is doing the really hard part here. It's converting the original coded and compressed video into a different video code basically by uncompressing the images and then recompressing them in the new format one at a time. A movie has hundreds of thousands of image frames, so this can take some time. In addition to this it's doing the same thing with the audio portion of the movie, and keeping the sound in synch. If your computer isn't completely stable and reliable, this is when you find out.

Once you have your output .avi file, you need to get it over to the linux computer so you can process it with ffmpeg. It's possible to get ffmpeg installed on your windows computer, but there's no way I can provide instructions for that in a blog post. If your computers are networked, you can just save the file over to the linux box. Otherwise you can burn the file to a DVD. It can be pretty large, though. Some of these files will be more than 2GB. You can also use an external hard drive. External HDD's are available in larger sizes. Most current linux distributions can read the files off of an NTFS formatted external hard drive. If the hard drive is formatted with FAT32 instead, you'll have to keep your file sizes under 2GB. However you get it there, you'll want the .avi file on a hard drive that's local to your linux box.

When you have the file on your linux box the last step is easy. You'll need ffmpeg. It comes with most linux distributions, but if you don't have it, get it the same way you get your other software. (Note for windows only users: linux usually comes with software to install thousands of useful programs like this for free.) While you're at it get vlc (videolan-client) as well for watching movies. Run ffmpeg on the file like this:

# ffmpeg -i IN.avi -s 352x288 -ab 32 -ac 1 -b 64 -ss 25:00.00 OUT.mpg

Replace IN.avi and OUT.mpg with the files you want of course. The options are like so:
-i IN.AVI - the input file
-s 352x288 - The output resolution. Use what's appropriate for your device.
-ab 32 - Audio bitrate 32kbps
-ac 1 - Mono output
-b 64 - Video bitrate, 64kbps
-ss 25:00.00 - Start 25 seconds into the video

How long it takes to convert varies with your computer power. On a Core2 Duo laptop the decrypting takes about 20 minutes. The convertion to AVI takes about an hour. On an Athlon 2500+ the .mpg conversion takes about 30 minutes. All of that for a typical 90 minute movie, using the settings above. The finished movie might be 96MB. Quality is about what you would expect for watching a DVD on your phone.

The last step is to get the .mpg movie into your device. SD media is great for this, or you can use whatever synch system comes with your device. Since it's unencrypted baseline video, it should play in almost anything that claims to be a video capable device. A two GB SD card holds about 20 typical movies.

If you have better ideas for how to do this more conveniently, I would love to see your comments.

Saturday, December 16, 2006

Flash HDD

A lot of noise is being made these days about Flash chips and their potential use as system hard drives. I thought I would write about some of my impressions on the subject.

The first issue is write cycles. In the past Flash media was good for less than a million writes or so. This was completely unsatisfactory for most uses as an operating system media because systems generally use the hard drive as a swap, or short term place to store programs and data when they're not busy. In an active system the swap memory can be rewritten thousands of times a day, as programs are swapped into and out of memory very actively.

There are operating systems that don't function in this way. Certainly this process was necessary when computers had little memory, but with cheap PCs able to handle eight or sixteen gigabytes of memory, but it doesn't seem to be as necessary as it once was. Choosing to avoid the issue in this way does limit your choice of OS, but not horribly so.

Recent advances in flash memory have extended the life of the flash memory into hundreds of millions, or more, of write cycles. If you're willing to accept a lifespan of five to ten years for your flash memory, you should be fine with what's available now. Certainly we can expect this trend to continue. Flash will get more durable life cycles until they finally are good long after one would normally consider them obsolete.

Currently available flash media (December 2006) comes in sizes up to 32GB for Secure Digital media. That's a lot of memory for a card that small, and it comes at a premium price. When I first bought a flash drive the largest available size was 64MB, and it was only a few years ago. Since sizes have increased five hundred times in just a few years, and manufacturers are even now working on several generations of denser media. Since a large operating system install should be no more than 8GB currently, and that size is available, Flash Media has cleared the hurdle of being large enough to handle the job. Although larger form factors are available now for 32GB media, the cost of the flash chips is enough to prevent a large market for the devices and so they're not yet common except for those who have no budget constraints.

Speed is another issue when considering Flash media for your system drive. Although Flash currently can be much slower than HDD media, that is changing as ever more chips are added and accessed in parallel. Already you can get media that reads and writes faster than an ATA HDD. Soon they will be much faster. More importantly, since Flash has no moving parts there is no latency to speak of and every file is as close as any other. This simplifies much of the disk access process and makes file access much faster. Speed is about to cease to be an issue for Flash media, and already for most uses it's faster than a hard drive. With Flash a cache might be necessary for some applications, but different from HDD media, the amount of time it takes to flush the write cache is predictable and controllable from the system rather than the controller or on-drive electronics. This makes shutdown issues go away almost entirely.

Power issues are important for storage media -- not only for battery life but also for heat. Here Flash has long been a big winner for cameras, smart phones and PDA's. Because Flash is a static medium, no energy is required to maintain the data stored within or make it accessible. There is no spin up time, no idle power at all. If you're not writing to or reading from it, it uses no power. Naturally devices that use no power generate no heat. Even in their most power hungry use, writing, Flash media doesn't take as much energy as HDD media at idle. The heat issue is an important one because the more thermal energy a device dissipates, the larger it must be to passively cool, and active cooling adds energy costs, size and noise as well.

Flash media has no moving parts. It is utterly silent. This by itself makes it a preferred medium for applications like fanless computers in audio recording environments and low energy entertainment center platforms where even small amounts of fan noise are unacceptable.

There are already many distributions of Linux that can be installed to Flash media. Soon this will be a standard install option across nearly all distributions. Installing to Flash media can be very handy for workstations. Rather than evolve a fancy network system for maintaining each user's settings in a portable way -- that doesn't work if the network is down or the user is offsite, the user can just extract his boot chip and take it with him. Then he can arrive with his full toolset and get to work without worrying about which applications don't install correctly at the new station or aren't integrated properly or don't have his preferred settings.

Price is often an issue. Flash media costs many times what Hard Drives cost for the same amount of storage. Today I can buy a 2GB SD media for about $30.00. It would take 250 of those to make the volume of a 500GB HDD that costs $150.00. For this article however I'm not talking about a volume for storing your media or your database. 8GB should be plenty right now for an operating system and suite of applications for normal use, and that can be had for a reasonable price so I think Flash has cleared the affordability constraint as an operating system install medium.

For these reasons I think we're approaching the day when a Flash Drive is a part of the motherboard on desktop and server systems for the purpose of OS and applications install at least as an option. It seems likely to become the standard for media center PCs as well. I also expect to see more support for installing to this sort of media. It seems reasonable that advances in Flash media will continue to outpace progress in other areas of information technology. In addition for highly portable devices like smartphones richer suites of applications should soon be available to exploit the advantages of having larger static storage available.

Some consolidation in the Flash memory industry seems likely as well. Highly competetive markets like this one erode profit margins and depreciate inventory disruptively. Obviously in this environment purchasing a competitor can be cheaper and more effective in the long term than inventing a new process that increases the storage density or speed of your product line.

Sunday, November 19, 2006

HDTV 720p, 1080i

It's 2006. There's a lot of product shipping right now in the HDTV market, and a bunch of people who are buying in early are going to be unhappy.
1080p is the full HDTV resolution that is supported by BluRay and HD-DVD. It looks totally awesome. It is supported on the PS3, since it's BluRay compatible.
One of the amazing things about the pace of tech progress is that often times the next generation of product will arrive at the same price point as the previous generation. Right now you can get a generous 42" 1080p monitor with HDMI and/or DVI input for under $1500. That's actually less than many of the 720p or 1080i monitors I've seen at the same size.
Early adopters usually expect to bear the brunt of the costs for new tech, but now that the 1080p monitors are available many of them will choose to upgrade from their previous 720p LCD or plasma selections. Someone who's considering getting one now shouldn't consider getting the lower resolution because they'll only have to upgrade later and pay the same price twice.

Sunday, November 12, 2006

Firemumble

The browser Firefox is getting a lot of attention lately because they've asked the Ubuntu people to stop shipping their modified version with the "Firefox" symbols. Ubuntu will be naming it "IceWeasel".

Not everybody remembers that there used to be a Firesomething extention for Firefox that allowed you to change the program name, or for it to change randomly. It can be a lot of fun. The developer has stopped supporting it, but apparently the extension can be modified to work with version 2.0, so we can have the fun all over again.

The joke here is that the Firefox team had to change the name a couple of times early on, so as not to conflict with extant products.

Saturday, November 11, 2006

Toynbee idea

As long as I'm blogging about Enigma, I may as well add another. Have you heard about Toynbee tiles? They're laid in streets all over the world, and one of them once nearly killed me. I was crossing a busy street and there in the asphalt was the sign:
TOyNBEE IDEA
IN KUbricK's 2001
RESURRECT DEAD
ON PLANET JUPiTER.
Struck dumb for a moment as I unravelled the meme, a bus almost smashed me.

Anyway, if the story intrigues you, you can check out the article on Toynbee at Wikipedia, read some of his works at Project Gutenberg, or (extra credit) find page 22 of the Feb 4, 1958 issue of The Atlanta Constitution. For sharing the enigma, you can as usual get themed merch from CafePress.

Where in the multiverse is John Titor?

John Titor was a poster on Usenet many years ago, who claimed to be from the future. His story remains an enigma. You can read more about him on Wikipedia.

If you're so inclined, you can buy John Titor themed stuff from CafePress. It's the kind of inside joke that takes a special person to appreciate. What intrigues me about the story is that I learned programming on an IBM 5100 and was familiar with some of the material discussed. I think I read some of his articles when they first appeared.

Discover the GRAIL

GRAIL is the Graphics and Imaging Laboratory of the University of Washington's Department of Computer Science and Engineering.

What's interesting about it is that they have a lot of papers available on computer science as applied to graphics.

Music from math

The application of math in music is an interesting concept with a long history. Rather than bore you with it, let me suggest Wolfram Tones as a primer.

This website has a number of generators that let you create music. It's done by the people behind Mathematica. One cool feature is that once you've found or made a composition you like, you can send it to yourself as ringtone.

Sunday, October 01, 2006

Let's talk about Blender

Blender is a 3d modelling program. It's great. It's free. If you are at all interested in creating computer illustrations or graphics, you should get this program.

Head on over to the blender nation blog and have a look at some of the things it can do.

You are going to need some patience to master Blender. It takes a while to get the hang of editing three dimensional objects and the user interface is very rich. Blender is feature filled and has a nice professional finish to it. The software is very powerful. If you want to render a movie like Elephant's Dream, the software allows for using a render farm.

Blender imports VRML 1.0, DXF (Autocad), Videoscape, STL, 3D Studio, AC3d, Collada, DEC objectfile, directX, Google Earth, Lightwave, Quake, Nendo, OpenFlight, ProEngineer, Radiosity, RawFaces, Stanford PLY, TrueSpace and Wavefront files.

Not everything has to be free. There are plentiful resources of pre-made objects available for sale that are royalty-free and that can save a ton of time. If you're interested in these, try out turbosquid.com. They have thousands available.

Modelling programs use descriptions of shapes called meshes to form the basis of objects that you can manipulate. You can find some meshes, and other things for free here:


The meshes only provide a framework for the objects. There are also materials, textures and finishes that are used to render your artwork into something hopefully photo-realistic.

Saturday, September 16, 2006

How is this for a lot of motivation? This crew keeps me busy!

Monday, August 28, 2006

First blogger post

This is an interesting exercise