For some time now I've been interested in Geographic Information Systems (GIS) for Linux. This is a natural combination since there is a huge amount of freely available geographic data available for free from the US government. GIS systems take datapoints, usually as geographic coordinates (longitude, latitude and elevation) and by associating various data (stream surveys, street plans, etc.) give a graphical representation that's very flexible. It's helpful to use them to make maps or visualize different elements.
There is a project with a long history in Linux that does this -- it's called GRASS. It was chosen for three projects in the 2008 Google Summer of Code. It's an active project with a long history and many users so it's likely to be around for quite a while longer, and it's licensed under the GNU GPL so price isn't an issue.
GRASS is pretty feature rich. GIS systems are always complex beasts as the various methods of storing, converting and visualizing geographic data are all rich fields with long histories and good fields for varying preferences. This system allows GIS data to be stored in any of the common databases including MS Access, MySQL, PostgrSQL, MS-SQL Server, Oracle, dBASE and others as well as various common formats or flat files. It can use live files created for and by ESRI's ArcGIS, which is the commonest commercial GIS program.
With the next version of GRASS a native Windows build will be available. For now the Windows version of the application is built under Cygwin.
Like many GPL licensed applications, GRASS has been included in a number of packages called distributions that include many complimentary applications that target an audience with a complete suite of applications and related tools that suit a common purpose, along with the Linux operating system and all of the usual applications as well. ArcheOS is an example of one that's targeted to archeologists that provides GRASS and related tools as well as a rich set of new toys to play with. I'll be using ArcheOS to set up a workstation system with GRASS. As of the current version (2.0.0) ArcheOS comes as a 1.2GB .iso file to burn to DVD for live DVD use or to install and includes version 6.2.3 (the most current stable release) of GRASS.
Anyway, give GRASS a try and tell me what you think.
Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts
Sunday, July 13, 2008
Friday, July 04, 2008
LTSP configuration (Gutsy) - Episode 2
LTSP is the Linux Terminal Server Project. Because it's popular with schools it's had quite a bit of development and it has been adopted by Ubuntu as part of their Edubuntu package. It's generally used to allow a server to provide the horsepower for a bunch of thin clients. We'll be expanding it to server other useful purposes.
We're going to use it to help clone a bunch of Windows XP computers.
In our first episode we built an LTSP server. If you haven't read that article yet or you don't have a good LTSP server (not in production!) to work with it would be good to go back there and follow the steps so you are better able to follow along.
There are several steps to perform here. We have to select an imaging platform that's bootable with ltsp. It has to not copy the whole drive -- just the blocks that have data in them. It has to be reasonably fast. We have to select a method of getting it the large image to the clients -- probably file sharing but possibly multicasting. We have to tie it all together. One advantage we have is that we have a large number of machines laying around with 40GB drives and gigabit ethernet to use as servers. They're surplus from a prior installation.
To get the image size down we need to use a project that uses ntfsclone, since that's the project that knows about the contents of NTFS formats and can copy only the blocks that have data. We need a project that uses ntfsclone and works with ltsp but allows us some flexibility in how we use it. I chose clonezilla. This project is a subproject of Diskless Remote Boot in Linux (DRBL), which is a similar project to ltsp. DRBL and clonezilla are projects of Taiwan's National Center for High-Performance Computing. It has handy installers, comes in bootable CD and pendrive formats, and a version is available for network booting. Although they claim to support multicasting the process is as yet unwieldy so we'll be using the ltsp server as a dhcp and file server, and clone multiple servers to meet our bandwidth requirements. Since we're using Ubuntu for the ltsp server, I decided to go with Clonezilla Live Experimental (Hardy). Download the .iso and burn it to a CD.
Before we put a lot of work into making it netbootable I should probably validate that it makes a good copy in a reasonable period of time. I'll be using a recently imaged old laptop that won't be a disaster if I mangle its image, just in case clonezilla does not work as advertised. My actual target clients are dual core notebooks with faster hard drives but the image is also five times the size. I'm looking for scalability on the server (serving many clients simultaneously) and on the network.
Boot the clonezilla CD from a client you would like to clone that's connected via network to your ltsp localnet.
pic
Choose the boot to ram option because we're going to use run from RAM if we PXE boot. After some text scrolls past you see this
pic
We'll choose the english version and don't touch keymaps.
Start clonezilla
device image disk/partition to/from image
We'll use the ssh server because we don't have samba or nfs set up yet on our ltsp server.
It will automatically detect our nic and network
DHCP is set up so we'll use that to get our address.
It detects our server and offers it as the default.
Port 22 is the default for ssh.
The default account is root. We don't allow remote access from a root account so we'll use this one.
Here we select a directory on the host. This is a good time to make the directory and ensure it is owned by the user you selected before.
We're warned we're about to be asked a password.
Are we sure we want to connect to a new server? Of course the answer is yes here.
Here's a prompt with no useful information for what we're doing. Press enter.
We're going to choose savedisk here to take a snapshot of the hard drive in this computer. When we restore we choose restoredisk instead.
We're going to use ntfsclone, so choose the first option here.
The default here is only to choose -c for wait for confirmation. We're going to clear that and have no options set on this screen.
The hard drive on this PC has 5.4 GB. Using -z1 we can bring that down to 2.1, which is better for our networking. -z2 is much slower for little improvement in image size and net, it's probably a loss in speed.
Here we choose an image name. This will actually be a subdirectory in the folder chosen previously, with various files in it.
There's only one drive in this machine. It's a laptop. As soon as we confirm this last entry, it will begin taking the image and storing it on the server. First run with 10/100 networking took 445 seconds. Second copy to server took 442 seconds. Download with 10/100 took 382 seconds, and at gigabit speed we get 365 seconds. Obviously bandwidth isn't our bottleneck. One thing to watch out for -- on the server, storing one image both CPUs hit about 50%, considerably more than their baseline 20%. This is likely due to the encryption overhead of SSH connections. The network usage goes in spikes of about 4-12 MB/s with gigabit networking. To improve this we'll need to a different network protocol to serve the images.
Now we check the image size on the network.
ltsp:/$ ls -hal /home/partimag/2008-07-05-00-img/
total 2.1G
That's good. Now we repeat the process but choose to download the image.
Test the image thoroughly. Are all the files there? Perform a chkdsk. No errors? Then we've got a viable copy but the speed needs work.
I'll try Samba next. We'll stick with the gigabit connection since it's up. I would go over the way I configured Samba, but you can figure it out from this useful page.
Testing with samba reveals that the server processor overhead for a single gigabit connection goes from the baseline of 20% to about 22%. We've removed the processor bottleneck. We're only using about 1/10th of a gigabit link on average. We have plenty of machines available so we'll probably go with six or eight clients per server depending on how much the load slows them down.
We're going to use it to help clone a bunch of Windows XP computers.
In our first episode we built an LTSP server. If you haven't read that article yet or you don't have a good LTSP server (not in production!) to work with it would be good to go back there and follow the steps so you are better able to follow along.
There are several steps to perform here. We have to select an imaging platform that's bootable with ltsp. It has to not copy the whole drive -- just the blocks that have data in them. It has to be reasonably fast. We have to select a method of getting it the large image to the clients -- probably file sharing but possibly multicasting. We have to tie it all together. One advantage we have is that we have a large number of machines laying around with 40GB drives and gigabit ethernet to use as servers. They're surplus from a prior installation.
To get the image size down we need to use a project that uses ntfsclone, since that's the project that knows about the contents of NTFS formats and can copy only the blocks that have data. We need a project that uses ntfsclone and works with ltsp but allows us some flexibility in how we use it. I chose clonezilla. This project is a subproject of Diskless Remote Boot in Linux (DRBL), which is a similar project to ltsp. DRBL and clonezilla are projects of Taiwan's National Center for High-Performance Computing. It has handy installers, comes in bootable CD and pendrive formats, and a version is available for network booting. Although they claim to support multicasting the process is as yet unwieldy so we'll be using the ltsp server as a dhcp and file server, and clone multiple servers to meet our bandwidth requirements. Since we're using Ubuntu for the ltsp server, I decided to go with Clonezilla Live Experimental (Hardy). Download the .iso and burn it to a CD.
Before we put a lot of work into making it netbootable I should probably validate that it makes a good copy in a reasonable period of time. I'll be using a recently imaged old laptop that won't be a disaster if I mangle its image, just in case clonezilla does not work as advertised. My actual target clients are dual core notebooks with faster hard drives but the image is also five times the size. I'm looking for scalability on the server (serving many clients simultaneously) and on the network.
Boot the clonezilla CD from a client you would like to clone that's connected via network to your ltsp localnet.
pic
Choose the boot to ram option because we're going to use run from RAM if we PXE boot. After some text scrolls past you see this
pic
We'll choose the english version and don't touch keymaps.
Start clonezilla
device image disk/partition to/from image
We'll use the ssh server because we don't have samba or nfs set up yet on our ltsp server.
It will automatically detect our nic and network
DHCP is set up so we'll use that to get our address.
It detects our server and offers it as the default.
Port 22 is the default for ssh.
The default account is root. We don't allow remote access from a root account so we'll use this one.
Here we select a directory on the host. This is a good time to make the directory and ensure it is owned by the user you selected before.
We're warned we're about to be asked a password.
Are we sure we want to connect to a new server? Of course the answer is yes here.
Here's a prompt with no useful information for what we're doing. Press enter.
We're going to choose savedisk here to take a snapshot of the hard drive in this computer. When we restore we choose restoredisk instead.
We're going to use ntfsclone, so choose the first option here.
The default here is only to choose -c for wait for confirmation. We're going to clear that and have no options set on this screen.
The hard drive on this PC has 5.4 GB. Using -z1 we can bring that down to 2.1, which is better for our networking. -z2 is much slower for little improvement in image size and net, it's probably a loss in speed.
Here we choose an image name. This will actually be a subdirectory in the folder chosen previously, with various files in it.
There's only one drive in this machine. It's a laptop. As soon as we confirm this last entry, it will begin taking the image and storing it on the server. First run with 10/100 networking took 445 seconds. Second copy to server took 442 seconds. Download with 10/100 took 382 seconds, and at gigabit speed we get 365 seconds. Obviously bandwidth isn't our bottleneck. One thing to watch out for -- on the server, storing one image both CPUs hit about 50%, considerably more than their baseline 20%. This is likely due to the encryption overhead of SSH connections. The network usage goes in spikes of about 4-12 MB/s with gigabit networking. To improve this we'll need to a different network protocol to serve the images.
Now we check the image size on the network.
ltsp:/$ ls -hal /home/partimag/2008-07-05-00-img/
total 2.1G
That's good. Now we repeat the process but choose to download the image.
Test the image thoroughly. Are all the files there? Perform a chkdsk. No errors? Then we've got a viable copy but the speed needs work.
I'll try Samba next. We'll stick with the gigabit connection since it's up. I would go over the way I configured Samba, but you can figure it out from this useful page.
Testing with samba reveals that the server processor overhead for a single gigabit connection goes from the baseline of 20% to about 22%. We've removed the processor bottleneck. We're only using about 1/10th of a gigabit link on average. We have plenty of machines available so we'll probably go with six or eight clients per server depending on how much the load slows them down.
Thursday, June 19, 2008
Ubuntu + LTSP + DrQueue = Render cluster
The latest version of Ubuntu incorporates the venerable LTSP project in an interesting way -- any chroot environment can be configured as an environment to be PXEBooted. Since PXEBoot has been built into every consumer machine for five years, many new things are possible.
LTSP is designed to be a way that ancient desktops and modern thin clients can be configured to save money on the point-of-access. This new facility means that much more can be done with it. Understanding how requires a bit of explaining.
A chroot environment is a configuration in Linux where the user can (CH)ange the (ROOT) directory to some subdirectory of the current computer. It's used in services to isolate a particular service or user's environment so that they can't access things they're not supposed to. It's like a limited virtual machine. It can be configured the same way as a normal environment would -- with local applications, events, all the usual stuff.
LTSP extends this by building the chroot environment into an image file that a booting machine can use as its own real environment. By controlling the chroot environments issued to various machines based on MAC address (an address unique to the machine or network card) one can assign a specific chroot environment to a particular machine. This allows the LTSP to issue a thin client linux to ancient computers that deliver a modern experience using the server's greater computation power. It also allows the system to send special environments based on the client's architecture. PowerPC Macintosh computers require a special one, as do some others. You can even PXE boot a virtual machine -- so as to leverage virtualization technologies and server consolidation dynamically. A controller process can be configured to monitor loads on your network at dynamically launch virtual machines to handle the loads as the need requires.
It has been possible for some time to build a redundant architecture for every common service that uses various network and software methods to assign work for one service to multiple servers. By leveraging this PXE boot, specific environments for specific services, and assigning machines to service tasks via MAC addresses it's possible to create a redundant architecture to provide all of these services that scales to any size.
This changes a great deal in infrastructure design. Every server can round-robin to whatever server is available. When services are slow: add another server to the list that receive the image for that service and boot it. It will automagically configure itself to receive a share of the load and serve clients. Need more power in your render cluster? Buy as many render nodes as you need and PXE Boot them -- no touch configuration. A node fails? That's fine. It's all redundant. Swap it out and move on. Even the LTSP servers themselves can be made redundant in this way, so that as long as one persists the architecture will survive.
What I think is cool about this: You can build the most powerful render cluster in the world without writing even a single line of code. That's right - the programmer-free cluster. It's all off the shelf hardware and software.
Over the next few weeks I'll be building a render cluster using cheap equipment. Watch this space to see what I can do with it.
Saturday, December 16, 2006
Flash HDD
A lot of noise is being made these days about Flash chips and their potential use as system hard drives. I thought I would write about some of my impressions on the subject.
The first issue is write cycles. In the past Flash media was good for less than a million writes or so. This was completely unsatisfactory for most uses as an operating system media because systems generally use the hard drive as a swap, or short term place to store programs and data when they're not busy. In an active system the swap memory can be rewritten thousands of times a day, as programs are swapped into and out of memory very actively.
There are operating systems that don't function in this way. Certainly this process was necessary when computers had little memory, but with cheap PCs able to handle eight or sixteen gigabytes of memory, but it doesn't seem to be as necessary as it once was. Choosing to avoid the issue in this way does limit your choice of OS, but not horribly so.
Recent advances in flash memory have extended the life of the flash memory into hundreds of millions, or more, of write cycles. If you're willing to accept a lifespan of five to ten years for your flash memory, you should be fine with what's available now. Certainly we can expect this trend to continue. Flash will get more durable life cycles until they finally are good long after one would normally consider them obsolete.
Currently available flash media (December 2006) comes in sizes up to 32GB for Secure Digital media. That's a lot of memory for a card that small, and it comes at a premium price. When I first bought a flash drive the largest available size was 64MB, and it was only a few years ago. Since sizes have increased five hundred times in just a few years, and manufacturers are even now working on several generations of denser media. Since a large operating system install should be no more than 8GB currently, and that size is available, Flash Media has cleared the hurdle of being large enough to handle the job. Although larger form factors are available now for 32GB media, the cost of the flash chips is enough to prevent a large market for the devices and so they're not yet common except for those who have no budget constraints.
Speed is another issue when considering Flash media for your system drive. Although Flash currently can be much slower than HDD media, that is changing as ever more chips are added and accessed in parallel. Already you can get media that reads and writes faster than an ATA HDD. Soon they will be much faster. More importantly, since Flash has no moving parts there is no latency to speak of and every file is as close as any other. This simplifies much of the disk access process and makes file access much faster. Speed is about to cease to be an issue for Flash media, and already for most uses it's faster than a hard drive. With Flash a cache might be necessary for some applications, but different from HDD media, the amount of time it takes to flush the write cache is predictable and controllable from the system rather than the controller or on-drive electronics. This makes shutdown issues go away almost entirely.
Power issues are important for storage media -- not only for battery life but also for heat. Here Flash has long been a big winner for cameras, smart phones and PDA's. Because Flash is a static medium, no energy is required to maintain the data stored within or make it accessible. There is no spin up time, no idle power at all. If you're not writing to or reading from it, it uses no power. Naturally devices that use no power generate no heat. Even in their most power hungry use, writing, Flash media doesn't take as much energy as HDD media at idle. The heat issue is an important one because the more thermal energy a device dissipates, the larger it must be to passively cool, and active cooling adds energy costs, size and noise as well.
Flash media has no moving parts. It is utterly silent. This by itself makes it a preferred medium for applications like fanless computers in audio recording environments and low energy entertainment center platforms where even small amounts of fan noise are unacceptable.
There are already many distributions of Linux that can be installed to Flash media. Soon this will be a standard install option across nearly all distributions. Installing to Flash media can be very handy for workstations. Rather than evolve a fancy network system for maintaining each user's settings in a portable way -- that doesn't work if the network is down or the user is offsite, the user can just extract his boot chip and take it with him. Then he can arrive with his full toolset and get to work without worrying about which applications don't install correctly at the new station or aren't integrated properly or don't have his preferred settings.
Price is often an issue. Flash media costs many times what Hard Drives cost for the same amount of storage. Today I can buy a 2GB SD media for about $30.00. It would take 250 of those to make the volume of a 500GB HDD that costs $150.00. For this article however I'm not talking about a volume for storing your media or your database. 8GB should be plenty right now for an operating system and suite of applications for normal use, and that can be had for a reasonable price so I think Flash has cleared the affordability constraint as an operating system install medium.
For these reasons I think we're approaching the day when a Flash Drive is a part of the motherboard on desktop and server systems for the purpose of OS and applications install at least as an option. It seems likely to become the standard for media center PCs as well. I also expect to see more support for installing to this sort of media. It seems reasonable that advances in Flash media will continue to outpace progress in other areas of information technology. In addition for highly portable devices like smartphones richer suites of applications should soon be available to exploit the advantages of having larger static storage available.
Some consolidation in the Flash memory industry seems likely as well. Highly competetive markets like this one erode profit margins and depreciate inventory disruptively. Obviously in this environment purchasing a competitor can be cheaper and more effective in the long term than inventing a new process that increases the storage density or speed of your product line.
The first issue is write cycles. In the past Flash media was good for less than a million writes or so. This was completely unsatisfactory for most uses as an operating system media because systems generally use the hard drive as a swap, or short term place to store programs and data when they're not busy. In an active system the swap memory can be rewritten thousands of times a day, as programs are swapped into and out of memory very actively.
There are operating systems that don't function in this way. Certainly this process was necessary when computers had little memory, but with cheap PCs able to handle eight or sixteen gigabytes of memory, but it doesn't seem to be as necessary as it once was. Choosing to avoid the issue in this way does limit your choice of OS, but not horribly so.
Recent advances in flash memory have extended the life of the flash memory into hundreds of millions, or more, of write cycles. If you're willing to accept a lifespan of five to ten years for your flash memory, you should be fine with what's available now. Certainly we can expect this trend to continue. Flash will get more durable life cycles until they finally are good long after one would normally consider them obsolete.
Currently available flash media (December 2006) comes in sizes up to 32GB for Secure Digital media. That's a lot of memory for a card that small, and it comes at a premium price. When I first bought a flash drive the largest available size was 64MB, and it was only a few years ago. Since sizes have increased five hundred times in just a few years, and manufacturers are even now working on several generations of denser media. Since a large operating system install should be no more than 8GB currently, and that size is available, Flash Media has cleared the hurdle of being large enough to handle the job. Although larger form factors are available now for 32GB media, the cost of the flash chips is enough to prevent a large market for the devices and so they're not yet common except for those who have no budget constraints.
Speed is another issue when considering Flash media for your system drive. Although Flash currently can be much slower than HDD media, that is changing as ever more chips are added and accessed in parallel. Already you can get media that reads and writes faster than an ATA HDD. Soon they will be much faster. More importantly, since Flash has no moving parts there is no latency to speak of and every file is as close as any other. This simplifies much of the disk access process and makes file access much faster. Speed is about to cease to be an issue for Flash media, and already for most uses it's faster than a hard drive. With Flash a cache might be necessary for some applications, but different from HDD media, the amount of time it takes to flush the write cache is predictable and controllable from the system rather than the controller or on-drive electronics. This makes shutdown issues go away almost entirely.
Power issues are important for storage media -- not only for battery life but also for heat. Here Flash has long been a big winner for cameras, smart phones and PDA's. Because Flash is a static medium, no energy is required to maintain the data stored within or make it accessible. There is no spin up time, no idle power at all. If you're not writing to or reading from it, it uses no power. Naturally devices that use no power generate no heat. Even in their most power hungry use, writing, Flash media doesn't take as much energy as HDD media at idle. The heat issue is an important one because the more thermal energy a device dissipates, the larger it must be to passively cool, and active cooling adds energy costs, size and noise as well.
Flash media has no moving parts. It is utterly silent. This by itself makes it a preferred medium for applications like fanless computers in audio recording environments and low energy entertainment center platforms where even small amounts of fan noise are unacceptable.
There are already many distributions of Linux that can be installed to Flash media. Soon this will be a standard install option across nearly all distributions. Installing to Flash media can be very handy for workstations. Rather than evolve a fancy network system for maintaining each user's settings in a portable way -- that doesn't work if the network is down or the user is offsite, the user can just extract his boot chip and take it with him. Then he can arrive with his full toolset and get to work without worrying about which applications don't install correctly at the new station or aren't integrated properly or don't have his preferred settings.
Price is often an issue. Flash media costs many times what Hard Drives cost for the same amount of storage. Today I can buy a 2GB SD media for about $30.00. It would take 250 of those to make the volume of a 500GB HDD that costs $150.00. For this article however I'm not talking about a volume for storing your media or your database. 8GB should be plenty right now for an operating system and suite of applications for normal use, and that can be had for a reasonable price so I think Flash has cleared the affordability constraint as an operating system install medium.
For these reasons I think we're approaching the day when a Flash Drive is a part of the motherboard on desktop and server systems for the purpose of OS and applications install at least as an option. It seems likely to become the standard for media center PCs as well. I also expect to see more support for installing to this sort of media. It seems reasonable that advances in Flash media will continue to outpace progress in other areas of information technology. In addition for highly portable devices like smartphones richer suites of applications should soon be available to exploit the advantages of having larger static storage available.
Some consolidation in the Flash memory industry seems likely as well. Highly competetive markets like this one erode profit margins and depreciate inventory disruptively. Obviously in this environment purchasing a competitor can be cheaper and more effective in the long term than inventing a new process that increases the storage density or speed of your product line.
Subscribe to:
Posts (Atom)