How to install Ubuntu Linux on the decTOP SFF computer
I recently bought a decTOP small form factor (SFF) computer. My goal was to build a cheap, fanless, quiet, and low power consumption Linux server. For $99 plus the cheapest available shipping, $40, my machine arrived 11 days after I placed the order.
This is a tiny computer, about the size of a Mac Mini. But, because it has no fan, it runs a bit quieter and, with the help of a 1-watt, 366 MHz CPU, consumes only 8 watts. For comparison, the G4 Mac Mini consumes about 20-30 watts, depending on load.
The decTOP comes with 128 MB of RAM in its sole SO-DIMM slot and a 10 GB 3.5 inch hard drive. I understand that it's a simple matter to replace the drive and to upgrade the memory to a maximum of 512MB.
It also comes with no operating system and the ability to boot only from a USB drive. This article details the steps I used to build the USB boot/installation drive and install Ubuntu 6.06 on the decTOP.
There is another article -- with additional decTOP links -- here on installing Ubuntu 6.06 on the decTOP with the aid of a Windows system. Fortunately ;), I run Mac OS X and Linux (Ubuntu 7.04), so that article didn't work for me. I did the installation of the Ubuntu 6.06 LTS Server Edition using my Ubuntu Linux box and a 1 GB USB flash drive -- although a 512 MB USB drive should work as well.
- Download the Ubuntu 6.06 server ISO image from the Ubuntu download page. Depending on your plans for the decTOP, you might want to choose the desktop version. Unless you have already upgraded your decTOP's memory, however, you'll want to stick with the 6.06 releases.
- Install the mbr, mtools, and syslinux packages on the Linux system you'll be using to prepare the USB drive. If you run Ubuntu or some other Debian-derived system, the following commands may do the work for you.
apt-get install mbr
apt-get install mtools
apt-get install syslinux
- Partition the USB drive with a single FAT-16 partition. I used the fdisk 'n' command to make the new primary partition 1. The fdisk 't' command can be used to change the partition type to FAT-16. My device name was /dev/sda.
- Make the FAT-16 partition the active partition. I used the fdisk 'a' command.
- Install a master boot record on the USB drive.
- Install syslinux on the USB drive. Note that the USB drive should not be mounted when you do this.
syslinux -s /dev/sda1
- Create a mountpoint and mount the ubuntu ISO image using the loopback device.
mount -o loop -t iso9660 ubuntu.iso /iso
- Create a mountpoint and mount the USB flash drive.
mount /dev/sda1 /usb
- Copy the contents of the ISO image to the USB drive. This will take some time.
cp -r . /usb/
- Copy the /usb/dists/dapper directory into a new /usb/dists/stable directory.
cp -r dapper/* stable
- Copy several files from /usb/install to the /usb root directory.
cp /usb/install/vmlinuz /usb/
cp /usb/install/mt86plus /usb/
cp /usb/install/initrd.gz /usb/
- Install the following text into a file named syslinux.cfg in the /usb root directory.
append initrd=initrd.gz ramdisk_size=24000 root=/dev/ram rw
- Flush all writes, unmount, and remove the USB drive. After the sync step, wait for all of the data to be written to the USB drive.
- Connect the ethernet adapter to the decTOP and connect it to your network to allow automatic configuration of the network interface.
- Insert the USB drive into the decTOP and power it up. The decTOP should automatically boot from the USB drive and start the Ubuntu installation.
- Answer only the first two questions concerning language selection and go to the next step, below.
- Press Alt-F2 (hold down the Alt key and press the F2 function key) to open a shell. Then press enter to start the shell.
- Create a /cdrom and a /dev/cdroms directory in the installation ramdisk
mkdir /cdrom /dev/cdroms
- Go to the /dev/cdroms directory and build a symlink from /dev/sda1 (that is likely the device name of your USB boot partition) to /dev/cdroms/cdrom0.
ln -s ../sda1/cdrom0
- While still in the shell, mount the USB drive to mimic an installation CD-ROM.
mount -t vfat /dev/cdroms/cdrom0 /cdrom
- Return to the installation program with Alt-F1 and continue the installation.
From this point, the process should be identical to a routine CD-ROM installation.
For a grand total of $140 and 8 watts of power consumption, I now have a near-silent Linux server running 24/7. You can telnet to it here and marvel at its blinding speed running a 250,000-sector Space Tyrant game.
|mail this link | permapage | score:9290 | -Ray, August 16, 2007 (Updated: April 26, 2011)|
Librenix T-Shirts and Coffee Mugs!
|For today's example of my (semi)elite C programming skilz, I submit for your inspection the Librenix T-Shirts! Yes, I created the images on these shirts and coffee mugs entirely with C code. While the code isn't up to the standards *cough* of my open source Space Tyrant project, at least the output is colorful and not entirely textual!|
click either image to see the T-Shirts, Coffee Mugs, etc.
(If you like the images but don't care for 'librenix' on your shirt, these same styles are available for all 50 US state names as well as with the signs of the zodiac here)
(and here are some modern prints)
|mail this link | permapage | score:9238 | -Ray, June 6, 2010 (Updated: May 13, 2014)|
Space Tyrant: Multithreading lessons learned on SMP hardware
|There is much to report in this update of Space Tyrant. Before getting into the new features and functions, I’ll dispense with the crisis of The Bug. |
For a couple of weeks, we had been noticing odd anomalies with Space Tyrant (ST) running on the virtual server at Ioresort.com (now offline -Ed.). We never saw the problem on any other box -- and it was tested on at least four other Linux boxes and a Mac OS X system. We did all manner of stress testing, locally and over the Internet, script based and even feeding the game the output of /dev/random. Nothing caused the anomaly on any other box.
At first, I suspected that it might just be an obscure problem with the virtual server itself; after all, I had been forced to modify the TLR code to get it to run properly there. That problem turned out to be merely a limitation of NFS, not a bug with the virtual server software. However, the environment was clearly different from any other system I had used which raised my suspicions -- and reduced my urgency about looking for the bug.
While the bug wasn’t frequent, it was persistent. The bug appeared to be related to corrupted buffers or corrupted buffer indexes. Out of idle curiosity, I lowered the number of buffers used by ST to see if that affected the bug. Somewhat counter-intuitively, it substantially raised the frequency of the problem.
Brian Estabrooks (the hero of this release) and I spent more and more of our efforts hunting this incredibly elusive bug until that was all we were doing. I implemented various diagnostic routines hunting for clues. The all seemed to point to buffer indexes being changed incorrectly. Both Brian and I audited the code. It seemed impossible for the indexes to be changed improperly. Brian even went so far as to replace the ring buffer scheme with a high watermark approach but to no avail.
While I continued to suspect it to be a simple logic error in the code, Brian turned his efforts elsewhere. What he came up with was quite interesting. It seems that on many hardware architectures (most? all?), modifying a bit field can temporarily modify other bit fields in the same word! Now, this isn’t a problem on a single-CPU system; it repairs the damage in the same operation, making it, effectively, atomic. On an SMP machine, however, two different CPU’s working on different bit fields of the same word simultaneously create havoc. The operation isn’t really atomic and it doesn’t work.
Did I mention that the virtual server is a 4-way Xeon system?
The ring buffer indexing in ST relies on unsigned integer bit fields to automate wrapping back around to the first buffer after using the last one. My parsimonious programming, of course, packed all the bit fields together, several to a word. Brian’s test version of ST added a pad after each buffer index to round it out so that each bit field lived alone in its own complete word. We abused the new version for nearly an hour before either of us would dare say it. The bug was gone.
So, the moral of this story is: Operations on sub-word fields affect other bits in that word (at least on many hardware architectures). Tread very carefully if multiple threads are accessing different bits in shared words. It may appear to work perfectly, only to crumble into a pile of smoldering rubble the first time it's loaded on a multiple CPU system!
Other than the primary lesson, some other good things came out of (the search for) the bug. Several other latent bugs were found and fixed and Brian and I are both much more intimate with the code.
And, on to the enhancements. ST is starting to look like an actual playable game. The following functions implement the new major features.
players(): We now have player rankings. It works by adding all the players’ ship resources to an integer array. Then it scans the universe looking for deployed fighters and adds those to the array as well. Currently, those two items comprise the total strength of a player.
It then sorts the array with a recursive bit-plane sort that I wrote for Starship Traders in 1998. The qsort() function in the C library was plenty fast, but took too much memory for my taste. Memory was a bit scarcer in those days and, worse, the SST software model gave each player his own copy of the server.
The sort reorders the array in place as follows. It scans the high-order bit in each element of the array. It then moves all elements starting with ‘1’ bits to the top and all starting with ‘0’ bits to the bottom. Next, it calls itself twice to reorder the first and second chunks of the array on the second bit. Each of those two instances of the sort then call the sort twice again, now giving 4 new sorts for the third bit, and so on. When all 32 bits are accounted for, the array is in the correct order with the top player on top, etc.
Scanning the entire universe can be expensive with a large map. Therefore, the player rankings function keeps the result and time stamps it. If another player asks for a player ranking within five seconds, the system just gives them the old one. After five seconds, however, any new request triggers a fresh listing.
autopilot(): We’ve added an autopilot to let a player find a specific sector -- or to locate the nearest planet. If you type a ‘0’ (zero), you’ll be prompted for a sector number within 1000 of the sector you’re currently in. You then will have the option of pressing ‘/’ to automatically warp to the destination sector.
If you’re looking for a planet, type the ‘L’ command that you would normally use to land on a planet in your sector. In the absence of a planet, the L key will engage the autopilot which will search for the nearest planet and give you a ‘/’ command to autowarp to it.
The new autopilot function consists of two other functions in addition to autopilot(), which is merely a control function. I had intended to use the old shortest path algorithm function from TLR but it was big and complicated. I decided to try to write a simpler, recursive shortest path algorithm instead. The new recursive function is much simpler but not quite as efficient as the giant for loop in TLR.
The actual algorithm is implemented in two functions called pathdepth() and pathcalc(). The pathdepth() function repeatedly calls pathcalc() with an increasing ‘depth’ parameter. ‘Depth’ tells pathcalc() how many levels deep to search before giving up.
The pathcalc() function simply looks to see if the sector it is looking at is the target sector. If not, it calls itself for each new sector that the current sector connects to. If the current sector is the target sector, it starts filling in an array for the autowarp() function to follow to reach the target sector. As the previous recursive calls to the pathcalc() function exit, they fill in the remainder of the path array.
And, yes, I seem to like reinventing the wheel. ;-)
The other interesting addition to the code is the backup thread. It is implemented by a function called backupdata() and works as follows: It scans the player data, the map data, and the history data looking for ‘dirty’ flags. (Whenever any persistent data is changed anywhere in the game, a dirty flag is set to tell the backup thread to write it out to disk.) This process is quite fast for a small game, but for a game with millions of sectors, it’s a significant waste of resources to scan the dirty flag array frequently.
Therefore, for the map and history data, I’ve implemented a ‘dirty block’ scheme as well. When a dirty flag is set, its corresponding dirty block flag is set too. Then, the backup thread need only scan the dirty block arrays, typically only about one percent the size of the arrays it represents. When a dirty block is found, only the hundred or so records it points to are scanned to find the actual dirty records for backup.
The backup file, named ‘st.9999.dat’ -- where ‘9999’ varies with the port number you run the game on -- goes into the current working directory from where you start the daemon. If the file doesn’t exist, a new game is started. Also, if you’ve modified the game in a way that changes the size of the data -- by increasing the map size, for example -- it will start a new game upon startup.
The game can be shut down from the command line by sending a signal 15 (kill -15 pid) or by the admin with the ^ command. Note that the first player to create an account in a new game automatically becomes the admin of the game!
makehistory(): The storing of historical data is new as well. Whenever another player attacks your ship while you’re logged off, you’ll get a report of the action and any losses when you next log on. Also, for remote deployed fighters, you never get immediate notification, so that information is stored in the history log even if you're logged on when it happens. You can view any accumulated event information since your login time by pressing the ‘e’ key.
deploy(): This simple function allows a player to deploy, or retrieve, guard fighters in a sector. Those fighters will not let another player pass through or view any of the contents of that sector. Any ships parked under the fighters are automatically protected against all attacks except for an attack by the fighters’ owner. Once the fighters are destroyed, of course, all ships there are visible and can be attacked.
There is also a newly implemented time limit in the game to limit the total online time of a day’s sessions to 4 hours. Like most other parameters, it can be changed by modifying a #define statement near the top of the code.
command(): The help page, a menu of available commands that a player can perform, has been redesigned and rewritten. This menu is attached to the '?' key.
The old debugger thread is gone, replaced by an in-game command function called showdata(). Press the ‘z’ key to see information on buffers, buffer indexes, and the backup thread’s state and history. Only if you’re serious about modifying the code will this information be useful.
The section of the gameloop thread that broadcasts radio and news messages has been modified to show only one of each type of message per pass. That way, replaying a long radio history won’t flood the output buffers and longer radio and news histories can therefore be retained.
The old jumprtn() movement function has been consolidated into the warprtn() function. It’s only slightly more complicated than having them separate.
The current source code can be downloaded from http://librenix.com/st/st.158.c. and the original article in this series is here. As usual, the compile script is embedded in the comments at the top of the source file. You’ll have to rename the source st.c for the script to work unchanged.
[A Space Tyrant home page has been created as a central index to the various ST articles, links, and files.]
|mail this link | permapage | score:9145 | -Ray, June 26, 2005 (Updated: July 26, 2008)|
Writing syslog messages to MySQL
Writing syslog messages to MySQL
Written by Rainer Gerhards (2005-08-02)
In this paper, I describe how to write syslog messages to a MySQL database. Having syslog messages in a database is often handy, especially when you intend to set up a front-end for viewing them. This paper describes an approach with rsyslogd, an alternative enhanced syslog daemon natively supporting MySQL. I describe the components needed to be installed and how to configure them.
In many cases, syslog data is simply written to text files. This approach has some advantages, most notably it is very fast and efficient. However, data stored in text files is not readily accessible for real-time viewing and analysis. To do that, the messages need to be in a database. There are various ways to store syslog messages in a database. For example, some have the syslogd write text files which are later feed via a separate script into the database. Others have written scripts taking the data (via a pipe) from a non-database-aware syslogd and store them as they appear. Some others use database-aware syslogds and make them write the data directly to the database. In this paper, I use that "direct write" approach. I think it is superior, because the syslogd itself knows the status of the database connection and thus can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish this, simply because I have initiated the rsyslog project with database-awareness as one goal.
One word of caution: while message storage in the database provides an excellent foundation for interactive analysis, it comes at a cost. Database i/o is considerably slower than text file i/o. As such, directly writing to the database makes sense only if your message volume is low enough to allow a) the syslogd, b) the network, and c) the database server to catch up with it. Some time ago, I have written a paper on optimizing syslog server performance. While this paper talks about Window-based solutions, the ideas in it are generic enough to apply here, too. So it might be worth reading if you anticipate medium high to high traffic. If you anticipate really high traffic (or very large traffic spikes), you should seriously consider forgetting about direct database writes - in my opinion, such a situation needs either a very specialised system or a different approach (the text-file-to-database approach might work better for you in this case).
Overall System Setup
In this paper, I concentrate on the server side. If you are thinking about interactive syslog message review, you probably want to centralize syslog. In such a scenario, you have multiple machines (the so-called clients) send their data to a central machine (called server in this context). While I expect such a setup to be typical when you are interested in storing messages in the database, I do not describe how to set it up. This is beyond the scope of this paper. If you search a little, you will probably find many good descriptions on how to centralize syslog. If you do that, it might be a good idea to do it securely, so you might also be interested in my paper on ssl-encrypting syslog message transfer.
No matter how the messages arrive at the server, their processing is always the same. So you can use this paper in combination with any description for centralized syslog reporting.
As I already said, I use rsyslogd on the server. It has intrinsic support for talking to MySQL databases. For obvious reasons, we also need an instance of MySQL running. To keep us focussed, the setup of MySQL itself is also beyond the scope of this paper. I assume that you have successfully installed MySQL and also have a front-end at hand to work with it (for example, phpMyAdmin). Please make sure that this is installed, actually working and you have a basic understanding of how to handle it.
Setting up the system
You need to download and install rsyslogd first. Obtain it from the rsyslog site. Make sure that you disable stock syslogd, otherwise you will experience some difficulties.
It is important to understand how rsyslogd talks to the database. In rsyslogd, there is the concept of "templates". Basically, a template is a string that includes some replacement characters, which are called "properties" in rsyslog. Properties are accessed via the "Property Replacer". Simply said, you access properties by including their name between percent signs inside the template. For example, if the syslog message is "Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports sending template text as a SQL statement to MySQL. As such, the template must be a valid SQL statement. There is no limit in what the statement might be, but there are some obvious and not so obvious choices. For example, a template "drop table xxx" is possible, but does not make an awful lot of sense. In practice, you will always use an "insert" statment inside the template.
An example: if you would just like to store the msg part of the full syslog message, you have probably created a table "syslog" with a single column "message". In such a case, a good template would be "insert into syslog(message) values ('%msg%')". With the example above, that would be expanded to "insert into syslog(message) values('Test')". This expanded string is then sent to the database. It's that easy, no special magic. The only thing you must ensure is that your template expands to a proper SQL statement and that this statement matches your database design.
Does that mean you need to create database schema yourself and also must fully understand rsyslogd's properties? No, that's not needed. Because we anticipated that folks are probably more interested in getting things going instead of designing them from scratch. So we have provided a default schema as well as build-in support for it. This schema also offers an additional benefit: rsyslog is part of Adiscon's MonitorWare product line (which includes open source and closed source members). All of these tools share the same default schema and know how to operate on it. For this reason, the default schema is also called the "MonitorWare Schema". If you use it, you can simply add phpLogCon, a GPLed syslog web interface, to your system and have instant interactive access to your database. So there are some benefits in using the provided schema.
The schema definition is contained in the file "createDB.sql". It comes with the rsyslog package. Review it to check that the database name is acceptable for you. Be sure to leave the table and field names unmodified, because otherwise you need to customize rsyslogd's default sql template, which we do not do in this paper. Then, run the script with your favourite MySQL tool. Double-check that the table was successfully created.
Next, we need to tell rsyslogd to write data to the database. As we use the default schema, we do NOT need to define a template for this. We can use the hardcoded one (rsyslogd handles the proper template linking). So all we need to do is add a simple selector line to /etc/rsyslog.conf:
In many cases, MySQL will run on the local machine. In this case, you can simply use "127.0.0.1" for database-server. This can be especially advisable, if you do not need to expose MySQL to any process outside of the local machine. In this case, you can simply bind it to 127.0.0.1, which provides a quite secure setup. Of course, also supports remote MySQL instances. In that case, use the remote server name (e.g. mysql.example.com) or IP-address. The database-name by default is "syslog". If you have modified the default, use your name here. Database-userid and -password are the credentials used to connect to the database. As they are stored in clear text in rsyslog.conf, that user should have only the least possible privileges. It is sufficient to grant it INSERT privileges to the systemevents table, only. As a side note, it is strongly advisable to make the rsyslog.conf file readable by root only - if you make it world-readable, everybody could obtain the password (and eventually other vital information from it). In our example, let's assume you have created a MySQL user named "syslogwriter" with a password of "topsecret" (just to say it bluntly: such a password is NOT a good idea...). If your MySQL database is on the local machine, your rsyslog.conf line might look like in this sample:
Save rsyslog.conf, restart rsyslogd - and you should see syslog messages being stored in the "systemevents" table!
The example line stores every message to the database. Especially if you have a high traffic volume, you will probably limit the amount of messages being logged. This is easy to acomplish: the "write database" action is just a regular selector line. As such, you can apply normal selector-line filtering. If, for example, you are only interested in messages from the mail subsystem, you can use the following selector line:
Review the rsyslog.conf documentation for details on selector lines and their filtering.
You have now completed everything necessary to store syslog messages to the MySQL database. If you would like to try out a front-end, you might want to look at phpLogCon, which displays syslog data in a browser. As of this writing, phpLogCon is not yet a powerful tool, but it's open source, so it might be a starting point for your own solution.
Rsyslogd writes syslog messages directly to the database. This implies that the database must be available at the time of message arrival. If the database is offline, no space is left or something else goes wrong - rsyslogd can not write the database record. If rsyslogd is unable to store a message, it performs one retry. This is helpful if the database server was restarted. In this case, the previous connection was broken but a reconnect immediately succeeds. However, if the database is down for an extended period of time, an immediate retry does not help. While rsyslogd could retry until it finally succeeds, that would have negative impact. Syslog messages keep coming in. If rsyslogd would be busy retrying the database, it would not be able to process these messages. Ultimately, this would lead to loss of newly arrived messages.
In most cases, rsyslogd is configured not only to write to the database but to perform other actions as well. In the always-retry scenario, that would mean no other actions would be carried out. As such, the design of rsyslogd is limited to a single retry. If that does not succeed, the current message is will not be written to the database and the MySQL database writer be suspended for a short period of time. Obviously, this leads to the loss of the current message as well as all messages received during the suspension period. But they are only lost in regard to the database, all other actions are correctly carried out. While not perfect, we consider this to be a better approach then the potential loss of all messages in all actions.
In short: try to avoid database downtime if you do not want to experience message loss.
Please note that this restriction is not rsyslogd specific. All approachs to real-time database storage share this problem area.
With minumal effort, you can use rsyslogd to write syslog messages to a MySQL database. Once the messages are arrived there, you can interactivley review and analyse them. In practice, the messages are also stored in text files for longer-term archival and the databases are cleared out after some time (to avoid becoming too slow). If you expect an extremely high syslog message volume, storing it in real-time to the database may outperform your database server. In such cases, either filter out some messages or think about alternate approaches involving non-real-time database writing (beyond the scope of this paper).
The method outline in this paper provides an easy to setup and maintain solution for most use cases, especially with low and medium syslog message volume (or fast database servers).
I would appreciate feedback on this paper. If you have additional ideas, comments or find bugs, please let me know.
References and Additional Material
Copyright (c) 2005 Rainer Gerhards and Adiscon.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at http://www.gnu.org/copyleft/fdl.html.
|mail this link | permapage | score:9105 | -rgerhards, August 4, 2005 (Updated: March 21, 2007)||