Librenix
Headlines | Linux | Apps | Coding | BSD | Admin | News
Information for Linux System Administration 

No Starch Press has published my Perl One-Liners book!

Up
vote
Down

My Perl One-Liners book was just published by No Starch Press!

Perl one-liners are small and awesome Perl programs that fit in a single line of code. They do one thing really well—like changing line spacing, numbering lines, performing calculations, converting and substituting text, deleting and printing specific lines, parsing logs, editing files in-place, calculating statistics, carrying out system administration tasks, or updating a bunch of files at once. Perl one-liners will make you a shell warrior: what took you minutes (or even hours) to solve will now take you only seconds! read more...
mail this link | permapage | score:9973 | -pkrumins, March 24, 2014

Tutorial: Create an NFS-like Storage Server with GlusterFS on Ubuntu 12.10

Up
vote
Down

This tutorial shows how to set up a standalone storage server on Ubuntu 12.10. Instead of NFS, I will use GlusterFS here. The client system will be able to access the storage as if it was a local filesystem. GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86_64 servers with SATA-II RAID and Infiniband HBA. read more...
mail this link | permapage | score:9970 | -falko, January 16, 2013

Python Client/Server Tutorial

Up
vote
Down

A tiny Python tutorial...
This application can easily be coded in Python with performance levels of thousands of transactions per second on a desktop PC. Simple sample programs for the server and client sides are listed below, with discussions following
read more...
permapage | score:9695 | -Ray, June 22, 2009

OpenShift Online: a non-developer guide

Up
vote
Down

OpenShift Online is a developer-targeted service, but this article is aimed at non-developers, just to highlight the aspect of the service you can take advantage of. read more...
permapage | score:9672 | -finid, March 31, 2013

Tutorial: Android app build environment with Eclipse, PhoneGap (Ubuntu 11.04)

Up
vote
Down

This tutorial describes how you can set up an development environment for building Android apps on an Ubuntu 11.04 desktop using Eclipse, the Android SDK, and PhoneGap. I will describe how to build Android apps from the command line with PhoneGap and from the GUI with Eclipse and PhoneGap and how to test them in an Android emulator and on a real Android device. PhoneGap allows you to develop your Android applications using web technologies such as HTML, CSS, and JavaScript (e.g. with JavaScript libraries such as jQuery/jQTouch), and it will turn these web apps into native Android apps (in fact, PhoneGap supports multiple platforms such as Android, iPhone, Palm, Windows Mobile, Symbian, so you can use the same sources to create apps for multiple platforms). read more...
mail this link | permapage | score:9433 | -falko, June 28, 2011

Using Git for Source Control

Up
vote
Down

GIT is the source control tool, a distributed version control system (dvcs) which is written in C. This provides a history of the files that are maintained by it. In distributed version control each user has a complete copy of the code so there is no central code repository. In this scenario an administrator makes changes, adds them to the index (called staging) and then adds them to the repository (called commit). Git will take this information and maintain a version history that users can track. This is all performed locally but could be synchronized with a remote repository. read more...
mail this link | permapage | score:9308 | -aweber, May 5, 2012

Forking vs. Threading

Up
vote
Down

What is Fork/Forking:
Fork is nothing but a new process that looks exactly like the old or the parent process but still it is a different process with different process ID and having it’s own memory. Parent process creates a separate address space for child. Both parent and child process possess the same code segment, but execute independently from each other.

What are Threads/Threading:
Threads are Light Weight Processes (LWPs). Traditionally, a thread is just a CPU (and some other minimal state) state with the process containing the remains (data, stack, I/O, signals). Threads require less overhead than “forking” or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. read more...
mail this link | permapage | score:9292 | -Napster, March 1, 2010

Develop your own Raspberry Pi OS

Up
vote
Down

Why use an existing operating system if you can bake your own? Well, for one thing, writing an OS is not an easy feat. It requires a lot of knowledge in a lot of areas regarding OS development. read more...
permapage | score:9281 | -finid, September 7, 2012

e-book: Sed One-Liners Explained

Up
vote
Down

I love writing about programming and I am happy to announce my second e-book called "Sed One-Liners Explained".

Sed one-liners are short sed scripts for everyday situations in the shell, such as changing line spacing, numbering lines, and converting and deleting text.

For example, the following sed one-liner numbers the lines of a file:

sed = file | sed 'N; s/n/: /'

Here is how it works - it's made out of two sed commands. The first one uses the = command that inserts a line containing the line number before every original line in the file. Then this output gets piped to the second sed command that joins two adjacent lines with the N command. When joining lines with the N command, a newline character n is placed between them. Therefore it uses the s command to replace this newline n with a colon followed by a space ": ".

The e-book is 98 pages long and it explains exactly 100 one-liners. It's divided into the following chapters:

Preface.
1. Introduction to sed.
2. Line Spacing.
3. Line Numbering.
4. Text Conversion and Substitution.
5. Selective Printing of Certain Lines.
6. Selective Deletion of Certain Lines.
7. Special sed Applications.
Appendix A. Summary of All sed Commands.
Appendix B. Addresses and Ranges.
Appendix C. Debugging sed Scripts with sed-sed.
Index.

Did you know that sed was as powerful as any other programming language? Someone even wrote Tetris in it.

After you read the e-book, you'll be able to write your own Tetris if you wanted to. read more...
mail this link | permapage | score:9230 | -pkrumins, September 19, 2011

C Source Code Example: Multithreaded RPC Server

Up
vote
Down

This can be considered an example, how to build multithreaded RPC Server for White Box Linux 3. Actually, approach described is universal , i.e. doesn't depend on particular code has been modified.

[Update June 26, 2005: Librenix is now featuring a C language multithreaded network socket programming game server project for Linux called Space Tyrant. As of June 26, there have been four code releases and the game is already playable from telnet.]

(here are some white canvas prints) read more...
mail this link | permapage | score:9225 | -Boris Derzhavets, May 14, 2005 (Updated: April 24, 2012)

Expect Script Examples

Up
vote
Down

Expressions, if statements, for loops, and while loops examples are covered in this mini-tutorial:
This article explains the following in the expect scripting language.
  • Expressions – arithmetic operation
  • if construct in expect
  • looping constructs
read more...
permapage | score:9212 | -Ray, January 21, 2011

E-book: Perl One-Liners Explained

Up
vote
Down

I'm happy to announce my 3rd e-book called "Perl One-Liners Explained."

Perl one-liners are small and awesome Perl programs that fit in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing files in-place, doing statistics, carrying out system administration tasks, updating a bunch of files at once, and many more.

Here is an example. Suppose you quickly need to generate a random, 8 character password. You can do it quickly with this Perl one-liner:

perl -le 'print map { ("a".."z")[rand 26] } 1..8'

Overall, the e-book has 111 pages and it explains 130 unique one-liners. Many of one-liners are presented in several different ways so the total number of one-liners in the book is over 200. read more...
mail this link | permapage | score:9175 | -pkrumins, February 4, 2012

Space Tyrant: Multithreading lessons learned on SMP hardware

Up
vote
Down

There is much to report in this update of Space Tyrant. Before getting into the new features and functions, I’ll dispense with the crisis of The Bug.

For a couple of weeks, we had been noticing odd anomalies with Space Tyrant (ST) running on the virtual server at Ioresort.com (now offline -Ed.). We never saw the problem on any other box -- and it was tested on at least four other Linux boxes and a Mac OS X system. We did all manner of stress testing, locally and over the Internet, script based and even feeding the game the output of /dev/random. Nothing caused the anomaly on any other box.

At first, I suspected that it might just be an obscure problem with the virtual server itself; after all, I had been forced to modify the TLR code to get it to run properly there. That problem turned out to be merely a limitation of NFS, not a bug with the virtual server software. However, the environment was clearly different from any other system I had used which raised my suspicions -- and reduced my urgency about looking for the bug.

While the bug wasn’t frequent, it was persistent. The bug appeared to be related to corrupted buffers or corrupted buffer indexes. Out of idle curiosity, I lowered the number of buffers used by ST to see if that affected the bug. Somewhat counter-intuitively, it substantially raised the frequency of the problem.

Brian Estabrooks (the hero of this release) and I spent more and more of our efforts hunting this incredibly elusive bug until that was all we were doing. I implemented various diagnostic routines hunting for clues. The all seemed to point to buffer indexes being changed incorrectly. Both Brian and I audited the code. It seemed impossible for the indexes to be changed improperly. Brian even went so far as to replace the ring buffer scheme with a high watermark approach but to no avail.

While I continued to suspect it to be a simple logic error in the code, Brian turned his efforts elsewhere. What he came up with was quite interesting. It seems that on many hardware architectures (most? all?), modifying a bit field can temporarily modify other bit fields in the same word! Now, this isn’t a problem on a single-CPU system; it repairs the damage in the same operation, making it, effectively, atomic. On an SMP machine, however, two different CPU’s working on different bit fields of the same word simultaneously create havoc. The operation isn’t really atomic and it doesn’t work.

Did I mention that the virtual server is a 4-way Xeon system?

The ring buffer indexing in ST relies on unsigned integer bit fields to automate wrapping back around to the first buffer after using the last one. My parsimonious programming, of course, packed all the bit fields together, several to a word. Brian’s test version of ST added a pad after each buffer index to round it out so that each bit field lived alone in its own complete word. We abused the new version for nearly an hour before either of us would dare say it. The bug was gone.

Yay!

So, the moral of this story is: Operations on sub-word fields affect other bits in that word (at least on many hardware architectures). Tread very carefully if multiple threads are accessing different bits in shared words. It may appear to work perfectly, only to crumble into a pile of smoldering rubble the first time it's loaded on a multiple CPU system!

Other than the primary lesson, some other good things came out of (the search for) the bug. Several other latent bugs were found and fixed and Brian and I are both much more intimate with the code.

And, on to the enhancements. ST is starting to look like an actual playable game. The following functions implement the new major features.

players(): We now have player rankings. It works by adding all the players’ ship resources to an integer array. Then it scans the universe looking for deployed fighters and adds those to the array as well. Currently, those two items comprise the total strength of a player.

It then sorts the array with a recursive bit-plane sort that I wrote for Starship Traders in 1998. The qsort() function in the C library was plenty fast, but took too much memory for my taste. Memory was a bit scarcer in those days and, worse, the SST software model gave each player his own copy of the server.

The sort reorders the array in place as follows. It scans the high-order bit in each element of the array. It then moves all elements starting with ‘1’ bits to the top and all starting with ‘0’ bits to the bottom. Next, it calls itself twice to reorder the first and second chunks of the array on the second bit. Each of those two instances of the sort then call the sort twice again, now giving 4 new sorts for the third bit, and so on. When all 32 bits are accounted for, the array is in the correct order with the top player on top, etc.

Scanning the entire universe can be expensive with a large map. Therefore, the player rankings function keeps the result and time stamps it. If another player asks for a player ranking within five seconds, the system just gives them the old one. After five seconds, however, any new request triggers a fresh listing.

autopilot(): We’ve added an autopilot to let a player find a specific sector -- or to locate the nearest planet. If you type a ‘0’ (zero), you’ll be prompted for a sector number within 1000 of the sector you’re currently in. You then will have the option of pressing ‘/’ to automatically warp to the destination sector.

If you’re looking for a planet, type the ‘L’ command that you would normally use to land on a planet in your sector. In the absence of a planet, the L key will engage the autopilot which will search for the nearest planet and give you a ‘/’ command to autowarp to it.

The new autopilot function consists of two other functions in addition to autopilot(), which is merely a control function. I had intended to use the old shortest path algorithm function from TLR but it was big and complicated. I decided to try to write a simpler, recursive shortest path algorithm instead. The new recursive function is much simpler but not quite as efficient as the giant for loop in TLR.

The actual algorithm is implemented in two functions called pathdepth() and pathcalc(). The pathdepth() function repeatedly calls pathcalc() with an increasing ‘depth’ parameter. ‘Depth’ tells pathcalc() how many levels deep to search before giving up.

The pathcalc() function simply looks to see if the sector it is looking at is the target sector. If not, it calls itself for each new sector that the current sector connects to. If the current sector is the target sector, it starts filling in an array for the autowarp() function to follow to reach the target sector. As the previous recursive calls to the pathcalc() function exit, they fill in the remainder of the path array.

And, yes, I seem to like reinventing the wheel. ;-)

The other interesting addition to the code is the backup thread. It is implemented by a function called backupdata() and works as follows: It scans the player data, the map data, and the history data looking for ‘dirty’ flags. (Whenever any persistent data is changed anywhere in the game, a dirty flag is set to tell the backup thread to write it out to disk.) This process is quite fast for a small game, but for a game with millions of sectors, it’s a significant waste of resources to scan the dirty flag array frequently.

Therefore, for the map and history data, I’ve implemented a ‘dirty block’ scheme as well. When a dirty flag is set, its corresponding dirty block flag is set too. Then, the backup thread need only scan the dirty block arrays, typically only about one percent the size of the arrays it represents. When a dirty block is found, only the hundred or so records it points to are scanned to find the actual dirty records for backup.

The backup file, named ‘st.9999.dat’ -- where ‘9999’ varies with the port number you run the game on -- goes into the current working directory from where you start the daemon. If the file doesn’t exist, a new game is started. Also, if you’ve modified the game in a way that changes the size of the data -- by increasing the map size, for example -- it will start a new game upon startup.

The game can be shut down from the command line by sending a signal 15 (kill -15 pid) or by the admin with the ^ command. Note that the first player to create an account in a new game automatically becomes the admin of the game!

makehistory(): The storing of historical data is new as well. Whenever another player attacks your ship while you’re logged off, you’ll get a report of the action and any losses when you next log on. Also, for remote deployed fighters, you never get immediate notification, so that information is stored in the history log even if you're logged on when it happens. You can view any accumulated event information since your login time by pressing the ‘e’ key.

deploy(): This simple function allows a player to deploy, or retrieve, guard fighters in a sector. Those fighters will not let another player pass through or view any of the contents of that sector. Any ships parked under the fighters are automatically protected against all attacks except for an attack by the fighters’ owner. Once the fighters are destroyed, of course, all ships there are visible and can be attacked.

There is also a newly implemented time limit in the game to limit the total online time of a day’s sessions to 4 hours. Like most other parameters, it can be changed by modifying a #define statement near the top of the code.

command(): The help page, a menu of available commands that a player can perform, has been redesigned and rewritten. This menu is attached to the '?' key.

The old debugger thread is gone, replaced by an in-game command function called showdata(). Press the ‘z’ key to see information on buffers, buffer indexes, and the backup thread’s state and history. Only if you’re serious about modifying the code will this information be useful.

The section of the gameloop thread that broadcasts radio and news messages has been modified to show only one of each type of message per pass. That way, replaying a long radio history won’t flood the output buffers and longer radio and news histories can therefore be retained.

The old jumprtn() movement function has been consolidated into the warprtn() function. It’s only slightly more complicated than having them separate.

The current source code can be downloaded from http://librenix.com/st/st.158.c. and the original article in this series is here. As usual, the compile script is embedded in the comments at the top of the source file. You’ll have to rename the source st.c for the script to work unchanged.

[A Space Tyrant home page has been created as a central index to the various ST articles, links, and files.]
mail this link | permapage | score:9159 | -Ray, June 26, 2005 (Updated: July 26, 2008)

Apache2, mod_rewrite tutorial: Redirect requests by device

Up
vote
Down

Since the massive rise of smartphones and tablets like the iPhone, iPad, Android phones and tablets, BlackBerries, etc. you might have considered creating a mobile version of your web site. This tutorial explains how to configure Apache to serve the mobile version of your web site if the visitor uses a mobile device, and the normal version if the visitor uses a normal desktop PC. This can be achieved with Apache's rewrite module. read more...
permapage | score:9146 | -falko, September 9, 2011

Tutorial: Build an Arduino laser game

Up
vote
Down

Arduino is an easy-to-use electronics platform. The entire platform, both the hardware and the software, is completely open source, and the language is loosely based on C/C++. Whether you're new to Arduino or a seasoned builder, this project has something for you. There's nothing quite as satisfying as creating an interactive physical object. Use this tutorial to create an interactive laser game called "'Duino tag," where players can play tag using devices built nearly from scratch. read more...
permapage | score:9119 | -solrac, February 2, 2009

Quick and Dirty Vimdiff Tutorial

Up
vote
Down

A quick how to for the famous Vimdiff utility. It lists the most relevant shortcuts and a screenshot that shows Vimdiff in action.
Vimdiff is a tool that comes bundled with Vim and its a wonderful tool for comparing code and merging changes.
read more...
permapage | score:9117 | -jinx_kid, September 19, 2008 (Updated: September 20, 2008)

Tutorial: Linux game programming with Ogre 3D

Up
vote
Down

This tutorial starts at the beginning with opening a window...
This tutorial series steps you through the process of creating a 3D shoot'em'up game using the popular and powerful Ogre 3D engine. The tutorials compile on both Windows and Linux.
read more...
permapage | score:9104 | -Ray, January 1, 2010

JSEclipse Tutorial: A JavaScript Eclipse Tool

Up
vote
Down

JSEclipse — an Eclipse plug-in that provides many of those same features for JavaScript applications. JSEclipse gives JavaScript developers the same ease of use that Eclipse has been providing in the Java language. This tutorial covers the JSEclipse tool, which plugs into Eclipse and provides features like code completion and templates. read more...
permapage | score:9091 | -solrac, December 23, 2007

Eclipse: Use gdb from CDT

Up
vote
Down

The GNU Debugger (gdb) is the most popular open source debugger in use. Learn how the Eclipse C/C++ Development Tooling (CDT) talks to the GNU Debugger. This concrete example of CDT-debugger interaction should be helpful for anyone wishing to interface a custom C/C++ debugger from CDT. read more...
permapage | score:9088 | -jmalasko, July 4, 2008

Space Tyrant: A threaded game server project in C

Up
vote
Down

[Update, June 25, 2005: A Space Tyrant home page has been created as a central index to the various ST articles, links, and files.]

[Update, March 21, 2007: A Space Tyrant has its own website! It's small but growing and will provide quick access to the latest code and developments in the ST universe.]

Space Tyrant: Today we kick off a new multithreaded, network socket programming project which we will call Space Tyrant. Our mission is to write an open source, multiplayer, networked, strategy game in the C programming language. The goal of this project is to make a solid code base which implements a simple space trading game upon which other games can then be built. The game will be a subset of The Last Resort (TLR) that currently runs at [offline]. This project will be a learning exercise for me as well as for any interested readers. The current state of the source code will be released with each article update.

The game design: While my TLR game consists of over 25,000 lines of C source code and supports a web interface as well as telnet and a graphical client, this code will be far smaller and simpler. It will initially only support telnet and will implement a far simpler game design.

Players will be able to telnet into the game, create an account, and play in a universe that contains ports, planets, as well as other players. Each player will be issued a starship, some cargo holds, and an amount of starship fuel. Additional fuel will be issued hourly and will accumulate in the starship. Fuel will be used to move the ship between sectors -- locations within the game universe -- and to dock with ports. Once a ship runs out of fuel it can't move at all until new fuel is issued.

Players will be able to buy and sell commodities (Iron, alcohol, and hardware) between the three different kinds of ports. Each port type will sell one of the three commodities and buy the other two. Prices will be based on supply and demand with rarely-used ports offering the better prices.

With the money players earn trading they will be able to buy more cargo holds to make their ships more efficient for trading. They will also be able to buy fighters -- small military drones -- that can be used to attack other ships or deployed to guard a sector and its contents. The fighters carried with a ship will guard it against attacks from other players.

Games will run for a predetermined length of time, then reset and start anew.

The programming model: Now, on to the software design. I've compared and considered various models for the server design. TLR is based on the forking model using inetd or xinetd to handle the listening and forking. While the forking model is inherently distributable to multiple processors, it introduces inefficiencies (forking multiple processes) and makes interprocess communications more difficult and slower.

Next, I considered a non-blocking, single process model. In this approach, one process handles everything in a single thread. It would use non-blocking IO (read and write functions that never wait for completion but, rather, return immediately if they aren't ready to read or write actual data). The thttpd web server is an example of a non-blocking, single process server. It's extremely fast and efficient. However, this model is quite complicated to code, and, I believe would make it more likely to introduce subtle timing bugs.

Next, I considered a pure multithreaded, single process model with a thread for each player. While appealing in many ways, this model would require the same kind of coordination between threads that the forking model requires between processes. Such interprocess communication would be simplified in that the various threads share memory, but the coordination issues otherwise remain the same.

Last, I considered another multithreaded model, this time with only IO threads for each user and a single thread that implements all game logic. While that one central thread might someday be a bottleneck that limits scalability on large SMP systems, it does distribute the IO on any additional processors that might be present, and requires minimal coordination. In short, this model combines the logic simplicity of the non-blocking single process model with the coding simplicity of the threaded model, while separating the IO from the main logic. There will also be two other simple threads in this model. There will be a thread that listens for new connections and spawns the IO threads for each new connection. There will also be a thread that writes the data to disk periodically.

This is the approach that I intend to take for this project. The code will be written for both Linux and Mac OS X.

More info: I have set up an email address for programmers following this series to provide recommendations, bug reports, and other feedback. Send email about this project to spacetyrant [at] librenix.com.
mail this link | permapage | score:9062 | -Ray, March 18, 2005 (Updated: July 26, 2008)
More coding articles...
Gallery Wrapped Abstract Art

coding headlines

Perl One-Liners Explained: Handy Regular Expressions

Add Video Streaming to C/C++ apps with Nex Gen Media Server API

Space Tyrant: A threaded C game project: First Code

Detailed Error Handling In Bash

Space Tyrant Index Page: Linux game server development project

Space Tyrant: A multiplayer network game for Linux

Compile Crystal Space in Ubuntu 10.04

Mono-culture and the .NETwork effect

Better Grails apps with CSS

Build Apps with Android SDK, Eclipse, PhoneGap (Ubuntu 10.10)

Scripting: Put a clock in your bash terminal

Programming the Sony PS3 SPE cores under Linux

Why software sucks

Dojo for Java programmers

Pattern matching in shell scripting

Tutorial: Socket Programming in C

Comparison: Zend PHP vs. Symfony vs. CakePHP

Why Programmers are not Software Engineers

DWR Java Ajax: User Interface (pdf)

Tutorial: Linux Dialog Boxes

Install WebSphere MQ on Linux

Review: DevelopGo: A Linux Live CD for Programmers

Online Ruby Interpreter

PHP vs. ASP

Tutorial: Write your own operating system

Programming Language Tradeoffs: 3GL vs 4GL

Tutorial: Write a Qt program with Qt Designer

A multithreaded web server for Linux (source code)

Tutorial: Install the Aptana AJAX IDE on Ubuntu

The Real Microsoft Monopoly

Unix: Shell Script Wrapper Examples

The life cycle of a programmer

Monitor a Linux service with watchdog scripting

Java J2EE Tutorial: Secret Santa Web Application

Mac Shell Scripting Tutorial

perl dispatch table examples

Tutorial: Creating graphics with PHP

Tutorial: Create a video site with PHP

Scripting: Bash Array Tutorial

Best practices for XML in Java programming

 

Firefox sidebar

Site map

Site info

News feed

Features

Login
(to post)

Search

 
Articles are owned by their authors.   © 2000-2012 Ray Yeargin