Librenix
Headlines | Linux | Apps | Coding | BSD | Admin | News
Information for Linux System Administration 

No Starch Press has published my Perl One-Liners book!

Up
vote
Down

My Perl One-Liners book was just published by No Starch Press!

Perl one-liners are small and awesome Perl programs that fit in a single line of code. They do one thing really well—like changing line spacing, numbering lines, performing calculations, converting and substituting text, deleting and printing specific lines, parsing logs, editing files in-place, calculating statistics, carrying out system administration tasks, or updating a bunch of files at once. Perl one-liners will make you a shell warrior: what took you minutes (or even hours) to solve will now take you only seconds! read more...
mail this link | permapage | score:9961 | -pkrumins, March 24, 2014

Tutorial: Create an NFS-like Storage Server with GlusterFS on Ubuntu 12.10

Up
vote
Down

This tutorial shows how to set up a standalone storage server on Ubuntu 12.10. Instead of NFS, I will use GlusterFS here. The client system will be able to access the storage as if it was a local filesystem. GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86_64 servers with SATA-II RAID and Infiniband HBA. read more...
mail this link | permapage | score:9737 | -falko, January 16, 2013

Unix: Shell Script Wrapper Examples

Up
vote
Down

Shell script wrappers can make the *nix command more transparent to the user. The most common shell scripts are simple wrappers around third party or system binaries. A wrapper is nothing but a shell script that includes a system command or utility.

Linux and Unix like operating system can run both 32bit and 64bit specific versions of applications. You can write a wrapper script that can select and execute correct version on a 32bit or 64bit hardware platform. In cluster environment and High-Performance computing environment you may find 100s of wrapper scripts written in Perl, Shell, and Python to get cluster usage, setting up shared storage, submitting and managing jobs, backups, troubleshooting, invokes commands with specified arguments, sending stdout to stdout and stderr to stderr and much more.

In this post, I will explains how to create a shell wrapper to enhance the basic troubleshooting tool such as ping and host. read more...
mail this link | permapage | score:9695 | -nixcraft, June 24, 2012

OpenShift Online: a non-developer guide

Up
vote
Down

OpenShift Online is a developer-targeted service, but this article is aimed at non-developers, just to highlight the aspect of the service you can take advantage of. read more...
permapage | score:9661 | -finid, March 31, 2013

e-book: Sed One-Liners Explained

Up
vote
Down

I love writing about programming and I am happy to announce my second e-book called "Sed One-Liners Explained".

Sed one-liners are short sed scripts for everyday situations in the shell, such as changing line spacing, numbering lines, and converting and deleting text.

For example, the following sed one-liner numbers the lines of a file:

sed = file | sed 'N; s/n/: /'

Here is how it works - it's made out of two sed commands. The first one uses the = command that inserts a line containing the line number before every original line in the file. Then this output gets piped to the second sed command that joins two adjacent lines with the N command. When joining lines with the N command, a newline character n is placed between them. Therefore it uses the s command to replace this newline n with a colon followed by a space ": ".

The e-book is 98 pages long and it explains exactly 100 one-liners. It's divided into the following chapters:

Preface.
1. Introduction to sed.
2. Line Spacing.
3. Line Numbering.
4. Text Conversion and Substitution.
5. Selective Printing of Certain Lines.
6. Selective Deletion of Certain Lines.
7. Special sed Applications.
Appendix A. Summary of All sed Commands.
Appendix B. Addresses and Ranges.
Appendix C. Debugging sed Scripts with sed-sed.
Index.

Did you know that sed was as powerful as any other programming language? Someone even wrote Tetris in it.

After you read the e-book, you'll be able to write your own Tetris if you wanted to. read more...
mail this link | permapage | score:9658 | -pkrumins, September 19, 2011

Tutorial: Android app build environment with Eclipse, PhoneGap (Ubuntu 11.04)

Up
vote
Down

This tutorial describes how you can set up an development environment for building Android apps on an Ubuntu 11.04 desktop using Eclipse, the Android SDK, and PhoneGap. I will describe how to build Android apps from the command line with PhoneGap and from the GUI with Eclipse and PhoneGap and how to test them in an Android emulator and on a real Android device. PhoneGap allows you to develop your Android applications using web technologies such as HTML, CSS, and JavaScript (e.g. with JavaScript libraries such as jQuery/jQTouch), and it will turn these web apps into native Android apps (in fact, PhoneGap supports multiple platforms such as Android, iPhone, Palm, Windows Mobile, Symbian, so you can use the same sources to create apps for multiple platforms). read more...
mail this link | permapage | score:9643 | -falko, June 28, 2011

Using Git for Source Control

Up
vote
Down

GIT is the source control tool, a distributed version control system (dvcs) which is written in C. This provides a history of the files that are maintained by it. In distributed version control each user has a complete copy of the code so there is no central code repository. In this scenario an administrator makes changes, adds them to the index (called staging) and then adds them to the repository (called commit). Git will take this information and maintain a version history that users can track. This is all performed locally but could be synchronized with a remote repository. read more...
mail this link | permapage | score:9635 | -aweber, May 5, 2012

Add Video Streaming to C/C++ apps with Nex Gen Media Server API

Up
vote
Down

Recently I took a closer look at Nex Gen Media Server and their API framework. NGMS is a multi-purpose streaming server which supports some of the popular streaming protocols such as RTSP, RTMP, Apple's HTTP Live, and MPEG-2 Transport Stream. NGMS comes with transcoding support and is able to capture and reformat live video streams and adapt them to be received by another type of device, such as capturing an HD video feed and converting it to be received by an iPhone over 3g. My focus was to integrate the NGMS API to control the streaming features directly from my own C application. In this example I am using Ubuntu Linux 10.04. read more...
mail this link | permapage | score:9610 | -falko, November 17, 2011

Develop your own Raspberry Pi OS

Up
vote
Down

Why use an existing operating system if you can bake your own? Well, for one thing, writing an OS is not an easy feat. It requires a lot of knowledge in a lot of areas regarding OS development. read more...
permapage | score:9606 | -finid, September 7, 2012

Detailed Error Handling In Bash

Up
vote
Down

Shell scripts are often running as background processes, doing useful things without running in a visible shell. To write such scripts can be quite painful, as all errors occur out of sight as well. While log files can hold a lot of information, finding the relevant information is a bit trickier. My solution is to log only the errors with all the details to a small database. This database contains tables for the message, the corresponding stack trace and the important environment variables. I have chosen for an SQLite database in this howto, but the same principle works with other databases as well. read more...
mail this link | permapage | score:9606 | -falko, February 21, 2013

E-book: Perl One-Liners Explained

Up
vote
Down

I'm happy to announce my 3rd e-book called "Perl One-Liners Explained."

Perl one-liners are small and awesome Perl programs that fit in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing files in-place, doing statistics, carrying out system administration tasks, updating a bunch of files at once, and many more.

Here is an example. Suppose you quickly need to generate a random, 8 character password. You can do it quickly with this Perl one-liner:

perl -le 'print map { ("a".."z")[rand 26] } 1..8'

Overall, the e-book has 111 pages and it explains 130 unique one-liners. Many of one-liners are presented in several different ways so the total number of one-liners in the book is over 200. read more...
mail this link | permapage | score:9595 | -pkrumins, February 4, 2012

Python Client/Server Tutorial

Up
vote
Down

A tiny Python tutorial...
This application can easily be coded in Python with performance levels of thousands of transactions per second on a desktop PC. Simple sample programs for the server and client sides are listed below, with discussions following
read more...
permapage | score:9546 | -Ray, June 22, 2009

Tutorial: Linux game programming with Ogre 3D

Up
vote
Down

This tutorial starts at the beginning with opening a window...
This tutorial series steps you through the process of creating a 3D shoot'em'up game using the popular and powerful Ogre 3D engine. The tutorials compile on both Windows and Linux.
read more...
permapage | score:9538 | -Ray, January 1, 2010

Better Grails apps with CSS

Up
vote
Down

This article focuses on dramatically changing the look and feel of a Grails application. With a few lines of CSS, you can change colors, fonts, and the spacing around block elements. Through partial templates and TagLibs, you can create some reusable snippets of code. In the end, you have all the benefits of the Grails framework and an application that has its very own look and feel, that looks nothing like an out-of-the-box Grails application you have ever seen.

(check out this dramatic canvas prints) read more...
mail this link | permapage | score:9530 | -solrac, January 22, 2009 (Updated: April 24, 2012)

Expect Script Examples

Up
vote
Down

Expressions, if statements, for loops, and while loops examples are covered in this mini-tutorial:
This article explains the following in the expect scripting language.
  • Expressions – arithmetic operation
  • if construct in expect
  • looping constructs
read more...
permapage | score:9507 | -Ray, January 21, 2011

Apache2, mod_rewrite tutorial: Redirect requests by device

Up
vote
Down

Since the massive rise of smartphones and tablets like the iPhone, iPad, Android phones and tablets, BlackBerries, etc. you might have considered creating a mobile version of your web site. This tutorial explains how to configure Apache to serve the mobile version of your web site if the visitor uses a mobile device, and the normal version if the visitor uses a normal desktop PC. This can be achieved with Apache's rewrite module. read more...
permapage | score:9498 | -falko, September 9, 2011

Dojo for Java programmers

Up
vote
Down

Many developers have strong skills in Java programming, but only limited experience in JavaScript. They can struggle with the conceptual leap from a strongly typed, object-oriented compilation language to a dynamic, weakly typed scripting language. This article helps you bridge the gap from Java code to Dojo, shows why it may be necessary to set context, and describes how to go about it. read more...
permapage | score:9474 | -jmalasko, October 21, 2008

Forking vs. Threading

Up
vote
Down

What is Fork/Forking:
Fork is nothing but a new process that looks exactly like the old or the parent process but still it is a different process with different process ID and having it’s own memory. Parent process creates a separate address space for child. Both parent and child process possess the same code segment, but execute independently from each other.

What are Threads/Threading:
Threads are Light Weight Processes (LWPs). Traditionally, a thread is just a CPU (and some other minimal state) state with the process containing the remains (data, stack, I/O, signals). Threads require less overhead than “forking” or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. read more...
mail this link | permapage | score:9470 | -Napster, March 1, 2010

Space Tyrant: Multithreading lessons learned on SMP hardware

Up
vote
Down

There is much to report in this update of Space Tyrant. Before getting into the new features and functions, I’ll dispense with the crisis of The Bug.

For a couple of weeks, we had been noticing odd anomalies with Space Tyrant (ST) running on the virtual server at Ioresort.com (now offline -Ed.). We never saw the problem on any other box -- and it was tested on at least four other Linux boxes and a Mac OS X system. We did all manner of stress testing, locally and over the Internet, script based and even feeding the game the output of /dev/random. Nothing caused the anomaly on any other box.

At first, I suspected that it might just be an obscure problem with the virtual server itself; after all, I had been forced to modify the TLR code to get it to run properly there. That problem turned out to be merely a limitation of NFS, not a bug with the virtual server software. However, the environment was clearly different from any other system I had used which raised my suspicions -- and reduced my urgency about looking for the bug.

While the bug wasn’t frequent, it was persistent. The bug appeared to be related to corrupted buffers or corrupted buffer indexes. Out of idle curiosity, I lowered the number of buffers used by ST to see if that affected the bug. Somewhat counter-intuitively, it substantially raised the frequency of the problem.

Brian Estabrooks (the hero of this release) and I spent more and more of our efforts hunting this incredibly elusive bug until that was all we were doing. I implemented various diagnostic routines hunting for clues. The all seemed to point to buffer indexes being changed incorrectly. Both Brian and I audited the code. It seemed impossible for the indexes to be changed improperly. Brian even went so far as to replace the ring buffer scheme with a high watermark approach but to no avail.

While I continued to suspect it to be a simple logic error in the code, Brian turned his efforts elsewhere. What he came up with was quite interesting. It seems that on many hardware architectures (most? all?), modifying a bit field can temporarily modify other bit fields in the same word! Now, this isn’t a problem on a single-CPU system; it repairs the damage in the same operation, making it, effectively, atomic. On an SMP machine, however, two different CPU’s working on different bit fields of the same word simultaneously create havoc. The operation isn’t really atomic and it doesn’t work.

Did I mention that the virtual server is a 4-way Xeon system?

The ring buffer indexing in ST relies on unsigned integer bit fields to automate wrapping back around to the first buffer after using the last one. My parsimonious programming, of course, packed all the bit fields together, several to a word. Brian’s test version of ST added a pad after each buffer index to round it out so that each bit field lived alone in its own complete word. We abused the new version for nearly an hour before either of us would dare say it. The bug was gone.

Yay!

So, the moral of this story is: Operations on sub-word fields affect other bits in that word (at least on many hardware architectures). Tread very carefully if multiple threads are accessing different bits in shared words. It may appear to work perfectly, only to crumble into a pile of smoldering rubble the first time it's loaded on a multiple CPU system!

Other than the primary lesson, some other good things came out of (the search for) the bug. Several other latent bugs were found and fixed and Brian and I are both much more intimate with the code.

And, on to the enhancements. ST is starting to look like an actual playable game. The following functions implement the new major features.

players(): We now have player rankings. It works by adding all the players’ ship resources to an integer array. Then it scans the universe looking for deployed fighters and adds those to the array as well. Currently, those two items comprise the total strength of a player.

It then sorts the array with a recursive bit-plane sort that I wrote for Starship Traders in 1998. The qsort() function in the C library was plenty fast, but took too much memory for my taste. Memory was a bit scarcer in those days and, worse, the SST software model gave each player his own copy of the server.

The sort reorders the array in place as follows. It scans the high-order bit in each element of the array. It then moves all elements starting with ‘1’ bits to the top and all starting with ‘0’ bits to the bottom. Next, it calls itself twice to reorder the first and second chunks of the array on the second bit. Each of those two instances of the sort then call the sort twice again, now giving 4 new sorts for the third bit, and so on. When all 32 bits are accounted for, the array is in the correct order with the top player on top, etc.

Scanning the entire universe can be expensive with a large map. Therefore, the player rankings function keeps the result and time stamps it. If another player asks for a player ranking within five seconds, the system just gives them the old one. After five seconds, however, any new request triggers a fresh listing.

autopilot(): We’ve added an autopilot to let a player find a specific sector -- or to locate the nearest planet. If you type a ‘0’ (zero), you’ll be prompted for a sector number within 1000 of the sector you’re currently in. You then will have the option of pressing ‘/’ to automatically warp to the destination sector.

If you’re looking for a planet, type the ‘L’ command that you would normally use to land on a planet in your sector. In the absence of a planet, the L key will engage the autopilot which will search for the nearest planet and give you a ‘/’ command to autowarp to it.

The new autopilot function consists of two other functions in addition to autopilot(), which is merely a control function. I had intended to use the old shortest path algorithm function from TLR but it was big and complicated. I decided to try to write a simpler, recursive shortest path algorithm instead. The new recursive function is much simpler but not quite as efficient as the giant for loop in TLR.

The actual algorithm is implemented in two functions called pathdepth() and pathcalc(). The pathdepth() function repeatedly calls pathcalc() with an increasing ‘depth’ parameter. ‘Depth’ tells pathcalc() how many levels deep to search before giving up.

The pathcalc() function simply looks to see if the sector it is looking at is the target sector. If not, it calls itself for each new sector that the current sector connects to. If the current sector is the target sector, it starts filling in an array for the autowarp() function to follow to reach the target sector. As the previous recursive calls to the pathcalc() function exit, they fill in the remainder of the path array.

And, yes, I seem to like reinventing the wheel. ;-)

The other interesting addition to the code is the backup thread. It is implemented by a function called backupdata() and works as follows: It scans the player data, the map data, and the history data looking for ‘dirty’ flags. (Whenever any persistent data is changed anywhere in the game, a dirty flag is set to tell the backup thread to write it out to disk.) This process is quite fast for a small game, but for a game with millions of sectors, it’s a significant waste of resources to scan the dirty flag array frequently.

Therefore, for the map and history data, I’ve implemented a ‘dirty block’ scheme as well. When a dirty flag is set, its corresponding dirty block flag is set too. Then, the backup thread need only scan the dirty block arrays, typically only about one percent the size of the arrays it represents. When a dirty block is found, only the hundred or so records it points to are scanned to find the actual dirty records for backup.

The backup file, named ‘st.9999.dat’ -- where ‘9999’ varies with the port number you run the game on -- goes into the current working directory from where you start the daemon. If the file doesn’t exist, a new game is started. Also, if you’ve modified the game in a way that changes the size of the data -- by increasing the map size, for example -- it will start a new game upon startup.

The game can be shut down from the command line by sending a signal 15 (kill -15 pid) or by the admin with the ^ command. Note that the first player to create an account in a new game automatically becomes the admin of the game!

makehistory(): The storing of historical data is new as well. Whenever another player attacks your ship while you’re logged off, you’ll get a report of the action and any losses when you next log on. Also, for remote deployed fighters, you never get immediate notification, so that information is stored in the history log even if you're logged on when it happens. You can view any accumulated event information since your login time by pressing the ‘e’ key.

deploy(): This simple function allows a player to deploy, or retrieve, guard fighters in a sector. Those fighters will not let another player pass through or view any of the contents of that sector. Any ships parked under the fighters are automatically protected against all attacks except for an attack by the fighters’ owner. Once the fighters are destroyed, of course, all ships there are visible and can be attacked.

There is also a newly implemented time limit in the game to limit the total online time of a day’s sessions to 4 hours. Like most other parameters, it can be changed by modifying a #define statement near the top of the code.

command(): The help page, a menu of available commands that a player can perform, has been redesigned and rewritten. This menu is attached to the '?' key.

The old debugger thread is gone, replaced by an in-game command function called showdata(). Press the ‘z’ key to see information on buffers, buffer indexes, and the backup thread’s state and history. Only if you’re serious about modifying the code will this information be useful.

The section of the gameloop thread that broadcasts radio and news messages has been modified to show only one of each type of message per pass. That way, replaying a long radio history won’t flood the output buffers and longer radio and news histories can therefore be retained.

The old jumprtn() movement function has been consolidated into the warprtn() function. It’s only slightly more complicated than having them separate.

The current source code can be downloaded from http://librenix.com/st/st.158.c. and the original article in this series is here. As usual, the compile script is embedded in the comments at the top of the source file. You’ll have to rename the source st.c for the script to work unchanged.

[A Space Tyrant home page has been created as a central index to the various ST articles, links, and files.]
mail this link | permapage | score:9343 | -Ray, June 26, 2005 (Updated: July 26, 2008)

Perl One-Liners Explained: Handy Regular Expressions

Up
vote
Down

This is the seventh part of a nine-part article on Perl one-liners.

Perl one-liners are short programs that do one and only one task well and they fit on a single line in the shell.

Perl is not Perl without regular expressions, therefore in this part I come up with and explain various Perl regular expressions. Please see part one for the introduction of the series.

This part explains the following regular expressions:
  • Match something that looks like an IP address.
  • Test if a number is in range 0-255
  • Match an IP address
  • Check if the string looks like an email address
  • Check if the string is a decimal number
  • Check if a word appears twice in the string
  • Increase all numbers by one in the string
  • Extract HTTP User-Agent string from the HTTP headers
  • Match printable ASCII characters
  • Match text between two HTML tags
  • Replace all bold tags with strong tag
  • Extract all matches from a regular expression
read more...
mail this link | permapage | score:9329 | -pkrumins, November 11, 2011
More coding articles...
Buy Art Online with a Free Return Policy

coding headlines

Unix signals list

Tutorial: Android development environment on Fedora 14

Open source Cloud Computing with PHP and MySQL

Scripting: Bash Array Tutorial

Tutorial: Build a C/C++ memory manager

Linux Kernel Debugging Tools

perl1line.txt: A handy Perl script collection

Build Apps with Android SDK, Eclipse, PhoneGap (Ubuntu 10.10)

Tutorial: Creating graphics with PHP

Eclipse: Use gdb from CDT

Tutorial: Build an Arduino laser game

DWR Java Ajax: User Interface (pdf)

Introduction to Perl one-liners

Programming the Sony PS3 SPE cores under Linux

perl dispatch table examples

Online Ruby Interpreter

Scripting: Put a clock in your bash terminal

C Source Code Example: Multithreaded RPC Server

Space Tyrant Index Page: Linux game server development project

Python for Kids

Quick and Dirty Vimdiff Tutorial

Space Tyrant: A multiplayer network game for Linux

Space Tyrant: A threaded game server project in C

Install FB4Linux in Eclipse

Vim Plugins: matchit.vim

Pattern matching in shell scripting

JSEclipse Tutorial: A JavaScript Eclipse Tool

Tutorial: Linux Dialog Boxes

Tutorial: Write a PHP wiki

Tutorial: UDP socket based client server C programs

Space Tyrant: A threaded C game project: First Code

Why software sucks

A multithreaded web server for Linux (source code)

String matching in regular expressions

Mono-culture and the .NETwork effect

Why Programmers are not Software Engineers

Sed One-Liners Explained

Mac Shell Scripting Tutorial

The Real Microsoft Monopoly

Compile Crystal Space in Ubuntu 10.04

 

Firefox sidebar

Site map

Site info

News feed

Features

Login
(to post)

Search

 
Articles are owned by their authors.   © 2000-2012 Ray Yeargin