pkrumins

Information for Linux System Administration

e-book: Sed One-Liners Explained

vote

I love writing about programming and I am happy to announce my second e-book called "Sed One-Liners Explained".

Sed one-liners are short sed scripts for everyday situations in the shell, such as changing line spacing, numbering lines, and converting and deleting text.

For example, the following sed one-liner numbers the lines of a file:

sed = file | sed 'N; s/n/: /'

Here is how it works - it's made out of two sed commands. The first one uses the = command that inserts a line containing the line number before every original line in the file. Then this output gets piped to the second sed command that joins two adjacent lines with the N command. When joining lines with the N command, a newline character n is placed between them. Therefore it uses the s command to replace this newline n with a colon followed by a space ": ".

The e-book is 98 pages long and it explains exactly 100 one-liners. It's divided into the following chapters:

Preface.
1. Introduction to sed.
2. Line Spacing.
3. Line Numbering.
4. Text Conversion and Substitution.
5. Selective Printing of Certain Lines.
6. Selective Deletion of Certain Lines.
7. Special sed Applications.
Appendix A. Summary of All sed Commands.
Appendix B. Addresses and Ranges.
Appendix C. Debugging sed Scripts with sed-sed.
Index.

Did you know that sed was as powerful as any other programming language? Someone even wrote Tetris in it.

After you read the e-book, you'll be able to write your own Tetris if you wanted to. read more...

mail this link | permapage | score:9385 | -pkrumins, September 19, 2011

ssh: Use Linux as a SOCKS5 proxy

vote

Did you know you could use ssh to create a SOCKS5 server?

This article explains how to create a SOCKS5 server in 1 command and how to add ip-based access control to it via iptables and tcp forwarder. read more...

permapage | score:9219 | -pkrumins, May 7, 2010

perl1line.txt: A handy Perl script collection

vote

The ultimate goal of the Perl One-Liners Explained article series was to release the perl1line.txt file. Last week I finished the series and now I am happy to announce perl1line.txt - a collection of handy Perl one-liner scripts.

The perl1line.txt file contains over a hundred short Perl one-line scripts for various text processing tasks. The file processing tasks include: changing file spacing, numbering lines, doing calculations, creating strings and arrays, converting and substituting text, selective printing and deleting of certain lines and text filtering and modifications through regular expressions.

The latest version of perl1line.txt is always at:

http://www.catonmat.net/download/perl1line.txt

Enjoy! It took me over 3 years to write all the one-liners down. read more...

mail this link | permapage | score:8998 | -pkrumins, November 21, 2011

Awk Tips, Tricks and Pitfalls

vote

This article covers 10 Awk programming language tips, tricks and pitfalls:

1. Be idiomatic!
2. Pitfall: shorten pipelines
3. Print lines using ranges
4. Split file on patterns
5. Locale-based pitfalls
6. Parse CSV
7. Pitfall: validate an IPv4 address
8. Check whether two files contain the same data
9. Pitfall: contexts and variable types in awk
10. Pulling out things read more...

permapage | score:8991 | -pkrumins, October 24, 2008

Vim plugins: snipmate.vim

vote

This article introduces the snipmate vim plugin.

Snipmate.vim is probably the best snippets plugin for vim. A snippet is a piece of often-typed text or programming construct that you can insert into your document by using a trigger followed by a .

For example, you type "for" and press TAB, and the plugin inserts "for (i = 0; i < n; i++) { }" in your code! read more...

permapage | score:8868 | -pkrumins, August 10, 2009

Perl One-Liners Explained: Handy Regular Expressions

vote

This is the seventh part of a nine-part article on Perl one-liners.

Perl one-liners are short programs that do one and only one task well and they fit on a single line in the shell.

Perl is not Perl without regular expressions, therefore in this part I come up with and explain various Perl regular expressions. Please see part one for the introduction of the series.

This part explains the following regular expressions:

Match something that looks like an IP address.
Test if a number is in range 0-255
Match an IP address
Check if the string looks like an email address
Check if the string is a decimal number
Check if a word appears twice in the string
Increase all numbers by one in the string
Extract HTTP User-Agent string from the HTTP headers
Match printable ASCII characters
Match text between two HTML tags
Replace all bold tags with strong tag
Extract all matches from a regular expression

Vim Plugins: ragtag.vim

vote

In this part I introduce you to a plugin called "ragtag.vim."

The best parts of RagTag are mappings for editing HTML tags. It has a mapping for quickly closing open HTML tags, a mapping for quickly turning the typed word into a pair of open/close HTML tags, several mappings for inserting HTML doctype, linking to CSS stylesheets, loading JavaScript ... and it includes mappings for wrapping the typed text in a pair of tags for PHP, or for ASP or eRuby, and {% .. %} for Django. read more...

permapage | score:8682 | -pkrumins, March 7, 2010

More CommandLineFu One-Liners Explained

vote

Remember the previous post on CommandLineFu One-Liners Explained?

This article explains the next ten top one-liners.

For example, did you know "dd if=/dev/dsp | ssh username@host dd of=/dev/dsp" outputs your microphone on remote computers's speaker? This article explains how it works. read more...

permapage | score:8575 | -pkrumins, March 25, 2010

I wrote my first programming e-book: Awk One-Liners Explained

vote

I really love writing about programming and Linux, and I just published my first e-book ever on Awk one-liners.

Awk one-liners are short Awk programs that fit on one line and do one particular task, such as numbering lines, double spacing lines, printing only lines that match a pattern, etc.

Here is an example. The following one-liner prints all users on the Linux system:

awk -F: '{ print $1 }' /etc/passwd

This one-liner works this way. The -F: command line argument specifies that the lines in the /etc/passwd file should be split into fields by the colon character. As we all know, the information in /etc/passwd file is colon separated, so the first field $1 gets set to the username, the second field $2 gets set to the password, the third field $3 gets set to user id, etc. This one-liner prints only the first field which is the username. Simple, isn't it?

In my 50 page long e-book I carefully explain 70 one-liners in similar way.

Here is the table of contents of the e-book:

Preface.
1. Introduction to Awk One-Liners.
2. Line Spacing.
3. Numbering and Calculations.
4. Text Conversion and Substitution.
5. Selective Printing and Deleting of Certain Lines.
6. String and Array Creation.
Appendix A: Awk Special Variables.
Appendix B: Idiomatic Awk.
Index.
read more...

mail this link | permapage | score:8500 | -pkrumins, June 24, 2011

Unix tools: lsof

vote

This is the third post in the article series about Unix and Linux utilities that you should know about. In this post I will take you through the useful lsof tool. If netcat was called the Swiss Army Knife of Network Connections, then I'd call lsof the Swiss Army Knife of Unix debugging.

Lsof follows Unix philosophy closely. It does just one task and it does it perfectly � it lists information about files opened by processes. An open file may be a regular file, a directory, a NFS file, a block special file, a character special file, a shared library, a regular pipe, a named pipe, a symbolic link, a socket stream, an Internet socket, a UNIX domain socket, and many others. Since almost everything in Unix is a file, you can imagine how incredibly useful lsof is! read more...

mail this link | permapage | score:8476 | -pkrumins, December 28, 2009

Vim Plugins: matchit.vim

vote

This is the third post in the article series "Vim Plugins You Should Know About". This time I am going to introduce you to a plugin called "matchit.vim".

Matchit extends the existing functionality of �%� key (percent key). I'll first briefly remind you what the original �%� does and then explain how matchit.vim enhances it. read more...

permapage | score:8456 | -pkrumins, February 6, 2009

pv: Pipe Viewer Unix Utility

vote

This article explains a little-known Unix utility called Pipe Viewer or pv for short.

Pipe viewer is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion. read more...

permapage | score:8400 | -pkrumins, February 3, 2009

Introduction to Perl one-liners

vote

Perl one-liners are small and awesome Perl programs that fit in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing files in-place, doing statistics, carrying out system administration tasks, updating a bunch of files at once, and many more. Perl one-liners will make you the shell warrior. Anything that took you minutes to solve, will now take you seconds! read more...

mail this link | permapage | score:8003 | -pkrumins, May 28, 2012

E-book: Perl One-Liners Explained

vote

I'm happy to announce my 3rd e-book called "Perl One-Liners Explained."

Perl one-liners are small and awesome Perl programs that fit in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing files in-place, doing statistics, carrying out system administration tasks, updating a bunch of files at once, and many more.

Here is an example. Suppose you quickly need to generate a random, 8 character password. You can do it quickly with this Perl one-liner:

perl -le 'print map { ("a".."z")[rand 26] } 1..8'

Overall, the e-book has 111 pages and it explains 130 unique one-liners. Many of one-liners are presented in several different ways so the total number of one-liners in the book is over 200. read more...

mail this link | permapage | score:7975 | -pkrumins, February 4, 2012

The Netcat Unix Utility

vote

Netcat is often referred to as a �Swiss Army knife� utility, and for a good reason. Just like the multi-function usefulness of the venerable Swiss Army pocket knife, netcat�s functionality is as helpful. Some of its features include port scanning, transferring files, port listening and it can be used a backdoor. read more...

permapage | score:7961 | -pkrumins, February 19, 2009

Sed One-Liners Explained

vote

My previous post was about Awk One-Liners Explained and now I bring to you Sed One-Liners Explained!

Most people are only familiar with one particular command of sed, namely the "s" (substitute) comand. s/comand/command/. That is unsatisfactory. Sed has at least 20 different commands for you.

For example, any ideas what this sed one-liner does?

sed '/n/!G;s/(.)(.*n)/&21/;//D;s/.//'

Read the article to find it out! read more...

permapage | score:7906 | -pkrumins, November 22, 2008

Using Bash Command Line with Vi Keyboard Shortcuts

vote

I am going to introduce you to Bash's Vi editing mode and give out a detailed cheat sheet with the default keyboard mappings for this mode.

Bash supports two editing modes - Vi and Emacs. Emacs is the default editing mode and I wrote about it already in this article Bash's Emacs Editing Mode. read more...

permapage | score:7818 | -pkrumins, October 13, 2008

Simplified: Set Operations in the Unix Shell

vote

Set Operations in the Unix Shell Simplified is a simplified version of the original "Set Operations in the Unix Shell" article.

This article implements 14 various set operations by using Unix utilities such as uniq, sort, diff, head, tail, comm, awk, wc, and others.

The operations implemented are: Set Membership. Set Equality. Set Cardinality. Subset Test. Set Union. Set Intersection. Set Complement. Set Symmetric Difference. Power Set. Set Cartesian Product. Disjoint Set Test. Empty Set Test. Minimum. Maximum.

The article is meant to be a concise reference guide and comes with a downloadable .txt cheat sheet of all the operations. read more...

mail this link | permapage | score:7625 | -pkrumins, December 5, 2008

Awk One-Liners Explained

vote

This article explains Awk one-liners that you might have seen floating around in awk1line.txt file.

Take this Awk one-liner for example. Do you know what it does?

��awk 'END { print NR }'

Turns out it emulates "wc -l", or returns number of lines in the file! read more...

permapage | score:7599 | -pkrumins, October 17, 2008

Ten more One-Liners from CommandLineFu explained

vote

This article explains another ten popular one-liners from the CommandLineFu website.

For example, did you know you could read Wikipedia via DNS? Here is the one-liner that does it: