Working with Unix Tools

L ine-oriented textual data streams are the lowest useful common denominator for a lot of data that passes through our hands. We use such streams to represent program source code, Web server log data, version control history, file lists, symbol tables, archive contents, error messages, profiling data, and so on. For many routine, everyday tasks, we might be tempted to process the data using a " Swiss army knife " scripting language such as Perl, Python, or Ruby. However, doing that often requires writing a small, self-contained program and saving it into a file. By that point, we've sometimes lost interest in the task and end up doing the work manually, if at all. Often, it's more effective to combine Unix toolchest programs into a short and sweet pipeline that we can run from our shell's command prompt. With modern shell command-line editing facilities, we can build our command bit by bit, until it molds into exactly the form that suits us. Nowadays, many different systems—including GNU/Linux, Mac OS X, and Microsoft Windows—offer the original Unix tools preinstalled or as free down-loads, so there's no excuse for not adding this approach to your arsenal. Many one-liners that you'll build around the Unix tools follow a pattern that goes roughly like this: data fetching, selection, processing , and summarization. You'll also need to apply some plumbing to these parts. Jump in to get a quick tour of the facilities. Most of the time, your data will be text that you can feed directly to a tool's standard input. If this isn't the case, you'll need to adapt your data. If you're dealing with object files, try a command like nm (Unix), dumpbin (Windows), or javap (Java) to dig into them. If you're working with files grouped into an archive, a command like tar, jar, or ar will list the archive's contents. If your data comes from a (potentially large) collection of files, find can locate those that interest you. Then again, to get your data over the Web, use wget. You can also use dd (and the special file /dev/zero), yes, or jot to generate artificial data, perhaps to run a quick benchmark. Finally, to process a com-piler's list of error messages, redirect its standard error to its standard output; the incanta-tion 2>&1 will do the trick. I've not covered many other cases, including relational databases, version control systems , mail …