IP Monitoring & Diagnostics With Command Line Tools: Part 5 - Using Shell Scripts

Shell scripts enable you to edit your diagnostic and monitoring commands into a script file so they can be repeated without needing to type them manually every time. Shell scripts also offer some unique and powerful features that help to build monitoring systems.

More articles in this series:

Shell scripts encapsulate simple interactive commands so they can be called to action to perform more complex tasks with looping and decision making based on their results. The Input/Output redirection feature alone makes them particularly good at capturing the current state of a server or process and saving it in a cached location where the reporting tools can find it.

What is shell scripting?

The command line shell accepts keystrokes and interprets them as calls to action. A script automates a sequence of commands by taking those same instructions from a file instead of the keyboard.

All of the constructs found in other programming languages, such as loops and conditional execution are available. Shell scripts add unique and interesting features that are not supported in other programming tools.

The first line of your script file

The first line of a UNIX shell script describes the interpreter that executes the script. It starts with a hash and exclamation symbol (#!) followed by the fully qualified path to the interpreter. Here are some examples:

Content Meaning
#!/bin/bash Call the widely supported Bourne Again (Bash) shell to action.
#!/usr/bin/python Run the script in the default Python interpreter.
#!/usr/bin/php Run the script in the default PHP interpreter.
#!/usr/bin/php_56 Run the script in a legacy PHP 5 interpreter.
#!/bin/false Inhibit the script from being run.


Legacy scripts might need to run in an older version of the interpreter. Choose that here if you need it.

I/O redirection

Each process expects one stream of input data and delivers two streams of output text. By default, the command line passes the keyboard characters to the standard input stream and both output streams go to your screen. The streams are called:


The normal output results are transmitted via STDOUT. If something goes wrong, the error messages are transmitted via STDERR. The two streams of text can be processed independently.

The shell can redirect the output streams to a different device, a log file or use them as input to another command. It can also redirect the input from another source instead of the keyboard.

Output redirection examples

Redirect the output of a command with the right facing caret (>) and save it in a file, overwriting any prior content:

ls -la > my_file_list.txt

Double caret symbols (>>) append to the file without overwriting it:

echo "Something went wrong" >> my_log_file.txt

In both cases, a new file will be created automatically if necessary.

This is how to discard the STDERR messages by redirecting them into a null device so they are ignored:

cat my_file.txt 2> /dev/null

Divert the STDOUT of a disk space check to a report file and save the STDERR messages in a log file:

df > output.txt 2>> error.log

Redirect both output streams to the same place. First, redirect STDOUT, then tell STDERR to follow it. This sends the contents of a data file through the sort command with a pipe and then stores the output with any errors that occur.

cat unsorted_data.txt | sort > output.txt 2>&1

Output redirection is incredibly powerful and will be used a great deal in a monitoring system to capture the results from the inspection commands.

Input redirection examples

There are subtle differences in the way the shell executes the commands when you use input redirection. You might only notice these if you monitor process listings while they run or inspect detailed file attributes and date stamps afterwards.

Print a file with one of these commands:

lpr file_to_print.txt
lpr < file_to_print.txt
cat file_to_print.txt | lpr

If you redirect the file using STDIN, or use cat to stream the file to the lpr command, the file name will not appear in the print queue listing. This enhances security and privacy.

Input redirection can read source text embedded within the script. The text is tagged to identify where it ends. We must append the redirected input with double carets (<<) to prevent the text from being executed as commands:

wc -c << EOF
How many characters are contained
in this example text before end of file tag?

Replace the -c on the wc command with -w to count words or -l to count lines.

Piped commands

The shell uses shorthand notation for chaining commands together by redirecting STDOUT and STDIN. The vertical bar character (|) creates a pipeline of instructions. Arranging the commands in the correct order is critical:

ls -la | cut -c35-37 | grep A | sort -r

The file listing from the ls command is sliced vertically with the cut command to only yield columns 35 to 37. The grep command discards any lines that do not contain a capital letter 'A'. Finally, the sort command arranges the remaining lines alphabetically in reverse order.

Shell meta-characters

Most punctuation symbols have a special meaning in the shell. We call them meta-characters. They may behave differently depending on the context:

• Command line parameters
• Describing file names
• Inside single quotes
• Inside double quotes
• Inside curly braces for variable substitution
• Inside back ticks for command substitution
• Inside dollar prefixed parenthesis for command substitution
• Conditional expressions between square brackets
• Regular expressions

Backslash symbols (\) placed in front of any meta-character deactivate any special treatment and render a literal character instead.

Using variables

Variables store values for later use in the script. By convention, system provided variables are upper case and user defined variable names are lower case.

Assign values to a variable using the equals symbol:


Prefix the name with a dollar sign to access the value. Optional curly braces help the interpreter distinguish variables from the surrounding code.

cat ${myvar}

Variable substitution

Using a variable before it has been defined will halt your script with an error. Defined but empty variables are problematic if you expect them to contain a non-null value.

Curly brace substitution can apply checks and alter the outcome before substituting the value. Adding a colon-dash (:-) or equals (:=) after the variable name will choose a specific behaviour.

If a variable contains a meaningful value, that will be returned by the substitution. Undefined and null variables will yield the replacement text instead. This handles the undefined and null variables more gracefully and script will not halt:

echo ${myvar:-replacement_string}
echo ${myvar:=replacement_string}

The colon-dash substitution leaves the source variable unchanged. The colon-equals substitution will assign the replacement_string to the source variable if it was called for.

Process hierarchy and inheritance

Calling a command to action creates a new child process for it. Each process has a unique ID value. List the processes and their PID values with the ps command:

ps -ef

root         1     0  0 Jan24 ?        00:00:06 /sbin/init
root      1677     1  0 Jan24 ?        00:00:00 /usr/sbin/ssh -D
root     29400  1677  0 09:55 ?        00:00:00 ssh: user [priv]
user     29511 29400  0 09:55 ?        00:00:00 sshd: user@pts/0
user     29515 29511  0 09:55 pts/0    00:00:00 -bash
user     32536 29515  0 10:51 pts/0    00:00:00 ps -ef

Compare the (parent) PPID values with the PID values in the previous line to trace the provenance. The OS commences running with the init process. Observe the ps command in its own process at the end of the list.

Child processes only inherit user-defined variables from the parent if they have been prepared first with an export command. Special system variables are always available. Anything passed as a command line argument is accessible to the child process.

Command substitution

A sub-shell executes commands in a child process and substitutes the STDOUT result in its place. The older technique of using back-tick characters is superseded by using parenthesis with a leading dollar sign. Any valid command can be executed in the sub-shell, including running another script. Here is an example that assigns the contents of a data file to a variable:

myvar=$(cat my_data_file.txt)

This example should be used with great care because it could allow command injection if the source file contains valid commands:

$(cat my_data_file.txt)

Never take user input data at face value. Always clean and filter it to remove injected commands.


The loop constructs each determine how to continue looping in a different way. They are functionally similar to other languages but the syntax is slightly different.

Loop type Behavior
Iterator Step through each item contained in a list with a for loop.
Step counter Conventional for loop similar to C-Language.
Loop while false While a condition is false, the loop continues.
Loop until false While a condition is true, the loop continues.


Making it runnable

When your script is ready to test, use the chmod command to apply the execute flag to it:

chmod +x your_new_script.sh

Without this, your script will not be allowed to run. Your account may need elevated privileges to do this.


Shell scripts are fundamental to building monitoring systems and diagnostic probes. Study the tools and practice writing simple scripts. Use online resources to extend your knowledge.

Because the edit-test cycle is so compact, shell scripts are useful for prototyping your monitoring architecture design before refactoring it into a compiled app.

We will use distributed shell scripts to implement a status screen to display performance measurements and flag problems on your systems.

You might also like...

CDN For Live And VOD

CDNs are much more than just high-speed links between Origins and ISP (Internet Service Provider) networks. Instead, they form a complete ecosystem of storage and processing, and they create new possibilities for highly efficient streaming at scale that will likely…

How Starlink Is Progressing As An Alternative To 5G

TV stations have mostly parked their satellite trucks and ENG vans in favor of mobile bi-directional wireless digital systems such as bonded cellular, wireless, and direct-to-modem wired internet connections. Is Starlink part of the future?

The Streaming Tsunami: Part 7 - How Immersive Experience Pushes Streaming Video Technology Forwards

We discuss the accelerating evolution of immersive media experiences & consumer technology, whether the mainstream media is keeping pace with the gamification of media consumption and the exponential growth in delivery capacity that will be required to support mass audience…

The Big Guide To OTT: Part 7 - Content Delivery Networks

Part 7 of The Big Guide To OTT is a set of three articles which examine the pivotal role of CDN’s, how they are evolving and how Open Caching aims to support broadcast grade streaming.

Scalable Dynamic Software For Broadcasters - The Book

Scalable Dynamic Software For Broadcasters is a free 88 page eBook containing a collection of 12 articles which give a detailed explanation of the principles, terminology and technology required to leverage microservices based, software only broadcast production infrastructure.