IP Monitoring & Diagnostics With Command Line Tools: Part 5 - Using Shell Scripts

Shell scripts enable you to edit your diagnostic and monitoring commands into a script file so they can be repeated without needing to type them manually every time. Shell scripts also offer some unique and powerful features that help to build monitoring systems.

More articles in this series:

Shell scripts encapsulate simple interactive commands so they can be called to action to perform more complex tasks with looping and decision making based on their results. The Input/Output redirection feature alone makes them particularly good at capturing the current state of a server or process and saving it in a cached location where the reporting tools can find it.

What is shell scripting?

The command line shell accepts keystrokes and interprets them as calls to action. A script automates a sequence of commands by taking those same instructions from a file instead of the keyboard.

All of the constructs found in other programming languages, such as loops and conditional execution are available. Shell scripts add unique and interesting features that are not supported in other programming tools.

The first line of your script file

The first line of a UNIX shell script describes the interpreter that executes the script. It starts with a hash and exclamation symbol (#!) followed by the fully qualified path to the interpreter. Here are some examples:

Content Meaning
#!/bin/bash Call the widely supported Bourne Again (Bash) shell to action.
#!/usr/bin/python Run the script in the default Python interpreter.
#!/usr/bin/php Run the script in the default PHP interpreter.
#!/usr/bin/php_56 Run the script in a legacy PHP 5 interpreter.
#!/bin/false Inhibit the script from being run.


Legacy scripts might need to run in an older version of the interpreter. Choose that here if you need it.

I/O redirection

Each process expects one stream of input data and delivers two streams of output text. By default, the command line passes the keyboard characters to the standard input stream and both output streams go to your screen. The streams are called:


The normal output results are transmitted via STDOUT. If something goes wrong, the error messages are transmitted via STDERR. The two streams of text can be processed independently.

The shell can redirect the output streams to a different device, a log file or use them as input to another command. It can also redirect the input from another source instead of the keyboard.

Output redirection examples

Redirect the output of a command with the right facing caret (>) and save it in a file, overwriting any prior content:

ls -la > my_file_list.txt

Double caret symbols (>>) append to the file without overwriting it:

echo "Something went wrong" >> my_log_file.txt

In both cases, a new file will be created automatically if necessary.

This is how to discard the STDERR messages by redirecting them into a null device so they are ignored:

cat my_file.txt 2> /dev/null

Divert the STDOUT of a disk space check to a report file and save the STDERR messages in a log file:

df > output.txt 2>> error.log

Redirect both output streams to the same place. First, redirect STDOUT, then tell STDERR to follow it. This sends the contents of a data file through the sort command with a pipe and then stores the output with any errors that occur.

cat unsorted_data.txt | sort > output.txt 2>&1

Output redirection is incredibly powerful and will be used a great deal in a monitoring system to capture the results from the inspection commands.

Input redirection examples

There are subtle differences in the way the shell executes the commands when you use input redirection. You might only notice these if you monitor process listings while they run or inspect detailed file attributes and date stamps afterwards.

Print a file with one of these commands:

lpr file_to_print.txt
lpr < file_to_print.txt
cat file_to_print.txt | lpr

If you redirect the file using STDIN, or use cat to stream the file to the lpr command, the file name will not appear in the print queue listing. This enhances security and privacy.

Input redirection can read source text embedded within the script. The text is tagged to identify where it ends. We must append the redirected input with double carets (<<) to prevent the text from being executed as commands:

wc -c << EOF
How many characters are contained
in this example text before end of file tag?

Replace the -c on the wc command with -w to count words or -l to count lines.

Piped commands

The shell uses shorthand notation for chaining commands together by redirecting STDOUT and STDIN. The vertical bar character (|) creates a pipeline of instructions. Arranging the commands in the correct order is critical:

ls -la | cut -c35-37 | grep A | sort -r

The file listing from the ls command is sliced vertically with the cut command to only yield columns 35 to 37. The grep command discards any lines that do not contain a capital letter 'A'. Finally, the sort command arranges the remaining lines alphabetically in reverse order.

Shell meta-characters

Most punctuation symbols have a special meaning in the shell. We call them meta-characters. They may behave differently depending on the context:

• Command line parameters
• Describing file names
• Inside single quotes
• Inside double quotes
• Inside curly braces for variable substitution
• Inside back ticks for command substitution
• Inside dollar prefixed parenthesis for command substitution
• Conditional expressions between square brackets
• Regular expressions

Backslash symbols (\) placed in front of any meta-character deactivate any special treatment and render a literal character instead.

Using variables

Variables store values for later use in the script. By convention, system provided variables are upper case and user defined variable names are lower case.

Assign values to a variable using the equals symbol:


Prefix the name with a dollar sign to access the value. Optional curly braces help the interpreter distinguish variables from the surrounding code.

cat ${myvar}

Variable substitution

Using a variable before it has been defined will halt your script with an error. Defined but empty variables are problematic if you expect them to contain a non-null value.

Curly brace substitution can apply checks and alter the outcome before substituting the value. Adding a colon-dash (:-) or equals (:=) after the variable name will choose a specific behaviour.

If a variable contains a meaningful value, that will be returned by the substitution. Undefined and null variables will yield the replacement text instead. This handles the undefined and null variables more gracefully and script will not halt:

echo ${myvar:-replacement_string}
echo ${myvar:=replacement_string}

The colon-dash substitution leaves the source variable unchanged. The colon-equals substitution will assign the replacement_string to the source variable if it was called for.

Process hierarchy and inheritance

Calling a command to action creates a new child process for it. Each process has a unique ID value. List the processes and their PID values with the ps command:

ps -ef

root         1     0  0 Jan24 ?        00:00:06 /sbin/init
root      1677     1  0 Jan24 ?        00:00:00 /usr/sbin/ssh -D
root     29400  1677  0 09:55 ?        00:00:00 ssh: user [priv]
user     29511 29400  0 09:55 ?        00:00:00 sshd: user@pts/0
user     29515 29511  0 09:55 pts/0    00:00:00 -bash
user     32536 29515  0 10:51 pts/0    00:00:00 ps -ef

Compare the (parent) PPID values with the PID values in the previous line to trace the provenance. The OS commences running with the init process. Observe the ps command in its own process at the end of the list.

Child processes only inherit user-defined variables from the parent if they have been prepared first with an export command. Special system variables are always available. Anything passed as a command line argument is accessible to the child process.

Command substitution

A sub-shell executes commands in a child process and substitutes the STDOUT result in its place. The older technique of using back-tick characters is superseded by using parenthesis with a leading dollar sign. Any valid command can be executed in the sub-shell, including running another script. Here is an example that assigns the contents of a data file to a variable:

myvar=$(cat my_data_file.txt)

This example should be used with great care because it could allow command injection if the source file contains valid commands:

$(cat my_data_file.txt)

Never take user input data at face value. Always clean and filter it to remove injected commands.


The loop constructs each determine how to continue looping in a different way. They are functionally similar to other languages but the syntax is slightly different.

Loop type Behavior
Iterator Step through each item contained in a list with a for loop.
Step counter Conventional for loop similar to C-Language.
Loop while false While a condition is false, the loop continues.
Loop until false While a condition is true, the loop continues.


Making it runnable

When your script is ready to test, use the chmod command to apply the execute flag to it:

chmod +x your_new_script.sh

Without this, your script will not be allowed to run. Your account may need elevated privileges to do this.


Shell scripts are fundamental to building monitoring systems and diagnostic probes. Study the tools and practice writing simple scripts. Use online resources to extend your knowledge.

Because the edit-test cycle is so compact, shell scripts are useful for prototyping your monitoring architecture design before refactoring it into a compiled app.

We will use distributed shell scripts to implement a status screen to display performance measurements and flag problems on your systems.

You might also like...

Standards: Part 9 - Standards For On-air Broadcasting & Streaming Services

Traditional on-air broadcasters and streaming service providers use many of the same standards to define how content is received from external providers and how it is subsequently delivered to the consumer. They may apply those standards in slightly different ways.

An Introduction To Network Observability

The more complex and intricate IP networks and cloud infrastructures become, the greater the potential for unwelcome dynamics in the system, and the greater the need for rich, reliable, real-time data about performance and error rates.

Designing IP Broadcast Systems: Part 3 - Designing For Everyday Operation

Welcome to the third part of ‘Designing IP Broadcast Systems’ - a major 18 article exploration of the technology needed to create practical IP based broadcast production systems. Part 3 discusses some of the key challenges of designing network systems to support eve…

What Are The Long-Term Implications Of AI For Broadcast?

We’ve all witnessed its phenomenal growth recently. The question is: how do we manage the process of adopting and adjusting to AI in the broadcasting industry? This article is more about our approach than specific examples of AI integration;…

Next-Gen 5G Contribution: Part 2 - MEC & The Disruptive Potential Of 5G

The migration of the core network functionality of 5G to virtualized or cloud-native infrastructure opens up new capabilities like MEC which have the potential to disrupt current approaches to remote production contribution networks.