IP Monitoring & Diagnostics With Command Line Tools: Part 3 - Monitoring Your Remote Systems

Monitoring what is happening in a remote system depends on being able to ask for something to be checked and having the results reported back to you. There are many ways to do this. This article looks at some simple examples.

Remote services

You can only communicate with a remote system if it is running a process that responds to your connection. They would most likely be background processes with no visible user interface.

Background processes can be started from the command line and detached from the parent account so they continue running after exiting from the session. Otherwise they would halt.  They could be started automatically using the rc command when a machine is rebooted.

There are several different ways to run background processes in the UNIX environment. The optimum choice depends on what you need that process to do.

Process type Description
Servers A web server waits for requests to arrive on port 80. It supervises multiple child processes to handle the requests.
Daemon This is a single process running in the background. The syslogd daemon collates messages from other processes and stores them in a shared log file.
Agent app An application, triggered on demand to act on your behalf, possibly via a ssh command.
Service listener Services start up when a remote machine connects to a specific port.  The listener runs the application configured for that port and redirects the incoming data stream to it.

Service listeners are a very efficient solution for monitoring a remote system.  They consume no resources when they are at rest and only start up (very quickly) on demand. They are very resilient to memory leak problems because they quit immediately on completion of the task.

Monitoring with ssh

Measure the available disk space on a remote system with the df command:

ssh [email protected] df

Filesystem 1K-blocks Used    Available Use% Mounted on
/dev/root  2451064   1017960 1330704   44%  /
none       512652         0  512652    0%   /dev
/tmp       516844       800  516044    1%   /tmp
/run       516844      2700  514144    1%   /run
/dev/shm   516844         4  516840    1%   /dev/shm

You must manually enter the password for the remote account for the time being until we install a shared security key. That is a straightforward process and I will cover it in the next article.

Monitoring processes can observe the status with any of the tools the remote account has permission to use.  If you elevate the privileges on the remote account you should also implement some firewall protection to block any unrecognised machines if they try to connect. You could use the TCP-Wrappers utility for that and establish some rules that lock out unauthorised access.

Using the netcat (nc) command

The nc command can be started up at both ends of a remote connection as a server or client. Sending files to remote systems with it is very easy. Use nc interactively and type your messages to it directly. It can also be used by other applications as an intermediary. There are security implications with routers for connecting outside of your local network because the chosen ports may not be accessible.

At the remote end, you might use nc to listen on a particular port number and store any incoming data in a specific file. This example sets up a service listener on the remote machine using port number 5555:

nc -l 5555 > file_name.txt

On the local machine, use another instance of nc to communicate with the remote system on that port and send it some data from one of your files.  Whatever you send will be stored in the target file:

nc host.example.com 5555 < filename.txt

If you want to send some input interactively from your keyboard, just fire up nc on your machine without an input file:

nc host.example.com 5555

Then anything you type will be transmitted and stored in the remote file until you exit. When you exit the client, the server instance of nc will also quit.

Terminate the local interactive session by pressing [Control] + [D] key and your local copy of nc will exit and signal the remote instance to close its file. If you are using nc from Windows you need to type [Control] + [Z] and then [Enter] to send a [Control] + [D].

You could use this mechanism to store interlocks on a remote system to stop and start processing operations.  If they check the lock file contents on the remote machine and find that it contains the word 'STOP', they simply abort their start-up and check again after a suitable interval.

To have nc running on a remote machine when you need it, something must have started it for you. Aside from starting it manually, you could do this remotely with a ssh command. This gets very complex - very quickly.

Advanced topics like this are quite fascinating to solve and there is always a solution. When you find yourself wrestling with a complex scenario like this, try to step back and analyse the requirements again, because you might be approaching it in completely the wrong way and need a different type of remote intermediary process to act for you.

Using HTTPD for monitoring

If you run an Apache or Nginx web server in your remote systems, they can perform tasks on your behalf and return the results to you in a convenient format.  An HTTP server does not have to return a web page.  A data object is returned when JavaScript uses the XMLHttpRequest mechanisms. Ignore the XML prefix, this works with any kind of data Such as:

Type Description
CSV Comma Separated Values, possibly as a single line or a grid for importing into Excel or a database.
TSV Use a tab character as a data item separator instead of a Comma.
XML Strictly formatted mark-up similar to HTML. It is a useful way to serialise complex data structures but adds a significant overhead when formatting the raw data.
JSON A compact serialisation format that is also legal JavaScript syntax.  This is very easy to create using PHP in the server.
TXT Unstructured data using any proprietary schema you want, or none at all.

To implement remote monitoring, with HTTPD, build a web page (I would recommend using PHP). Let that code call out to the command line environment and execute a command there. Capture the result and output that as the page contents.

The individual components are quite simple to build and we will examine several examples of those in upcoming articles.

Using curl and wget

The curl and wget tools can communicate with remote systems and are easy to use. These commands could send a request to the status URL in your web server and store the result in a temporary file for processing locally.

The wget tool is primarily designed to download webpages.  It can use HTTP, HTTPS or FTP protocols to access the contents of a remote web server.

The curl tool is designed for transferring files of any kind and can use more sophisticated mechanisms. It can access remote resources with secure copy, SMB file sharing as well as all the protocols that wget supports.  You could also use curl to fetch mail messages from a mail server using the POP3 protocol.

The two tools are often confused with each other because both will retrieve webpages. The curl tool is much more powerful than wget and is useful for accessing many more remote resources than wget can reach.

Here is an example of using wget to retrieve a web page:

wget -O d_space.txt https://www.***.com/d_space.php

We could use curl to do the same thing:

curl -o d_space.txt https://www.***.com/d_space.php

Note that the output filename option flags are different for each command. The file is saved in the current working directory in the d_space.txt file.

Because we are using HTTP connections via port 80, this can work-around port mapping problems that you might have with other protocols.  Port 80 is often opened on routers for access to web pages hosted on servers in another sub-net.

The curl command can upload files to a remote location via the SMB file sharing protocol but with wget, you could only do that using the HTTP POST protocol as if you were submitting a form.

Clearly curl is much more powerful but wget is simpler to use for many tasks. We will take a deeper dive into what wget and curl can do in a later article.

Conclusion

Integrating our individual computers and devices so they behave like a single system becomes more practical with messages and calls to action moving around the system automatically.

In the next few articles we will explore how the UNIX command line environment can amplify your skills with these tools.  For example, you can:

• Install a shared key on a remote system to simplify using ssh.
• Redirect the output into a file.
• Filter and edit the results.
• Chain multiple tools together to execute a pipeline in one call to action.
• Wrap commands into a script and call them like you might use a macro in MS office.
• Schedule the execution of scripts on a regular basis.

You might also like...

Minimizing OTT Churn Rates Through Viewer Engagement

A D2C streaming service requires an understanding of satisfaction with the service – the quality of it, the ease of use, the style of use – which requires the right technology and a focused information-gathering approach.

Production Control Room Tools At NAB 2024

As we approach the 2024 NAB Show we discuss the increasing demands placed on production control rooms and their crew, and the technologies coming to market in this key area of live broadcast production.

Designing IP Broadcast Systems: Where Broadcast Meets IT

Broadcast and IT engineers have historically approached their professions from two different places, but as technology is more reliable, they are moving closer.

Network Orchestration And Monitoring At NAB 2024

Sophisticated IP infrastructure requires software layers to facilitate network & infrastructure planning, orchestration, and monitoring and there will be plenty in this area to see at the 2024 NAB Show.

Audio At NAB 2024

The 2024 NAB Show will see the big names in audio production embrace and help to drive forward the next generation of software centric distributed production workflows and join the ‘cloud’ revolution. Exciting times for broadcast audio.