IP Monitoring & Diagnostics With Command Line Tools: Part 4 - SSH Public Keys

Installing public SSH keys created on your workstation in a server will authenticate you without needing a password. This streamlines the SSH interaction and avoids the need to use stored and visible passwords in your scripts.

More articles in this series:

By creating and installing SSH public keys in your remote systems, remote commands can then be called to action as easily as running them on the local machine.

Older techniques that hid passwords in secret files or magically typed them on a virtual keyboard are no longer necessary. Everything is now much more secure.

Why this is a good thing

Initiating remote activity on a server or calling a remote agent to action with SSH is convenient because you can use the resulting output on a client machine dedicated to monitoring. Automated monitoring is impractical if a password must be entered manually to retrieve a result from the agent. Avoid this by creating an SSH key and copying it to the target server. The key authenticates the client machine when it requests a connection to login or execute a command.

Creating an SSH shared key

Use the ssh-keygen utility to manufacture the secure key package on your client machine. It will ask you where to store the keys it generates and what to use as a pass-phrase. Press the [RETURN] key in both cases to use the default. The files are stored in your home directory and the pass-phase will be empty. This chooses RSA encryption by default.


Generating public/private rsa key pair.

Enter file in which to save the key (/Home/user/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /Home/user/.ssh/id_rsa.

Your public key has been saved in /Home/user/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:tGXh2ZwadGcV84BkmQ1PIVseES3H5n8aYQ9FzDpTBfo [email protected]

The key's randomart image is:
+---[RSA 3072]----+
|          o.**%#*|
|         o B+%oo%|
|        . * * oO.|
|       . + o .B .|
|        S .  .E*.|
|              . +|
|               o.|
|              .  |
|                 |

Optional command line options will configure the keys in other ways. The default should work just fine for now.

You now have a public key file that authenticates your account on the client computer.

Deploy the key

The public key file can be deployed to the remote login account with the ssh-copy-id utility. If the remote system is correctly set up, this is all you need to do.

Use this command:

ssh-copy-id {user_account}@{hostname_or_IP_address}

You should see something like this:

ssh-copy-id [email protected]

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the
new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed
-- if you are prompted now it is to install the new keys

user @'s password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

The public key has been installed in the list of authorised keys on the server and remote commands should now work correctly.

When you access the remote server with SSH, your computer will pass the key details to it. The target login account will check the list of authorised keys for a match. If your key is present in the list, the remote command is accepted and called to action. It is a very simple and elegant solution.

What might go wrong

If this did not work, it must be fixed on the remote server. Any of these broken configuration items could prevent the key installation from working. Correcting them requires elevated administrator privileges:

• The .ssh directory may not exist in the target home folder. This needs to be present in the home directory for the target login account. Use the mkdir command to create it.
• The .ssh directory may have incorrect permissions. They should be set to 700 with the chmod command.
• The authorized_keys file may not exist. Create an empty file if necessary.
• The authorized_keys file may have incorrect permissions. Set the permissions to 644 with the chmod command.
• The SSH configuration may not have the correct options enabled to allow public key authentication. You will need to edit it to turn SSH key based authorisation on.
• The SSH config file may not have the authorized_keys file name description enabled. Check that this is also active and correctly describing the location and name of the file.
• Try the ssh-copy-id command from the client again after fixing the rest of these issues.
• Alternatively, manually upload your key file to the .ssh directory and append it to the authorized_keys file by hand. Be careful to append and not overwrite so that previously installed keys are undamaged.
• Check that your key is only appended once.
• Once these configurations are all correct, the SSH service needs to be restarted to load the configuration changes.

Now it all works!

After correcting these issues, this remote disk space command (df) works as expected without needing a password:

ssh [email protected] df

Filesystem         1K-blocks       Used  Available Use% Mounted on
/dev/root            2451064    1020972    1327692  44% /
none                  512652          0     512652   0% /dev
/tmp                  516844        816     516028   1% /tmp
/run                  516844       4664     512180   1% /run
/dev/shm              516844          4     516840   1% /dev/shm
/dev/vg1/volume_1 7681053612 5856395392 1824555820  77% /volume1

List the contents of a remote directory

Use the ls -la command on the remote machine to inspect the remote .ssh directory in the target account:

ssh [email protected] ls -la ./.ssh

drwx------  2 admin users 4096 Feb  2 10:11 .
drwxr-xr-x  4 admin users 4096 Feb  3 10:40 ..
-rw-r--r--  1 admin users  571 Feb  2 10:59 authorized_keys
-rwxrwxrwx+ 1 admin users  571 Feb  2 09:54 id_rsa.pub

This is where the SSH authorized_keys file lives.

Copying files securely

Use the scp command to remotely copy files the machines. This example puts an image file on a remote machine. The first parameter is the source file and the second is the target destination where it is copied to.

scp diagram.png {user_account}@{hostname}:./remote_diagram.png
This is the complementary command to pull that image file back again:

scp {user_account}@{hostname}:./remote_diagram.png recovered_diagram.png
Now remove the temporary remote file to clean up after testing:

ssh {user_account}@{hostname} rm remote_diagram.png

Use the same technique to acquire a web server error log. The remote account may need additional permissions to read this file:

scp {user_account}@{hostname}:/var/log/httpd/apache-error_log error.log

Check the contents of the error log to see if there are any security intrusion attempts on the remote server or errors to fix in your web site code.

Acquire and filter remote data

In this example, the cat, grep and tail commands are used to acquire the status of a Synology NAS. The whole dmesg file is copied to the client machine and filtered locally:

ssh {user_account}@{hostname} "cat /var/log/dmesg" |
grep ' operational ' |
tail -4

[Jan 1 00:05:31] md/raid:md2: device sda5 operational as raid disk 0
[Jan 1 00:05:31] md/raid:md2: device sdd5 operational as raid disk 3
[Jan 1 00:05:31] md/raid:md2: device sdc5 operational as raid disk 2
[Jan 1 00:05:31] md/raid:md2: device sdb5 operational as raid disk 1

Move the second double quote to the end of the line. Now the dmesg file is filtered remotely and only the result lines are delivered to the client machine.

ssh {user_account}@{hostname} "cat /var/log/dmesg |
grep ' operational ' |
tail -4"

The grep and tail commands now happen in the remote machine and the ssh command transfers the filtered result. Much less data is being hauled across the network. If network capacity is at a premium you might prefer the second method.

Check a remote network config

The ifconfig command lists the available network devices. Examine the network settings in a remote machine with this command:

ssh [email protected] ifconfig

What actually happens during the transaction?

A conversation happens between the client and server when the ssh command is being used. Add the -v flag to the ssh command to observe the exchange. Change this to the -vv or -vvv flags to see progressively more verbose output.


Even the smallest of configuration errors can stop something working. Following the remedial process carefully step by step will solve the problem. Take care and do not panic if the first thing you try does not work. Concentrate on one problem at a time. I needed to fix all of the issues described above in my test system. Any single one of them was a potential show stopper.

Placing quote symbols in different locations in the command line without changing anything else will control whether to call commands to action in the local machine or the remote server. Sometimes the effects of introducing a single character into the command line can have profound and subtle effects.

When you discover something that is broken, decide whether it is a candidate for creating a diagnostic test that could be run automatically to report a systemic problem that might occur again.

You might also like...

Standards: Part 9 - Standards For On-air Broadcasting & Streaming Services

Traditional on-air broadcasters and streaming service providers use many of the same standards to define how content is received from external providers and how it is subsequently delivered to the consumer. They may apply those standards in slightly different ways.

An Introduction To Network Observability

The more complex and intricate IP networks and cloud infrastructures become, the greater the potential for unwelcome dynamics in the system, and the greater the need for rich, reliable, real-time data about performance and error rates.

Designing IP Broadcast Systems: Part 3 - Designing For Everyday Operation

Welcome to the third part of ‘Designing IP Broadcast Systems’ - a major 18 article exploration of the technology needed to create practical IP based broadcast production systems. Part 3 discusses some of the key challenges of designing network systems to support eve…

What Are The Long-Term Implications Of AI For Broadcast?

We’ve all witnessed its phenomenal growth recently. The question is: how do we manage the process of adopting and adjusting to AI in the broadcasting industry? This article is more about our approach than specific examples of AI integration;…

Next-Gen 5G Contribution: Part 2 - MEC & The Disruptive Potential Of 5G

The migration of the core network functionality of 5G to virtualized or cloud-native infrastructure opens up new capabilities like MEC which have the potential to disrupt current approaches to remote production contribution networks.