IP Monitoring & Diagnostics With Command Line Tools: Part 2 - Testing Remote Connections

In the previous article, we set the scene for working with the Command Line Interface (CLI) on a UNIX system. Now we will explore some techniques for performing basic tests on our network infrastructure to check for potential problems.


More articles in this series:


The examples will mainly use IP version 4 addresses because they are more concise. In general, IP version 6 addresses work in all the same places. Occasionally you will need to add a modifier to a command to select which type of addressing you want to use.

At each IP address, we connect to a numbered port. Many of these are well known and configured identically everywhere. Think of an IP address being like a street name and a port being a house number in that street.

Security concerns will become more important when you are connecting to machines outside of your local network.

Things that can break

There are many reasons for a communication breakdown between two machines. Here is a list of them arranged in a sensible order for checking:

  • Faulty or unplugged cables and patch panels. You will need to visit the hardware and check this. Either end could be unplugged or the cable could be damaged.
  • Hardware Mac addresses could be duplicated although it is very rare. This could happen if you buy a batch of cheap Ethernet interface cards from an unreliable supplier.
  • Network switch configuration issues disallowing sockets to connect to each other. You will need to consult with your IT department so they can check the configuration.
  • IP address could be misconfigured as illegal values. There are several strategies for discovering what devices are present on your local network. They won't see devices that are powered off though. Later, we will build a tool to aggregate the list from several sources and display the result.
  • DNS issues that compromise the IP address. Work-around this by using IP addresses directly to avoid a DNS lookup.
  • Sometimes a route stops working due to a breakdown or reconfiguration and the alternative route is less optimal (more hops and higher latency).
  • Routers must be set up to allow port forwarding to reach the target port on the destination machine.
  • TCP Ports could be blocked by a firewall in the destination machine.
  • The target ports might not have a listener service configured to respond. Attempt a portscan on the target to see what is open and usable.

Checking the DNS

The Domain Name System (DNS) is a reference list of human readable names cross referenced against the physical IPv4 and IPv6 addresses. It is possible for DNS records to be legal but incorrect.

The first step is to use the nslookup command to resolve a fully qualified machine name to an IP address.  Here is an example of one that resolves a simple DNS record to an IP address:

nslookup www.myorg.com Server:  202.15.130.50
Address: 202.15.130.50#53>
Non-authoritative answer:
Name: www.myorg.com
Address: 10.208.35.165

The first two lines in the result describe the DNS name server that resolved your query and which port was used to connect to it.

These are the most likely response codes you will see.  There are a few others but they are very rare:

Result code Meaning
NOERROR A valid record was found.
NXDOMAIN The domain did not exist in any of the DNS name servers that were reachable from your computer. Check the spelling of the domain name you looked for.
SERVFAIL Might indicate that the DNS name server is broken or a firewall is blocking you. Escalate the problem to the people who look after your DNS server.
REFUSED The name server is prohibited from giving you an answer. This is also cause for escalation.

 

If the nslookup succeeded, the IP address returned might still be incorrect even though it was valid. Compare the IP address against your documentation to ensure it is the one you expect.

If you know the correct IP address, then reverse the lookup with the whois command to see if it is configured with a different domain name.

whois 10.208.35.165

% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object

   ...
   {very detailed information about the owner
   of that IP address here}
   ...

% This query was served by the RIPE Database Query Service
  version 1.104 (BLAARKOP)

Checking the connection

The ping command is a useful tool for determining whether a target node is reachable. If it is, what is the latency of the connection?

If we don't tell it to stop, ping will run continuously until you manually halt it with a [Control] + [C] keystroke. Add the -c {count} option to the command to request a limited number of test cycles.

Here is a single ping test on a machine that is reachable:

ping -c 1 212.159.13.50

PING 212.159.13.50 (212.159.13.50): 56 data bytes
64 bytes from 212.159.13.50:
icmp_seq=0 ttl=61 time=12.600 ms

--- 212.159.13.50 ping statistics ---
1 packets transmitted, 1 packets received,
0.0% packet loss
round-trip
min/avg/max/stddev = 12.600/12.600/12.600/0.000 ms

If the device is not reachable, then the ping command will tell you there was a 100% packet loss. Indicating a count of 1 will fire the ping test just once but the result will not appear until that test completes the timeout after a few seconds. Prove this with a known to be illegal IP address:

ping -c 1 254.254.254.254
PING 254.254.254.254 (254.254.254.254): 56 data bytes

--- 254.254.254.254 ping statistics ---
1 packets transmitted, 0 packets received,
100.0% packet loss

Tracing the route

Probe the connection and see if we can reach 'there' from 'here'.Something may be blocking us along the way, or there may be a breakage in the route altogether.Rather like an open circuit in a traditional wired system.

The ping command established whether the other end is reachable.  If it is, then we could use the traceroute command to examine the route that our packets are taking to get there. This may indicate a sub-optimal route which would cause performance issues.

The route is dynamic and might change unexpectedly.  In a high availability environment, there may be more than one route to the destination.  The connection will use the fastest route available but if that fails, the fallback alternative may be slower.  You will need elevated privileges to run the traceroute command.

Here is a partial traceroute result to a working destination:

traceroute 212.159.13.50

br/> traceroute to 212.159.13.50 (212.159.13.50), 64 hops max,
52 byte packets
1  192.168.1.1 (192.168.1.1)  1.400 ms  1.125 ms  1.088 ms
2  195.166.130.249 (195.166.130.249)  7.685 ms 
    7.803 ms  7.750 ms
3  gi2-23-442.ptn-gw02.plus.net (84.93.253.75) 
    8.453 ms  8.334 ms
    gi2-1-444.ptn-gw01.plus.net (84.93.253.79) 
    8.194 ms
4  * * gi2-23-442.ptn-gw02.plus.net (84.93.253.75) 
    8.178 ms !X
5  * * *
6  *

The asterisks indicate response timeouts.  If you see multiple lines of three asterisks, the traceroute cannot get any further. It does not necessarily mean the node is unreachable. The mechanism that traceroute uses might be purposely blocked to prevent intruders deducing the inner workings of the network.

Checking for open ports

If we cannot ping the target system, but we think it is up and running, (perhaps with ICMP packets being blocked), then try an alternative.  The nmap tool could run a portscan for example. You may need to install the nmap tool before using it.

Running unauthorised portscans on machines owned by other organisations can get you into serious trouble because it looks like an intrusion attempt. Always ask first and warn your own IT department too.

Once you have established that the target destination is reachable, check whether the ports that your applications needs are open.  Consult your documentation to see what they are. Then use the nmap command to run a portscan on the target. This will report whether they are reachable and which listening services are configured on them.

nmap myserver.mynet.com

Starting Nmap 7.60 ( https://nmap.org ) at
   2022-11-17 12:42 GMT

Nmap scan report for  myserver.mynet.com (192.168.1.24)

Not shown: 995 closed ports

PORT     STATE SERVICE
80/tcp   open  http
81/tcp   open  hosts2-ns
443/tcp  open  https
3306/tcp open  mysql
8200/tcp open  transit2

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

Run the same check on your own system to see whether other nodes might be able to connect back to you. Substitute localhost for the target host name or use the IP address 127.0.0.1 instead.

Conclusion

After eliminating hardware problems due to faulty cabling and potential issues with the router or network switch, we used the command line to run some diagnostics. The ping tool tested basic connectivity. Then we traced the route between the two systems. Finally, we ran a portscan to see what remote services were available. Some tools require elevated privileges. Manage that carefully and grant them only on a temporary basis when needed. Operating with minimum privileges whenever possible will significantly reduce the risk of accidents. In the next article we will see how to send and receive information from one machine to another and how to ask remote machines to tell us about their status.

You might also like...

Essential Guide: Delivering High Availability Cloud

Delivering high availability cloud for broadcast production and transmission environments requires engineers to think in terms of resilience from the very beginning of the design.

IP Monitoring & Diagnostics With Command Line Tools: Part 6 - Advanced Command Line Tools

We continue our series with some small code examples that will make your monitoring and diagnostic scripts more robust and reliable

Learning From The Experts At The BEITC Sessions at 2023 NAB Show

Many NAB Shows visitors don’t realize that some of the most valuable technical information released at NAB Shows emanates from BEITC sessions. The job titles of all but one speaker in the conference are all related to engineering, technology, d…

Empowering Cloud Through Microservices - Part 2

Achieving higher levels of scalability and flexibility demands a hybrid approach were microservices run in the cloud, on- and off-prem. Also, common APIs hide the complexity of the infrastructure from the user to provide a seamless operation.

System Showcase: Gravity Media Opens New Production Centre In London For Remote And On-Prem Projects

Production and media services company Gravity Media has just completed the first phase of technology integration at its new London Production Centre White City, located on the fifth floor of The Westworks building in London. The state-of-the-art facility, officially opened…