Generally, we want to provide the best possible service to our users. Meaning that we want to remove broken servers as soon as possible and add them back when they are better as soon as possible.

To do this we need to understand the application and how it behaves, especially under stress.

The reason why this is important is that the reality of monitoring is a tradeoff between responsiveness vs false positives leading to bouncing.

If we make the health checking too fast, then a busy server may not have the time to respond and therefore it might be removed from the cluster meaning the other servers have more load in already a potentially busy time.

There are several dials to turn to get optimal health checking and this is how to set them.

Monitoring timeout

How long should we wait to try to connect before we give up and mark it as down?

If you are using a fast-monitoring method such as TCP connect then this can be fairly low 1s

However, if you are using application-level monitoring such as HTTP get or custom monitoring this would need to be longer.

A busy web server could take up to 60s to respond, however you probably want to fail it if it is running this slowly.

Monitoring interval

How long should we wait between monitoring attempts?

Firstly, we need to make sure this is set to longer values than the monitoring timeout value or the monitoring agents will start to overlap which will lead to more complexity to interpret behaviors.

The next thing to consider is the impact of monitoring frequency changes.

What is the impact of the monitoring on the application server? For example, if you have 10 VIPS using a full HTTP monitor every second you create an additional 600 requests per minute on the server. This may be a lot depending on the server’s capacity to handle these.

For low resource monitoring such as TCP it is not such an issue.

Monitoring In Count and Out Count

This feature helps to prevent servers from bouncing in and out of service.

It is possible your application server could have a temporary “glitch” and fail a single check but it’s not severe enough to want to remove the whole server from the cluster.

Conversely once a server is removed, we want to be confident it can provide a good service before we add it back.

Generally, we suggest Out Count of 1-2, meaning it has to fail the test 1-2 times, and an In Count setting of 3 meaning it needs to pass 3 successful tests before it’s added back.

Please navigate to the Real Servers > Advanced properties for the Virtual Service in question and adjust the timings we needed.

Custom authored monitors need to be written using the PERL languange, and Edgenexus provides a template in the admin guide found here: https://www.edgenexus.io/documentation.

When writing a monitor, you need to be aware of some STRICT Variables that are used. These are listed below.

# my $host = $_[0]; - This uses the “Address” from IP Services--Real Server section
# my $port = $_[1]; - This uses the “Port” from IP Services--Real Server section
# my $content = $_[2]; - This uses the “Required Content” value from the Library--Real Server Monitoring section
# my $notes = $_[3]; - This uses the “Notes” column in Real Server section of IP Services
# my $page = $_[4]; - This uses the “Page Location” values from Library--Real Server Monitor section
# my $user = $_[5]; - This uses the “User” value from the Library--Real Server Monitor section
# my $password = $_[6]; - This uses the “Password” value from the Library--Real Server Monitor section

The monitor code must return the result as:

• 1 for Success
• 2 for Failure

An example monitor code is shown below.

#Monitor-Name: Edgenexus-Agfa-Status-Health-Check
use strict;
use warnings;
########################################################################################################
# Edgenexus custom health checking Copyright Edgenexus 2023
########################################################################################################
#
#
# This is a Perl script for Edgenexus customer health checking
# The monitor name as above is displayed in the dropdown of Available health checks
# There are 3 value passed to this script
#
# I am using:
# _[0] IP address of the server to be health checked
# _[2] Required content - 200OK
# _[4] Page - Location and Port to use
# my $host = $_[0]; - This uses the “Address” from IP Services--Real Server section
# my $port = $_[1]; - This uses the “Port” from IP Services--Real Server section
# my $content = $_[2]; - This uses the “Required Content” value from the Library--Real Server Monitoring section
# my $notes = $_[3]; - This uses the “Notes” column in Real Server section of IP Services
# my $page = $_[4]; - This uses the “Page Location” values from Library--Real Server Monitor section
# my $user = $_[5]; - This uses the “User” value from the Library--Real Server Monitor section
# my $password = $_[6]; - This uses the “Password” value from the Library--Real Server Monitor section
#
# The script will return the following values
# 1 is the test is successful
# 2 if the test is unsuccessful
#
sub monitor
{
my $host = $_[0]; ### Host IP or name
my $content = $_[2]; ### Required Response
my $cvar = $_[4]; ### Path to be checked (Path, Port, prefix)
my ($page, $port, $prefix) = split(/,\s*/, $cvar);
$host = "$host:$port";
my @lines = `/usr/bin/wget -q -S --tries=1 --timeout=1 --no-check-certificate --output-document=- $prefix://$host$page 2>&1`;
if (join("",@lines))
{
print "$prefix://$host$page looking for - $content - Healhcheck check successful\n";
return(1);
}
else
{
print "$prefix://$host$page looking for - $content - Healhcheck check failed.\n";
return(2)
}
}
monitor(@ARGV);

The main method used for common monitors is to use the GUI.

You can see an example of how to utilise the built-in methods in the example video below, and of course in the current user guides on the Edgenexus website.

Used to check the availability and responsiveness of servers. It operates by sending Internet Control Message Protocol (ICMP) echo request packets, commonly known as “pings,” to the target server. If the server is operational, it responds with an ICMP echo reply. The monitor measures the round-trip time for the ping and assesses the server’s availability based on whether the response is received within a predefined time frame. This method is straightforward and efficient for verifying servers’ basic connectivity and responsiveness, making it an essential component in network monitoring and load-balancing setups to ensure continuous service availability.

TCP Connection
A TCP connection real server monitor checks the health and responsiveness of backend servers by establishing TCP connections to specified ports. This type of monitor periodically attempts to open a TCP connection to a server to verify its availability and network reachability. If the server completes the handshake and responds within an acceptable timeframe, it is deemed healthy and capable of handling client requests. Conversely, if the server fails to respond, is slow to connect, or rejects the connection, the monitor flags it as unhealthy. This allows the load balancer to reroute traffic away from the problematic server, ensuring continuous and reliable service delivery by directing traffic only to servers confirmed to be operational.

ICMP Unreachable
An ICMP Unreachable real server monitor uses Internet Control Message Protocol (ICMP) messages to determine the availability and responsiveness of backend servers. Specifically, it sends ICMP echo requests (pings) to the target server and waits for a response. If the server is unreachable or fails to respond within a specified time frame, the monitor flags it as down or unhealthy. This type of monitoring is crucial for quickly detecting network issues or server outages, ensuring that traffic is only directed to accessible servers and capable of handling requests, thereby maintaining high availability and performance of the application.

RDP
An RDP (Remote Desktop Protocol) real server monitor continuously assesses the availability and responsiveness of servers that provide remote desktop services. This monitor performs regular health checks by attempting to establish RDP sessions with the servers to ensure they can accept and handle remote desktop connections. By doing so, it verifies that the servers are not only operational but also performing efficiently under load. If a server fails to respond or exhibits poor performance, the monitor flags it as unhealthy, triggering the load balancer to reroute connections to healthier servers. This ensures uninterrupted remote desktop access for users, maintains service reliability, and helps in promptly identifying and addressing any issues affecting the remote desktop services.

200OK
A 200OK real server monitor is used in load balancing to verify the operational status of backend servers by ensuring they return a standard HTTP 200 OK response. This monitor sends periodic HTTP GET requests to the servers and expects a 200 OK response, indicating that the server is functioning correctly and can handle client requests. If a server fails to return a 200 OK response, it is marked as unhealthy, and the load balancer stops directing traffic to it until it recovers. This type of monitoring is crucial for maintaining high availability and reliability of web applications, as it ensures only healthy servers are included in the load balancing pool, thereby enhancing user experience and minimizing downtime.

DICOM
A DICOM real server monitor ensures the availability and performance of servers that handle Digital Imaging and Communications in Medicine (DICOM) data, which is critical in healthcare environments for managing medical imaging information. This monitor continuously performs health checks on DICOM servers by sending DICOM C-ECHO requests and evaluating the server’s responses. If a server fails to respond appropriately or within the expected timeframe, the monitor flags it as unhealthy, enabling automatic failover mechanisms to reroute traffic to functioning servers. This ensures uninterrupted access to vital medical imaging data, supporting timely diagnostics and patient care by maintaining the seamless operation of DICOM services.

When a server fails a health check, the load balancer is immediately informed and can stop directing traffic to that server, preventing potential service disruptions. This dynamic traffic management ensures that only healthy servers receive incoming requests, thereby avoiding overloading any single server and ensuring a smooth and uninterrupted user experience.

Additionally, real server monitors help in identifying performance bottlenecks and server failures quickly, allowing for prompt corrective actions. This proactive approach minimizes downtime, enhances fault tolerance, and maintains high availability of the application, which is critical for meeting service level agreements (SLAs) and ensuring user satisfaction.

Incorporating real server monitors in your load-balancing strategy ensures that your application remains robust, responsive, and reliable by dynamically adapting to the real-time health status of your servers.

The EdgeADC has a variety of pre-built health check monitors, and the capability for you to configure custom ones using the GUI. In addition to this you can also write specific custom ones using PERL and upload them to the EdgeADC.

Direct Server Return or DSR also known DR – Direct Routing, allows the server behind the load balancer to respond directly to the client bypassing the ADC on the response. DSR is only suitable for use with Layer 4 load balancing. Therefore, Caching and Compression are not available with this option chosen. 

This mode can only be used with TCP, UDP, and TCP/UDP service types.

Load balancing persistence policies are also restricted to Least Connections, Shared IP List Based, Round Robin and IP List Based.

Using DSR also requires Real Server changes to be done. Please refer to the Real Server Changes section in the Administrator guide.

The #Monitor-Name: command is the name used for the Perl Script stored on the ADC. If you do not include this line, then your script will not be found!

The following are mandatory:

• #Monitor-Name
• use strict;
• use warning;

The Perl scripts are run in a CHROOTED environment. They often call another application such as WGET or CURL. Sometimes these need to be updated for a specific features, such as SNI.

Dynamic Values

• my $host = $_[0]; – This uses the “Address” from IP Services–Real Server section
• my $port = $_[1]; – This uses the “Port” from IP Services–Real Server section
• my $content = $_[2]; – This uses the “Required Content” value from the Library–Real Server Monitoring section
• my $notes = $_[3]; – This uses the “Notes” column in Real Server section of IP Services
• my $page = $_[4]; – This uses the “Page Location” values from Library–Real Server Monitor section
• my $user = $_[5]; – This uses the “User” value from the Library–Real Server Monitor section
• my $password = $_[6]; – This uses the “Password” value from the Library–Real Server Monitor section

See below images that show where the Dynamic Values are obtained from:

Custom Health Checks have two outcomes:

• Successful
  Return Value 1
  Print a success message to Syslog
  Mark the Real Server Online (provided IN COUNT match)
• Unsuccessful
  Return Value 2
  Print a message saying Unsuccessful to Syslog
  Mark the Real Server Offline (provided OUT Count match)

An Example Custom Monitor

#Monitor-Name: Edgenexus-Agfa-Status-Health-Check
use strict;
use warnings;
########################################################################################################
# Edgenexus custom health checking Copyright Edgenexus 2023
########################################################################################################
#
#
# This is a Perl script for Edgenexus customer health checking
# The monitor name as above is displayed in the dropdown of Available health checks
# There are 3 value passed to this script
#
# I am using:
# _[0] IP address of the server to be health checked
# _[2] Required content - 200OK
# _[4] Page - Location and Port to use
# my $host = $_[0]; - This uses the “Address” from IP Services--Real Server section
# my $port = $_[1]; - This uses the “Port” from IP Services--Real Server section
# my $content = $_[2]; - This uses the “Required Content” value from the Library--Real Server Monitoring section
# my $notes = $_[3]; - This uses the “Notes” column in Real Server section of IP Services
# my $page = $_[4]; - This uses the “Page Location” values from Library--Real Server Monitor section
# my $user = $_[5]; - This uses the “User” value from the Library--Real Server Monitor section
# my $password = $_[6]; - This uses the “Password” value from the Library--Real Server Monitor section
#
# The script will return the following values
# 1 is the test is successful
# 2 if the test is unsuccessful
#
sub monitor
{
  my $host = $_[0]; ### Host IP or name
  my $content = $_[2]; ### Required Response
  my $cvar = $_[4]; ### Path to be checked (Path, Port, prefix)
  my ($page, $port, $prefix) = split(/,\s*/, $cvar);
  $host = "$host:$port";
  my @lines = `/usr/bin/wget -q -S --tries=1 --timeout=1 --no-check-certificate --output-document=- $prefix://$host$page 2>&1`;
  if (join("",@lines))
  {
    print "$prefix://$host$page looking for - $content - Healhcheck check successful\n";
    return(1);
  }
else
  {
    print "$prefix://$host$page looking for - $content - Healhcheck check failed.\n";
    return(2)
  }
}
monitor(@ARGV);

NOTE: 

Custom Monitoring – Use of global variables is not possible. Use local variables only – variables defined inside functions

Use of RegEx – All regular expressions must use a Perl-compatible statement syntax.

The EdgeADC is equipped with a few commonly used monitoring methods by default, but it is very simple to construct your own server monitor using the provided elements.

Alongside the default monitors, 200OK and DICOM present within the Monitoring manager, there are also additional preset monitors available under the Real Server Monitor menu within the Basic Tab. These are:

• Ping/ICMP Echo
• TCP Connection
• ICMP Unreachable
• RDP (Remote Desktop Protocol)

Using the Monitoring Manager, you can create monitors that best suit your needs. The EdgeADC Administrator Guide describes how to create a monitor in full.