Reading Ganglia
From Yale HPC Wiki
Ganglia is our cluster monitoring and resource analysis tool at
http://hpc-status.wss.yale.edu
Both users and system administrators can benefit from ganglia monitoring once you're familiar with reading the information it presents.
Contents |
Different Page Views
On the Yale HPC Grid there are three basic layers:
- The Grid View
- The Cluster View and
- The Node View
The Grid View
Ganglia's front page is the "Grid View". It lists the grid and each cluster along with plots of total load and memory usage. view of the grid along with overall views of each cluster. If you are a user of more than one cluster and looking to find the most room to run your jobs, use the load plots in ganglia to determine free space.
The grid view also lists "Alerts" on the cluster. These are system warnings or failures the administrators need to resolve. In most cases compute nodes are removed and added to the clusters transparently, so there's little to be concerned about regarding outages.
The Cluster View
The "Cluster View" can be selected via the "Choose a Source" pulldown at the top of the grid view or by clicking on the Load/Memory plots for a cluster. At the top of the cluster view, ganglia displays Overview plots. These plots show cluster metrics, including cumulative Load, CPU, Memory and Network Usage. Cluster overview plots, along with compute node overview plots, display the most important system metrics in an easy to read logical format. Overview plots are discussed in detail below.
Cluster plots also display the number of nodes down and a sorted list of each compute node according to a specified metric. The metric, time window, and sort order can be chosen via the pull downs at the top of the page. If for example, I wanted to sort each of the nodes in the cluster by the amount of network traffic in, I would choose "Metric: bytes_in" and the compute nodes will be sorted accordingly.
The Node View
At the "Node View", users have most likely submitted their job, found the job is running in qstat (Running jobs have an "R" state in the "S" column), and determined where their job is running using "qstat -n". The node overview nearly mimics the cluster overview. In the node view, graphs for each metric are displayed individually.
What a Fully Utilized System Looks Like
The ideal load on a system varies depending on the limiting resource of an application, but generally a well behaved application will follow the notes below. Each of these points can be verified by looking at the overview plots.
Overview Plots
Overview plots are available at the cluster and node views, not the grid view. These plots combine statistics gathered from the respective source and overlay the information in a meaningful way. In most cases, these plots are the most important too for understanding how an application is running on a system.
Load
On the Load plot, the blue line (Running Processes) is overlapping or near the red line (number of processors). Additionally, the area below the blue and red lines is shaded completely gray.
It's important to note that load is not always equal to the number of reserved CPUs. When a job is most efficient, system load is equal to the number of CPUs, but sometimes, because of user choice or bounding conditions on the application, system load can be higher or lower.
Memory
On the Memory plot, the entire area below the red line (Total Memory) is either green or blue. Purple areas are bad and indicate that the system is swapping memory. If the system is swapping memory, your memory requirements are too large and should be reconsidered or the application should be moved.
CPU Usage
On the CPU Usage plot, most of the area should be shaded blue. If some amount is gray, this node is not 100% utilized. If some areas are red or orange, this means the system is taking CPU time to do things other than compute your job. For example, in most cases, lots of red means network activity, probably NFS. We recommend reducing this fraction of the CPU activity if possible.
Network Activity
Generally speaking the Network Plots should be as low as possible. Of course some network activity is required, but anything going over the network to be completed is at least an order of magnitude slower than doing the same thing on the local machine. High speed interconnects mitigate this speed hit, so if your application is network bound you might want to try and run it on a cluster with infiniband or myrinet.
A Picture of a Balanced System Load
Examples of Behavior Observed in Ganglia
NFS Activity
The example below shows what NFS load looks like in ganglia. Notice how system load relates to network activity. If the CPU has to work on system functions it's taking resources away from your application. Of course network activity is necessary, but it should be minimized when possible. If your application does this a lot you might want to request access to a cluster with a high performance cluster file system like lustre.
Network Bound MPI Application
This example shows a network bound MPI application. Notice the approximately 30% idle CPU usage. This is probably caused by delays in communication between the MPI processes. Also note the relatively high network usage 10-20MB/s. Nearly all MPI applications benefit from high speed interconnects and this is a good "picture" of why.
Out of Memory and Swapping
The next two examples show cases where the memory requirement for the node was just too much. In both cases the machine eventually died trying to make room in memory for the job. The moral of this story is, "Don't let the memory show purple". If it does, your job likely will die or slow down to a painful crawl.
Here the job dies almost immediately.
Here the node continues to run by utilizing swap space, but swap usage is so slow it generally just causes the job to virtually pause.
