Jboss Monitoring Using Zabbix

Monitoring of JBoss Application is required for delivering comprehensive fault management and proactive alarm notifications, checking for impending problems, triggering appropriate actions and gathering performance data for planning, analysis, and reporting.

Some of the components that can be monitored in JBoss are:

·                  JVM Heap & Non Heap Memory Usage

·                  Garbage Collection Information

·                  Class Loading and Unloading Information

·                  Running Thread Information

·                  Server Response Time

·                  Enterprise JavaBeans (EJBs)

·                  Metrics of all web applications

·                  Java Database Connectivity (JDBC) Pools

·                  Hardware and Operating system information (disk/cpu/memory/load, etc..)

·                  Other Jboss attributes

Open source Available tools which can be used for monitoring are Zabbix, Nagios and Zennos. In this blog, I have used Zabbix (Low cost effective solution) Open source distributed monitoring tool for Jboss and Linux System monitoring. Zabbix supports both polling and trapping techniques to collect data from monitored hosts. A flexible notification mechanism allows easy and quickly configure different types of notifications for pre-defined events.

Jboss Application server monitoring is logically divided into two sections:

 Java, which include JBoss and JVM Monitoring.

 Hardware & Operating System Monitoring.

Jboss and JVM monitoring

JMX stands for Java Management Extensions, and is a facility to allow for remote clients to connect to a JVM, and manage/monitor running applications in that JVM. This management is typically done through MBeans. MBeans are the heart of the JMX specification. 

In Jboss, JMX is being configured by enabling the JVM setting in "$JBOSS_HOME/bin/run.conf" file with the below mentioned options (stated JMX on 9999 port):

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9999" 

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false" 

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

In order to monitor the Jboss information, JMX defines a method for Java developers to integrate their applications with existing network management software by dynamically assigning Java objects with management attributes and operations.

In Jboss there are two ways that can grab the jmx attribute values.

 Jboss Twiddle utility: A simple command line tool that allows for interaction with a remote JMX server instance. This tool is called twiddle (for twiddling bits via JMX) and is located in the bin directory of the Jboss distribution. The only issue with this tool is it provides very basic information which is not sufficient for monitoring all Jboss parameters.

 JMX Command line utility: This utility allows system administrators to retrieve JMX management data by querying Mbeans directly on Java applications such as memory use, class loaded, garbage collection counts, etc.

I have used Skajla-JMX command line utility (Open Source JMX command line utility specially designed for monitoring Jboss application ) allows the administrator to instrument their Java application with a Zabbix agent. Below mentioned is the monitoring architecture.

To capture JVM parameter, first check all available message bean parameters which user can capture using this JMX command line utility, below mentioned is the command:

#java -jar skajla-JMXClient.jar 9999 info

e.g Lets say we want to capture ‘PS Perm Gen init memory’, then below mentioned is the command

#java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpassword 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init

To check the syntax of skajla-JMX client, do the following

# java -jar skajla-JMXClient.jar 
Usage :skajla-JMXClient.jar

JMX Command input parameter detail

JMX Server IP : IP Address of the jmx server  
JMX PORT      : Port on which JMX service is running
JMX USER      : Username required to connect to the JMX  
JMX PASSWORD  : Password required to connect to the JMX  
MESSAGE BEAN  : Target Bean for which attribute Value is required  
COMMAND       : Operation to run or attribute to fetch, Attributes begin with
a capital letter.  
ATTRIBUTE     : Optional, require in the cases when command output into multiple values

Skajla-JMX Command Line Utility and Zabbix Monitoring Architecture

Monitoring Network Architecture

In order to monitor the Jboss Server, Below mentioned is the minimal Server requirement (Fig. 2 Minimal Monitoring Network Architecture :

 Zabbix Server

 Zabbix Database Server

 Zabbix Web Server

 Monitoring Server (Depends on the Business Requirement)

Zabbix Agent Configuration for jboss Monitoring

Below mentioned is the required configuration, which we need to add in the zabbix agent configuration in  zabbix_agentd.conf file.

Step-1: Add the User Parameter to zabbix agentd configuration file.

User parameter has the following syntax:

UserParameter=key,command

Parameter	Description
Key	Unique Item Key
Command	Command to be executed to evaluate value of the key.

Example : To monitoring Perm Gen init memory, below mentioned is the required configuration

UserParameter=permgen_init, java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init

Note: The advantage of using skajla-jmxclient is that it gives only the required value of queried MBean parameter, which eases to capture the information for monitoring.

You can easily get other parameter values which you want to monitor by just replacing values of ”java.lang:type=MemoryPool,name=PS Perm Gen with required parameters” in above mentioned userParameter configuration

Step-2: Restart the Zabbix agentd

Step-3: Configuration of Zabbix frontend

Step-A: Add monitoring items

Login the zabbix frontend

|_ Configuration

|_ Host

|_ Select monitored server

|_ Items->Create Item

To Configure the item, the key point which you have to take care is that,you must set the item key same as set in UserParameter in zabbix_agentd.conf and item type must be Zabbix agent

Step-B: Create Graphs

Login the zabbix frontend

|_Configuration

|_Hosts

|_ Select host

|_ Graph

|_ Create Graph

For refrence, you can download Linux Jboss Zabbix Template

JBoss Monitoring Graphs (Also Defined in the Linux Jboss Template)

Downloads

 Download Linux Jboss Template

 Download JMXClient Executable JAR of File:

               http://jmxclient.svn.sourceforge.net/viewvc/jmxclient/skajla-JMXClient.jar?revision=3

 Download JMXClient Source Code:

               http://sourceforge.net/projects/jmxclient/

JMX Command Line Client

skajla-JMXClient is a open source, command-line based JMX client, which provides a centralized console for managing application clusters and distributed-application environments. The skajla-JMXClient mission is to provide an open source management platform, which can be used to manage and monitor complete production environment using open source monitoring tools like Zabbix, Naggios. 

skajla-JMXClient can be used to connect to the SUN JVM JDK 1.5.0 JMX Agent to set logging levels remotely and to get information on the running JVM. It is specially designed to return only the output value, which can be easily monitored using monitoring tools.

e.g Zabbix Agent configuration for capturing Perm Gen init memory.

UserParameter=permgen_init, java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init

Above mentioned command will return Perm Gen Init memory value only, which can be easily capture by Zabbix or any other monitoring tool.

Command to check all available message bean which user can capture using this utilty:

#java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpass info

To use, do the following to obtain usage: 
# java -jar skajla-JMXClient.jar 
Usage :skajla-JMXClient.jar

OR, To Display All Registered bean info 

skajla-JMXClient.jar  

Note : Provide multiple commands & attributes separated by comma(,)

Command Options Description:  
JMX Server IP : IP Address of the jmx server  
JMX PORT : Port on which JMX service is running
JMX USER : Username required to connect to the JMX  
JMX PASSWORD : Password required to connect to the JMX  
MESSAGE BEAN : Target Bean for which attribute Value is required  
COMMAND : Operation to run or attribute to fetch, Attributes begin with
a capital letter.  
ATTRIBUTE : Optional, require in the cases when command output into multiple values
e.g Max Heap Memory Value.

Requirement: 
JDK1.6,If connecting to a SUN 1.5.0 JDK JMX Agent, remote side must be started with 
system properties such as the following:
     -Dcom.sun.management.jmxremote.port=
     -Dcom.sun.management.jmxremote.authenticate=false
     -Dcom.sun.management.jmxremote.ssl=false

Download Executable JAR File:

http://jmxclient.svn.sourceforge.net/viewvc/jmxclient/skajla-JMXClient.jar?revision=3

Download Source Code:
http://sourceforge.net/projects/jmxclient/

How to Check Jar File Version

To have access to the version of a jar file, you should have access to the jar file. Precisely, you need to take a look at the jar file's manifest.

In case of the J2EE application

String checkjarFilevesion() {

try {

if(logger.isInfoEnabled())

logger.info("Verifying Jar Version");

String jarFilePath = Thread.currentThread().getContextClassLoader().getResource("../lib/javax.jar").getPath().replaceAll("%20", " ").toString();

JarFile jarfile = new JarFile(jarFilePath);

//Implementation-Version: 3.2.6.SP1

Manifest manifest = jarfile.getManifest();

String version = manifest.getMainAttributes().getValue("Implementation-Version");

return version;

} catch (IOException e) {

// TODO Auto-generated catch block

if(logger.isInfoEnabled())

logger.info("Invalid JAR file path");

}

return null;

}