Pages

Wednesday, July 7, 2010

Jboss Monitoring Using Zabbix

Monitoring of JBoss Application is required for delivering comprehensive fault management and proactive alarm notifications, checking for impending problems, triggering appropriate actions and gathering performance data for planning, analysis, and reporting.

Some of the components that can be monitored in JBoss are:
·                  JVM Heap & Non Heap Memory Usage
·                  Garbage Collection Information
·                  Class Loading and Unloading Information
·                  Running Thread Information
·                  Server Response Time
·                  Enterprise JavaBeans (EJBs)
·                  Metrics of all web applications
·                  Java Database Connectivity (JDBC) Pools
·                  Hardware and Operating system information (disk/cpu/memory/load, etc..)
·                  Other Jboss attributes

Open source Available tools which can be used for monitoring are Zabbix, Nagios and Zennos. In this blog, I have used Zabbix (Low cost effective solution) Open source distributed monitoring tool for Jboss and Linux System monitoring. Zabbix supports both polling and trapping techniques to collect data from monitored hosts. A flexible notification mechanism allows easy and quickly configure different types of notifications for pre-defined events.

Jboss Application server monitoring is logically divided into two sections:
 Java, which include JBoss and JVM Monitoring.
 Hardware & Operating System Monitoring.

Jboss and JVM monitoring
JMX stands for Java Management Extensions, and is a facility to allow for remote clients to connect to a JVM, and manage/monitor running applications in that JVM. This management is typically done through MBeans. MBeans are the heart of the JMX specification. 

In Jboss, JMX is being configured by enabling the JVM setting in "$JBOSS_HOME/bin/run.conf" file with the below mentioned options (stated JMX on 9999 port):
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9999" 
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false" 
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"

In order to monitor the Jboss information, JMX defines a method for Java developers to integrate their applications with existing network management software by dynamically assigning Java objects with management attributes and operations.

In Jboss there are two ways that can grab the jmx attribute values.
 Jboss Twiddle utility: A simple command line tool that allows for interaction with a remote JMX server instance. This tool is called twiddle (for twiddling bits via JMX) and is located in the bin directory of the Jboss distribution. The only issue with this tool is it provides very basic information which is not sufficient for monitoring all Jboss parameters.

 JMX Command line utility: This utility allows system administrators to retrieve JMX management data by querying Mbeans directly on Java applications such as memory use, class loaded, garbage collection counts, etc.

I have used Skajla-JMX command line utility (Open Source JMX command line utility specially designed for monitoring Jboss application ) allows the administrator to instrument their Java application with a Zabbix agent. Below mentioned is the monitoring architecture.

To capture JVM parameter, first check all available message bean parameters which user can capture using this JMX command line utility, below mentioned is the command:
#java -jar skajla-JMXClient.jar 9999 info

e.g Lets say we want to capture ‘PS Perm Gen init memory’, then below mentioned is the command
#java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpassword 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init

To check the syntax of skajla-JMX client, do the following
# java -jar skajla-JMXClient.jar 
Usage :
skajla-JMXClient.jar

JMX Command input parameter detail
JMX Server IP : IP Address of the jmx server  
JMX PORT      : Port on which JMX service is running
JMX USER      : Username required to connect to the JMX  
JMX PASSWORD  : Password required to connect to the JMX  
MESSAGE BEAN  : Target Bean for which attribute Value is required  
COMMAND       : Operation to run or attribute to fetch, Attributes begin with
a capital letter.  
ATTRIBUTE     : Optional, require in the cases when command output into multiple values

Skajla-JMX Command Line Utility and Zabbix Monitoring Architecture

Monitoring Network Architecture
In order to monitor the Jboss Server, Below mentioned is the minimal Server requirement (Fig. 2 Minimal Monitoring Network Architecture :
 Zabbix Server
 Zabbix Database Server
 Zabbix Web Server
 Monitoring Server (Depends on the Business Requirement)

Zabbix Agent Configuration for jboss Monitoring
Below mentioned is the required configuration, which we need to add in the zabbix agent configuration in  zabbix_agentd.conf file.
Step-1: Add the User Parameter to zabbix agentd configuration file.
User parameter has the following syntax:
UserParameter=key,command
Parameter
Description
Key
Unique Item Key
Command
Command to be executed to evaluate value of the key.

Example : To monitoring Perm Gen init memory, below mentioned is the required configuration
UserParameter=permgen_init, java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init

Note: The advantage of using skajla-jmxclient is that it gives only the required value of queried MBean parameter, which eases to capture the information for monitoring.

You can easily get other parameter values which you want to monitor by just replacing values of ”java.lang:type=MemoryPool,name=PS Perm Gen with required parameters” in above mentioned userParameter configuration

Step-2: Restart the Zabbix agentd
Step-3: Configuration of Zabbix frontend
Step-A: Add monitoring items
Login the zabbix frontend
|_ Configuration
|_ Host
|_ Select monitored server
|_ Items->Create Item

To Configure the item, the key point which you have to take care is that,you must set the item key same as set in UserParameter in zabbix_agentd.conf and item type must be Zabbix agent
Step-B: Create Graphs
Login the zabbix frontend
|_Configuration
|_Hosts
|_ Select host
|_ Graph
|_ Create Graph



JBoss Monitoring Graphs (Also Defined in the Linux Jboss Template)




 


Downloads
 Download Linux Jboss Template
 Download JMXClient Executable JAR of File:
               http://jmxclient.svn.sourceforge.net/viewvc/jmxclient/skajla-JMXClient.jar?revision=3
 Download JMXClient Source Code:
               http://sourceforge.net/projects/jmxclient/



30 comments:

Nuscly said...

Great article of good quality ! It miss these kind of article in zabbix doc.

cowmix said...

This is great stuff!

Is there anyway you can post a version of your zabbix_agentd.conf file that gives the other mbeans you are monitoring in your example?

thanks!

cowmix said...

This is great!

Is there anyway you can post a version of your zabbix_agentd.conf file that shows the other mbeans you are monitoring in your example?

Again, thanks!

S.Kajla said...

Thanks for your valuable comment.

Below mentioned is the link of zabbix_agentd.conf
configuration file having mbeans information, which I have configured in the zabbix jboss template
Zabbix_agentd_mbeans.conf

Lukasz said...

Very nice tool. However I don't know how to extract for example Java version from system properties returned by the command: java -jar skajla-JMXClient.jar 127.0.0.1 9999 user pass 'java.lang:type=Runtime' SystemProperties. Is it possible using only JMXClient?

S.Kajla said...

Directly it is not possible to fetch Java version from system properties, because currently JMXClient supports 2 Level value hierarchy. In the next version of JMXCient, I will add this feature.

Currently you can use awt command piped with JMSClient to extract values from the result.

Thanks

Janssen said...

Tested with JBoss 5.1 and java version 1.6.20 worked.
I have a server with JBoss 4.0.3 and 5.1.11 and java version did not work.
Have you tested with that configuration?

How can I get a complete list of MBeans with skajla-JMXClient.jar? so I can see all attributes.

Thanks,

Janssen

S.Kajla said...

Thanks for comment.

For MBean information, please use this command :
skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpassword info

It is tested out with Jboss EAP 5.0/5.0 GA/5.1 GA.

I will tested out with jboss 4.0.3 Version and update you accordingly.

Janssen said...

Thanks the quick response.

This is the output I get when running skajla-JMXClient.jar localhost-9999 admin jmxpassword info
Usage :
JMXClient
OR, To Display All Registered bean info
JMXClient
Note : Provide multiple commands & attributes seperated by comma(,)

By intuition, I tested the command "java -jar skajla-JMXClient.jar localhost 9999 admin jmxpassword info bean" and got a list that I wanted.

I also tested the version of JBoss 4.0.3 and it worked. I must have given some vacillation in command.

Thank you. You do not know how I helped. Before I wore the Zapcat, but did not like to monitor a host by two different ports.

S.Kajla said...

Thanks for the comment. I have designed this JMXClient to keep these kind of scenarios.

For bean information, below mentioned command is sufficient :
"java -jar skajla-JMXClient.jar localhost 9999 admin jmxpassword info"
bean argument is not required.


If I missed out any scenarios then please do post your comment.

Andrew said...

Hi There. I'm trying to use this on a Zabbix test implementation but I keep getting the error:
"MBean Not Registered Exception in thread "main" java.lang.NullPointerException at jmx.JMXClient.main(JMXClient.java:59)"

I'm running with the following command:
"java -jar skajla-JMXClient.jar 9999 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init"

I did some playing around with the source code to get a list of all the MBeans running and this is running.

This is running JBoxx 4.0.5.

Any help would be greatly appreciated.

S.Kajla said...

Hi Andrew,

Start Jboss with below mentioned parameters:
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9999"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"


the command which you are using doesn't have jboss server IP, Jmx username and Password, please use below mentioned command:
java -jar skajla-JMXClient.jar 192.168.1.1 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init

Andrew said...

Thanks I managed to get it sorted. The cause was idiocy on my side.

I've switched to a local only jmxremote port for security reasons so I've had to rewrite some of the code for the connection.

I've got it working but while doing some testing I've found that some of the registered MBeans returned using Info give me errors when I try to run the command line specifically to return their values.

For example an 'info' run shows me that the object 'JMImplementation:type=MBeanServerDelegate' is registered with the 'Type' attribute but running the commandline with the command :
"java -jar skajla-JMXClient.jar 9999 'JMImplementation:type=MBeanServerDelegate' Type" returns the exception "javax.management.AttributeNotFoundException: null"
Adding in println's to show the progression of the variables along each step shows that all the correct ObjectNames and Strings are being passed to the getAttribute method but for some MBeans it just gives and exception but for others it works 100%.

I'm not sure what the cause might be.

Janssen said...

I'm running java 1.5.0_11 on a server with JBoss 4.0.3 SP1 and when I run java -jar skajla-JMXClient.jar localhost 9999 admin jmxpassword info returns me the following error:

Exception in thread "main" java.lang.UnsupportedClassVersionError: Bad version number in .class file
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)

S.Kajla said...

This error is due to the java version, I will check the same with Java 1.5.0_11.

mean while you can download the source code and compile the same on your system.

Flavia said...

Hi Kajla,

Thanks for your great solution. We've had already implemented it and it works just as you said into your article. There's just one thing that I need a further help. Is there any possibility to get graphs that shows the same JBOSS ones but per web application installed on server? The same way JON (JBOSS Operations Network from Red Hat) does? Thanks in advance.

Flávia.

Flavia said...

Hi Kajla,

Thank you so much for your great solution. We've already implemented it into Zabbix and works great. However, I'd like to know if it's possible to show the available graphs which are in your solution per server, per application too.

Thanks in advance.

Flavia

S.Kajla said...

Thanks for your comment.

If you are running two JVM in one server, then you have to repeat all the command and add capturing parameter in your template.

If possible, then please send the detail of your network, so that I will suggest you accordingly.

S.Kajla said...

Hi Flavia,

Please use this command to check all available parameters and values we can capture.

Command "java -jar skajla-JMXClient.jar localhost 9999 admin jmxpassword info bean"

You will get the list of all the available parameters using this command

Gábor said...

Hi Surender.
I'm looking after your script, but didn't find it on sourceforge.
Is it reacheable today?
Please send me a link, or some advices to have it.
I'd like to use zabbix for jboss.

THX
Csomy, Budapest, Hungary

Gábor said...

Hi Surender!
I'd like to use your script, skajla-JMXClient.jar to monitor jboss with zabbix.
I was looking for it on links published before, but didn't find anything on sourceforge.
Could you put a direct link to download the jar file?
It would be very useful.
Thaks
Csomy, Budapest, Hungary

S.Kajla said...

It seems like sourceforge is not working, I will check it and update you accordingly.

But if you want it urgently, then mail me on s.kajla@gmail.com, I will send it to you directly via mail.

JWalker said...

User Parameters.
UserParameter=Perm_Gen_Init, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage init
UserParameter=Perm_Gen_Max, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage max
UserParameter=Perm_Gen_Used, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage used
UserParameter=Perm_Gen_Committed, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=MemoryPool,name=PS Perm Gen' CollectionUsage committed
UserParameter=Classes_Loaded, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=ClassLoading' LoadedClassCount
UserParameter=Classes_Unloaded, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=ClassLoading' UnloadedClassCount
UserParameter=Classes_Total, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=ClassLoading' TotalLoadedClassCount
UserParameter=Heap_Committed, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' HeapMemoryUsage committed
UserParameter=Heap_Initial, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' HeapMemoryUsage init
UserParameter=Heap_Max, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' HeapMemoryUsage max
UserParameter=Heap_Used, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' HeapMemoryUsage used
UserParameter=Non_Heap_Committed, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' NonHeapMemoryUsage committed
UserParameter=Non_Heap_Initial, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' NonHeapMemoryUsage init
UserParameter=Non_Heap_Max, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' NonHeapMemoryUsage max
UserParameter=Non_Heap_Used, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' NonHeapMemoryUsage used
UserParameter=No_of_Objects_Pending_Finalization, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Memory' ObjectPendingFinalizationCount
UserParameter=Total_Threads_Started, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Threading' TotalStartedThreadCount
UserParameter=Total_Threads, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Threading' ThreadCount
UserParameter=Peak_Threads, /usr/local/jdk1.6.0_18/bin/java -jar /opt/zabbix/lib/skajla-JMXClient.jar localhost 9999 admin jmxpass 'java.lang:type=Threading' PeakThreadCount

chuck1972 said...

Hi, nice work, thx.

I can use the command line and it works (Ican see the returned value). But when I try to add item to graph it dispays "[no data]"

Command line example:
[root@uenum bin]# java -jar skajla-JMXClient.jar 10.10.50.107 9999 admin admin java.lang:type=Memory HeapMemoryUsage used
395440784

zabbix_agent.conf example:
UserParameter=Heap_Used,java -jar skajla-JMXClient.jar 10.10.50.107 9999 admin admin java.lang:type=Memory HeapMemoryUsage used

Item is named Heap_Used and it is the same name as in zabbix_agent.conf.


What am I doing wrong?

chuck1972 said...

Hi, thanks for good work.

I try to monitor a remote Jboss server over the skajla-JMXClient.jar tool.

When I isue a command in the command promt the tool works (I can see the reterned value).

But when I add an item in the Zabbix, I get "[no data]"?!?!

I followed your guide. What am I doing wrong?

example line from local zabbix_agent.conf:
UserParameter=Heap_Used,java -jar skajla-JMXClient.jar 10.10.50.107 9999 admin admin java.lang:type=Memory HeapMemoryUsage used

example from command line:
[root@uenum bin]# java -jar skajla-JMXClient.jar 10.10.50.107 9999 admin admin java.lang:type=Memory HeapMemoryUsage used
395440784


Item key is configured as :
"Heap_Used" and type is "Zabbix agent".

Thx, for replay.

Veegee said...

Hi Surender,

Great work !! Need your help, I want to invoke mbean operations which have multiple parameters but does not know the right syntax, For example I want to call

java.lang:type=Threading dumpAllThreads

but does not know how to pass parameters. It takes two boolean parameters.

Thanks

Anonymous said...

Hi S.Kajla,

Great work, many thanks ;)

Best regards,
Roman Melko

Chris said...

Hoi S.Kajla,

Many Tanks, helped me alot with monitoring Jboss with zabbix. I got one question: I wan't to monitor the ConnectionPools towards an oracle DB as well. Any hints how to do this?

Thanks & Regards,
Chris

Prabhu Gopalakrishnan said...

Hi Kajla,
I have tried configuring to monitor JBOSS in our SAVVION Application, but I failed to get the output.
Can you please help to how to configure the script to monitor Savvion JBOSS (5.0.1)

Thanks

Prabhu Gopalakrishnan said...

Hi Kajla,

Please help me.... I am getting below error when I run "java -jar skajla-JMXClient.jar localhost 18793 admin admin info mbeans"

I have tried replacing mbeans as bean, beans as well. nothing worked.

Also I am unable to add JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=9999" in the jboss startup as it is showing error.

C:\SAVVION_INSTALL\SBM76SP1\jboss\lib>java -jar skajla-JMXClient.jar localhost 18793 admin admin info mbeans
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error duri
ng JRMP connection establishment; nested exception is:
java.io.EOFException]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at jmx.core.Client.getRegisteredMBeans(Client.java:226)
at jmx.JMXClient.main(JMXClient.java:41)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exce
ption is:
java.io.EOFException]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
... 3 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.io.EOFException
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
... 8 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:228)
... 12 more