In this last article of the series I'll show how to use the native JMX support within the JDK to implement a notification mechanism which alerts a listener when the HEAP memory is above a certain threshold.
As discussed in my previous article this approach is ideal because is push instead of pull, is not intrusive and places minimal computing demand on your application.
These are the key components to the solution illustrated in this article:
- MemoryWarningService: This component acts as a listener and registers itself with the Memory MBean to receive notifications. It is configurable with a threshold in the form of a percentage between 0 and 1 (where 1 is 100%)
- MemoryThreadDumper: This component is invoked when the MemoryWarningService is notified that the HEAP usage is above the threshold and its responsibility is to write a thread dump to a file
- MemoryWarningServiceConfigurator: This component is an MBean and exposes a method to change the threshold of the MemoryWarningService.
The solution provides also a MemoryHeapFiller class used to fill up the HEAP while testing the application and a MemTest class to bootstrap the Spring environment.
While the application is running (play with the MemoryHeapFiller settings) You can fire the JConsole at URL: service:jmx:rmi://localhost/jndi/rmi://localhost:8888/jemosJmxConnector connecting as jemosAdmin / secure and change the threshold to various values.
The code is not meant for production: it is not robust, there are numerous comments missing, and the filename where to write the thread dump is hard-coded; it represents, however, a good starting point.
The code is attached below. You will need Maven to build it.
Download Jemos-jmx-experiments-0.0.1-SNAPSHOT-project
I tried a scenario with initial threshold to be 0.5, I changed it to 0.3 and then to 0.8. The results are shown below:
2011-08-15 21:53:21 ClassPathXmlApplicationContext [INFO] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@a4a63d8: startup date [Mon Aug 15 21:53:21 BST 2011]; root of context hierarchy
2011-08-15 21:53:21 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemos-jmx-appCtx.xml]
2011-08-15 21:53:21 PropertyPlaceholderConfigurer [INFO] Loading properties file from class path resource [jemos-jmx.properties]
2011-08-15 21:53:21 PropertyPlaceholderConfigurer [INFO] Loading properties file from URL [file:/C:/Users/mtedone/.secure/jmxconnector-credentials.properties]
2011-08-15 21:53:21 ThreadPoolTaskScheduler [INFO] Initializing ExecutorService 'myScheduler'
2011-08-15 21:53:21 ClassPathXmlApplicationContext [INFO] Bean 'myScheduler' of type [class org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2011-08-15 21:53:21 DefaultListableBeanFactory [INFO] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@30296f76: defining beans [propertyConfigurer,loggerConfigurator,memoryWarningServiceConfigurator,memoryHeapFiller,memoryThreadDumper,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,mbeanExporter,jemosJmxServer,rmiRegistry,clientConnector,memoryMxBean,memoryWarningService,org.springframework.scheduling.annotation.internalAsyncAnnotationProcessor,org.springframework.scheduling.annotation.internalScheduledAnnotationProcessor,myScheduler]; root of factory hierarchy
2011-08-15 21:53:21 AnnotationMBeanExporter [INFO] Registering beans for JMX exposure on startup
2011-08-15 21:53:21 RmiRegistryFactoryBean [INFO] Looking for RMI registry at port '8888'
2011-08-15 21:53:23 RmiRegistryFactoryBean [INFO] Could not detect RMI registry - creating new one
2011-08-15 21:53:23 ConnectorServerFactoryBean [INFO] JMX connector server started: javax.management.remote.rmi.RMIConnectorServer@4355d3a3
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Bean with name 'jemosJmxServer' has been autodetected for JMX exposure
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Bean with name 'loggerConfigurator' has been autodetected for JMX exposure
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Bean with name 'memoryWarningServiceConfigurator' has been autodetected for JMX exposure
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Located managed bean 'loggerConfigurator': registering with JMX server as MBean [jemos.mbeans:type=config,name=LoggingConfiguration]
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Located MBean 'jemosJmxServer': registering with JMX server as MBean [jemos.mbeans:name=jemosJmxServer,type=RMIConnectorServer]
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Located managed bean 'memoryWarningServiceConfigurator': registering with JMX server as MBean [jemos.mbeans:type=config,name=MemoryWarningServiceConfiguration]
2011-08-15 21:53:23 MemoryWarningService [INFO] Percentage is: 0.5
2011-08-15 21:53:23 MemoryWarningService [INFO] Listener added to JMX bean
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
2011-08-15 21:53:37 MemoryWarningService [INFO] Percentage is: 0.3
2011-08-15 21:53:37 MemoryWarningServiceConfigurator [INFO] Memory threshold set to 0.3
Adding data...
2011-08-15 21:53:38 MemoryWarningService [WARN] Memory usage low!!!
2011-08-15 21:53:38 MemoryWarningService [WARN] percentageUsed = 0.3815679398794023
2011-08-15 21:53:38 MemoryThreadDumper [WARN] Stacks dumped to: C:/tmp/stacks.dump
Adding data...
Adding data...
Adding data...
2011-08-15 21:53:45 MemoryWarningService [INFO] Percentage is: 0.8
2011-08-15 21:53:45 MemoryWarningServiceConfigurator [INFO] Memory threshold set to 0.8
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
2011-08-15 21:54:01 MemoryWarningService [WARN] Memory usage low!!!
2011-08-15 21:54:01 MemoryWarningService [WARN] percentageUsed = 0.8383333266727508
2011-08-15 21:54:02 MemoryThreadDumper [WARN] Stacks dumped to: C:/tmp/stacks.dump
Adding data...
Adding data...
Adding data...
Exception in thread "JMX server connection timeout 24" java.lang.OutOfMemoryError: Java heap space
The Memory Warning Service
/**
*
*/
package uk.co.jemos.experiments.jmx;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryNotificationInfo;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryType;
import javax.annotation.PostConstruct;
import javax.management.Notification;
import javax.management.NotificationEmitter;
import javax.management.NotificationListener;
import org.springframework.beans.factory.annotation.Autowired;
/**
* A component which sends notifications when the HEAP memory is above a certain
* threshold.
*
* @author mtedone
*
*/
public class MemoryWarningService implements NotificationListener {
/** This bean's name */
public static final String MBEAN_NAME = "jemos.mbeans:type=monitoring,name=MemoryWarningService";
/** The application logger */
private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger
.getLogger(MemoryWarningService.class);
@Autowired
private NotificationEmitter memoryMxBean;
@Autowired
private MemoryThreadDumper threadDumper;
/** A pool of Memory MX Beans specialised in HEAP management */
private static final MemoryPoolMXBean tenuredGenPool = findTenuredGenPool();
/**
* {@inheritDoc}
*/
@Override
public void handleNotification(Notification notification, Object handback) {
if (notification.getType().equals(
MemoryNotificationInfo.MEMORY_THRESHOLD_EXCEEDED)) {
long maxMemory = tenuredGenPool.getUsage().getMax();
long usedMemory = tenuredGenPool.getUsage().getUsed();
LOG.warn("Memory usage low!!!");
double percentageUsed = (double) usedMemory / maxMemory;
LOG.warn("percentageUsed = " + percentageUsed);
threadDumper.dumpStacks();
} else {
LOG.info("Other notification received..."
+ notification.getMessage());
}
}
/**
* It sets the threshold percentage.
*
* @param percentage
*/
public void setPercentageUsageThreshold(double percentage) {
if (percentage <= 0.0 || percentage > 1.0) {
throw new IllegalArgumentException("Percentage not in range");
} else {
LOG.info("Percentage is: " + percentage);
}
long maxMemory = tenuredGenPool.getUsage().getMax();
long warningThreshold = (long) (maxMemory * percentage);
tenuredGenPool.setUsageThreshold(warningThreshold);
}
@PostConstruct
public void completeSetup() {
memoryMxBean.addNotificationListener(this, null, null);
LOG.info("Listener added to JMX bean");
}
/**
* Tenured Space Pool can be determined by it being of type HEAP and by it
* being possible to set the usage threshold.
*/
private static MemoryPoolMXBean findTenuredGenPool() {
for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
// I don't know whether this approach is better, or whether
// we should rather check for the pool name "Tenured Gen"?
if (pool.getType() == MemoryType.HEAP
&& pool.isUsageThresholdSupported()) {
return pool;
}
}
throw new AssertionError("Could not find tenured space");
}
}
The Memory Thread Dumper
/**
*
*/
package uk.co.jemos.experiments.jmx;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import org.apache.commons.io.IOUtils;
import org.springframework.stereotype.Component;
/**
* This component dumps the thread stack to the file system.
*
* @author mtedone
*
*/
@Component
public class MemoryThreadDumper {
/** The application logger */
private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger
.getLogger(MemoryThreadDumper.class);
/**
* It dumps the Thread stacks
*
* @throws IOException
*/
public void dumpStacks() {
// hard-coded: This needs to be changed to a property or something
String stackFileName = "C:/tmp/stacks.dump";
ThreadMXBean mxBean = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = mxBean.getThreadInfo(
mxBean.getAllThreadIds(), 0);
Map<Long, ThreadInfo> threadInfoMap = new HashMap<Long, ThreadInfo>();
for (ThreadInfo threadInfo : threadInfos) {
threadInfoMap.put(threadInfo.getThreadId(), threadInfo);
}
File dumpFile = new File(stackFileName);
BufferedWriter writer = null;
try {
writer = new BufferedWriter(new FileWriter(dumpFile));
this.dumpTraces(mxBean, threadInfoMap, writer);
LOG.warn("Stacks dumped to: " + stackFileName);
} catch (IOException e) {
throw new IllegalStateException(
"An exception occurred while writing the thread dump");
} finally {
IOUtils.closeQuietly(writer);
}
}
private void dumpTraces(ThreadMXBean mxBean,
Map<Long, ThreadInfo> threadInfoMap, Writer writer)
throws IOException {
Map<Thread, StackTraceElement[]> stacks = Thread.getAllStackTraces();
writer.write("Dump of "
+ stacks.size()
+ " thread at "
+ new SimpleDateFormat("yyyy/MM/dd HH:mm:ss z")
.format(new Date(System.currentTimeMillis())) + "\n\n");
for (Map.Entry<Thread, StackTraceElement[]> entry : stacks.entrySet()) {
Thread thread = entry.getKey();
writer.write("\"" + thread.getName() + "\" prio="
+ thread.getPriority() + " tid=" + thread.getId() + " "
+ thread.getState() + " "
+ (thread.isDaemon() ? "deamon" : "worker") + "\n");
ThreadInfo threadInfo = threadInfoMap.get(thread.getId());
if (threadInfo != null) {
writer.write(" native=" + threadInfo.isInNative()
+ ", suspended=" + threadInfo.isSuspended()
+ ", block=" + threadInfo.getBlockedCount() + ", wait="
+ threadInfo.getWaitedCount() + "\n");
writer.write(" lock=" + threadInfo.getLockName()
+ " owned by " + threadInfo.getLockOwnerName() + " ("
+ threadInfo.getLockOwnerId() + "), cpu="
+ mxBean.getThreadCpuTime(threadInfo.getThreadId())
/ 1000000L + ", user="
+ mxBean.getThreadUserTime(threadInfo.getThreadId())
/ 1000000L + "\n");
}
for (StackTraceElement element : entry.getValue()) {
writer.write(" ");
writer.write(element.toString());
writer.write("\n");
}
writer.write("\n");
}
}
}
The Memory Service Configuration MBean
package uk.co.jemos.experiments.jmx.mbeans;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.jmx.export.annotation.ManagedOperation;
import org.springframework.jmx.export.annotation.ManagedOperationParameter;
import org.springframework.jmx.export.annotation.ManagedOperationParameters;
import org.springframework.jmx.export.annotation.ManagedResource;
import org.springframework.stereotype.Component;
import uk.co.jemos.experiments.jmx.MemoryWarningService;
@Component
@ManagedResource(objectName = MemoryWarningServiceConfigurator.MBEAN_NAME, //
description = "Allows clients to set the memory threshold")
public class MemoryWarningServiceConfigurator implements
ApplicationContextAware {
/** The application logger */
private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger
.getLogger(MemoryWarningServiceConfigurator.class);
public static final String MBEAN_NAME = "jemos.mbeans:type=config,name=MemoryWarningServiceConfiguration";
private ApplicationContext ctx;
@ManagedOperation(description = "Sets the memory threshold for the memory warning system")
@ManagedOperationParameters({ @ManagedOperationParameter(description = "The memory threshold", name = "memoryThreshold"), })
public void setMemoryThreshold(double memoryThreshold) {
MemoryWarningService memoryWarningService = (MemoryWarningService) ctx
.getBean("memoryWarningService");
memoryWarningService.setPercentageUsageThreshold(memoryThreshold);
LOG.info("Memory threshold set to " + memoryThreshold);
}
@Override
public void setApplicationContext(ApplicationContext applicationContext)
throws BeansException {
ctx = applicationContext;
}
}
The Spring configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">
<bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" >
<property name="locations">
<list>
<value>classpath:jemos-jmx.properties</value>
<value>file:///${user.home}/.secure/jmxconnector-credentials.properties
</value>
</list>
</property>
</bean>
<context:component-scan base-package="uk.co.jemos.experiments.jmx" />
<context:mbean-export default-domain="jemos.mbeans" />
<bean id="jemosJmxServer" class="org.springframework.jmx.support.ConnectorServerFactoryBean"
depends-on="rmiRegistry">
<property name="objectName" value="connector:name=rmi" />
<property name="serviceUrl"
value="service:jmx:rmi://localhost/jndi/rmi://localhost:${jemos.jmx.rmi.port}/jemosJmxConnector" />
<property name="environment">
<!-- the following is only valid when the sun jmx implementation is used -->
<map>
<entry key="jmx.remote.x.password.file" value="${user.home}/.secure/jmxremote.password" />
<entry key="jmx.remote.x.access.file" value="${user.home}/.secure/jmxremote.access" />
</map>
</property>
</bean>
<bean id="rmiRegistry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean">
<property name="port" value="${jemos.jmx.rmi.port}" />
</bean>
<bean id="clientConnector" class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean"
depends-on="jemosJmxServer">
<property name="serviceUrl"
value="service:jmx:rmi://localhost/jndi/rmi://localhost:${jemos.jmx.rmi.port}/jemosJmxConnector" />
<property name="environment">
<map>
<entry key="jmx.remote.credentials">
<bean
factory-method="commaDelimitedListToStringArray">
<constructor-arg value="${jmx.username},${jmx.password}" />
</bean>
</entry>
</map>
</property>
</bean>
<bean id="memoryMxBean" class="java.lang.management.ManagementFactory"
factory-method="getMemoryMXBean" />
<bean id="memoryWarningService" class="uk.co.jemos.experiments.jmx.MemoryWarningService">
<property name="percentageUsageThreshold" value="0.5" />
</bean>
<task:annotation-driven scheduler="myScheduler" />
<task:scheduler id="myScheduler" pool-size="10" />
</beans>
Happy technology!
Recent Comments