VisualVM has made it a bit easier for us to monitor remote apps. You will have to start jstatd process on your remote system and then you can monitor how your remote VM is feeling.

However.. there are some pitfalls. Here are 2 of the most annoying once:

When starting jstatd you are getting Security Exceptions

To solve this error you will need to create a policy file containing something like this (call it jstatd.all.policy):

grant codebase "file:${java.home}/../lib/tools.jar" {

permission java.security.AllPermission;

};

Now start jstatd with the following command:

jstatd -J-Djava.security.policy=jstatd.all.policy

You are trying to connect to the remote host but getting Socket connection refused.

This is most likely caused because of an underlying exception – connection refused to host 127.0.0.1 or something simmilar (not the IP you are trying to access). This is easily solved by adding the following parameter to jstatd startup command:

-J-Djava.rmi.server.hostname=<your ip or hostname>

… Naturally there are more pitfalls but I am not going to cover all of them here. VirtualVM, just like legacy jConsole may be very usefull for monitoring how some applications act in the testing environment before you put them into producstion. Usually things that do not appear on your development machine while profiling may appear in the test environment where it is hard to notice them without profiling.

One of such problems may be MySQL Connector/J PermGen leak caused by Statement cancelation timer (something i have encountered recently and trying to come around)

This is probably the easiest way to access a MySQL database from Java. Everything seemed to be fine until i recently ran a few days long test of the daemon application on one of our test servers. It did not like me in the long run, i am still not sure why and whether it really was a Conenctor/J problem or not.

In any case, the problem i have faced was CPU usage suddenly started raising to about 50% (in a period of one day) and it wasnt a spike – it was a steady raise. When i checked the log file there were no warnings except one i got a at the end of the day: MySQL connection link failure.

Now one may wonder – how is this related to MySQL Connector. Well, it seems to me that it was due to Statement cancellation timer lockup that it started causing problems. Right now i have modified the code of the application to cancel all statemnts explicitly as well as some forced Garbage Collection, just to make sure that nothing unwanted slips by. The new test has been deployed to the servers and we are about to see if it has any effect.

Another note about Java (not really MySQL related):
It seems to be smart using “-server” option when starting a VM if the application is supposed to run for a very long time. The performance optimization is huge: we are talking about almost 2x performance optimization compared to “-client” VM (default).