Saturday, December 27, 2014

Remote JMX access to WildFly (or JBoss AS7) using JConsole

One of the goals of JBoss AS7 was to make it much more secure by default, when compared to previous versions. One of the areas which was directly impacted by this goal was that you could no longer expect the server to expose some service on a port and get access to it without any authentication/authorization. Remember that in previous versions of JBoss AS you could access the JNDI port, the JMX port without any authentication/authorization, as long as those ports were opened for communication remotely. Finer grained authorizations on such ports for communications, in JBoss AS7, allows the server to control who gets to invoke operations over that port.

Of course, this is not just limited to JBoss AS7 but continues to be the goal in WildFly (which is the rename of JBoss Application Server). In fact, WildFly has gone one step further and now has the feature of "one single port" for all communication.

JMX communication in JBoss AS7 and WildFly

With that background, we'll now focus on JMX communication in JBoss AS7 and WildFly. I'll use WildFly (8.2.0 Final) as a reference for the rest of this article, but the same details apply (with minor changes) to other major versions of JBoss AS7 and WildFly, that have been released till date.

WildFly server is composed of "subsystems", each of which expose a particular set of functionality. For example, there's the EE subsystem which supports the Java EE feature set. Then there's the Undertow subsystem which supports web/HTTP server functionality. Similarly, there's a JMX subsystem which exposes the JMX feature set on the server. As you all are aware, I'm sure, JMX service is standardly used for monitoring and even managing Java servers and this includes managing the servers remotely. The JMX subsystem in WildFly allows remote access to the JMX service and port 9990 is what is used for that remote JMX communication.

JConsole for remote JMX access against JBoss AS7 and WildFly

Java (JDK) comes bundled with the JConsole tool which allows connecting to local or remote Java runtimes which expose the JMX service. The tool is easy to use, all you have to do is run the jconsole command it will show up a graphical menu listing any local Java processes and also an option to specify a remote URL to connect to a remote process:

# Start the JConsole

Let's assume that you have started WildFly standalone server, locally. Now when you start the jconsole, you'll notice that the WildFly Java process is listed in the local running processes to which you can connect to. When you select the WildFly Java instance, you'll be auto connected to it and you'll notice MBeans that are exposed by the server. However, in the context of this article, this "local process" mode in JConsole isn't what we are interested in.

Let's use the "Remote process" option in that JConsole menu which allows you to specify the remote URL to connect to the Java runtime and username and password to use to connect to that instance. Even though our WildFly server is running locally, we can use this "Remote process" option to try and connect to it. So let's try it out. Before that though, let's consider a the following few points:

  1. Remember that the JMX subsystem in WildFly allows remote access on port 9990
  2. For remote access to JMX, the URL is of the format - service:jmx:[vendor-specific-protocol]://[host]:[port]. The vendor specific protocol is the interesting bit here. In the case of WildFly that vendor-specific-protocol is http-remoting-jmx.
  3. Remember that WildFly is secure by default which means that just because the JMX subsystem exposes 9990 port for remote communication, it doesn't mean it's open for communication to anyone. In order to be allowed to communicate over this port, the caller client is expected to be authenticated and authorized. This is backed by the "ManagementRealm" in WildFly. Users authenticated and authorized against this realm are allowed access to that port.

Keeping those points in mind, let's first create a user in the Management Realm. This can be done using the add-user command line script (which is present in JBOSS_HOME/bin folder). I won't go into the details of that since there's enough documentation for that. Let's just assume that I created a user named "wflyadmin" with an appropriate password in the Management Realm. To verify that the user has been properly created, in the right realm, let's access the WildFly admin console at the URL http://localhost:9990/console. You'll be asked for username and password for access. Use the same username and password of the newly created user. If the login works, then you are good. If not, then make sure you have done things right while adding the new user (as I said I won't go into the details of adding a new user since it's going to just stretch this article unnecessarily long).

So at this point we have created a user named "wflyadmin" belonging to ManagementRealm. We'll be using this same user account for accessing the JMX service on WildFly, through JConsole. So let's now bring up the jconsole as usual:


On the JConsole menu let's again select the "Remote process" option and use the following URL in the URL text box:


Note: For JBoss AS 7.x and JBoss EAP 6.x, the vendor specific protocol is remoting-jmx and the port for communication is 9999. So the URL will be service:jmx:remoting-jmx://localhost:9999

In the username and password textboxes, use the same user/pass that you newly created. Finally, click on Connect. What do you see? It doesn't work! The connection fails. So what went wrong?

Why isn't the JConsole remote access to WildFly not working?

You did all the obvious things necessary to access the WildFly JMX service remotely but you keep seeing that JConsole can't connect to it. What could be the reason? Remember, in one of those points earlier, I noted that the "vendor specific protocol" is an interesting bit? We use http-remoting-jmx and that protocol internally relies on certain WildFly/JBoss specific libraries, primarily for remote communication and authentication and authorization. These libraries are WildFly server specific and hence aren't part of the standard Java runtime environment. When you start jconsole, it uses a standard classpath which just has the relevant libraries that are part of the JDK/JRE.

To solve this problem, what you need to do is bring in the WildFly server specific libraries into the classpath of JConsole. Before looking into how to do that, let's see which are the WildFly specific libraries that are needed. All the necessary classes for this to work are part of the jboss-cli-client.jar which is present in JBOSS_HOME/bin/client/ folder. So all we need to do in include this jar in the classpath of the jconsole tool. To do that we use the -J option of jconsole tool which allows passing parameters to the Java runtime of jconsole. The command to do that is:

$JAVA_HOME/bin/jconsole -J-Djava.class.path=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/jconsole.jar:/opt/wildfly-8.2.0.Final/bin/client/jboss-cli-client.jar

(Note that for Windows the classpath separator is the semi-colon character instead of the colon)

Note, the server specific jar for JBoss AS 7.x and JBoss EAP 6.x is named jboss-client.jar and is present at the same JBOSS_HOME/bin/client directory location.

So we are passing -Djava.class.path as the parameter to the jconsole Java runtime, using the -J option. Notice that we have specified more than just our server specific jar in that classpath. That's because, using the -Djava.class.path is expected to contain the complete classpath.  We are including the jars from the Java JDK lib folder that are necessary for JConsole and also our server specific jar in that classpath.

Running that command should bring up JConsole as usual and let's go ahead and select the "Remote process" option and specify the same URL as before:


and the same username and password as before and click Connect. This time you should be able to connect and should start seeing the MBeans and others services exposed over JMX.

How about providing a script which does this necessary classpath setup?

Since it's a common thing to try and use JConsole for remote access against WildFly, it's reasonable to expect to have a script which sets up the classpath (as above) and you could then just use that script. That's why WildFly ships such a script. It's in the JBOSS_HOME/bin folder and is called (and jconsole.bat for Windows). This is just a wrapper script which internally invokes the jconsole tool present in Java JDK, after setting up the classpath appropriately. All you have to do is run:


What about using JConsole from a really remote machine, against WildFly?

So far we were using the jconsole tool that was present on the same machine as the WildFly instance, which meant that we have filesystem access to the WildFly server specific jars present in the WildFly installation directory on the filesystem. This allowed us to setup the classpath for jconsole to point to the jar on the local filesystem?

What if you wanted to run jconsole from a remote machine against a WildFly server which is installed and running on a different machine. In that case, your remote client machine won't be having filesystem access to the WildFly installation directory. So to get jconsole running in such a scenario, you will have to copy over the JBOSS_HOME/bin/jboss-cli-client.jar to your remote client machine, to a directory of your choice and then setup the classpath for jconsole tool as explained earlier and point it to that jar location. That should get you access to JMX services of WildFly from jconsole on a remote machine.


More questions?

If you still have problems getting this to work or have other questions, please start a discussion in the JBoss community forums here

Sunday, December 07, 2014

WildFly 8.2.0.Final release - Quick overview of the changes

It's been a while since I last wrote on this blog. Although I have had some topics that I wanted to blog about, I just haven't found enough time to do it. I finally decided to write this up today after I saw a mail from one of the JBoss community members, checking up on why there haven't been any updates here lately (thanks for checking, Bhaskar! :)).

Before I move on to some technical things, a quick personal update - It's now been more than a year now since I changed jobs. I no longer work at Red Hat, JBoss. My (almost) 5 years at JBoss have been very fruitful and I enjoyed being part of the (JBoss AS/WildFly) application server development team. Last year, I decided to move on to something different and the right opportunity came along and I decided to take it up. Some of you know that I've been involved with the JBoss community for longer than the 5 years that I had been employed at Red Hat. I have been a JBoss community member since around 2004/2005, so even though I have moved on from Red Hat, I am still active in the JBoss forums.

Now that you all know what I've been upto, let's move on to some technical things.

WildFly 8.2.0.Final released!

The WildFly team just released the 8.2.0.Final version of WildFly some days back. As usual, it's available for download on the project's download page This is mainly a bug fix (plus some features) release in the 8.x series. I for one, was pleased to see this release happen because it allows the community to receive bug fixes on top of 8.1.0.Final release, which has been tested/used in the community for quite some time now. The WildFly team has indicated that this will be the last release in the 8.x series which sounds reasonable, given that the development team has already moved on to work on the 9.x series. It's never easy to work/maintain more than one major version of the code, especially in the context of bug fixes and backward compatibility.

What does WildFly 8.2.0.Final contain?

The complete overview of changes in available in this announcement on Jason's blog CDI spec upgrade and improved WebSocket support are the main items in terms of feature set. There's also this note in the release notes:
- EJBs in WARs now inherit the WAR security domain

There was a recent forum thread, where one of the users asked what that really means. Here's some background to that change As noted in that JIRA, this was feature request that was raised in the context of EJBs packaged in .war deployments. Most of you, I guess, will be aware that Java EE spec allows EJBs to be deployed as part of the .war deployment. What this means is that you can place your EJB classes within the .war/WEB-INF/classes or within a jar in .war/WEB-INF/lib. Although, this sounds straightforward for the end users, there are some technical implications to this (given the way Java EE "components" and "modules" are defined and configured within the server ecosystem). One such detail, is the way one configures the EJBs that are part of the .war deployment. Remember that if this EJB was part of a separate EJB module (within a .jar packaging outside of the .war) then one would use the ejb-jar.xml (and the WildFly specific jboss-ejb3.xml) as the deployment descriptors to configure it. This applies to EJBs deployed in a .war deployment too. i.e. one can use those same files for configuring EJBs. Now since these EJBs are part of a .war, the .war itself can/will have a deployment descriptor if its own (the web.xml and jboss-web.xml).

With that context, consider a case where the you have EJBs within the .war deployment and your .war deployment descriptor (the jboss-web.xml) configures a specific security domain for that deployment. Now remember that the EJBs too can configure a security domain (in jboss-ejb3.xml) and if none is configured and security is enabled on some EJBs, then the default "other" security domain gets used. So let's say your .war deployment, in which the EJBs reside, states that it wants to use "foo-bar" security domain and the EJBs, within that deployment, don't specify any specific security domain. So what should one expect in such scenario? Should the EJBs use the security domain configured at the .war level or should they default to the "other" security domain (since the EJB deployment descriptors don't specify any specific security domain configuration). The previous versions of WildFly had decided to use the default "other" security domain for the EJBs in such a case. Of course, the EJBs could use a jboss-ejb3.xml to set a different security domain, one which matches the jboss-web.xml. So that JIRA which I linked to earlier requested for a better, smarter and a more logical default in such cases. So starting this 8.2.0.Final version of WildFly, if you have a .war containing the EJBs and the EJBs don't define a security domain, then the security domain for any secured EJBs in that deployment *defaults* to the one that's defined at the .war deployment level. If the .war deployment doesn't set any specific security domain, then it ultimately, defaults to the "other" security domain. A good and logical change IMO. This will reduce some of the "surprises" that users have reported with previous version of WildFly, when it came to the security domain usage of EJBs in .war deployments.

For more about this, you can read the discussion here and ask any questions you have around this, in that thread.

What are the other notable things in WildFly 8.2.0.Final?

Although, not specific to 8.2.0.Final, the WildFly release contain a "patch" distribution which you can use if you already are using WildFly 8.1.0.Final and just want to "upgrade" to this new release. WildFly 8 has patch management built in and one can apply this patch on top of an existing 8.1.0.Final version.

What's next for WildFly?

As noted earlier, the WildFly development team has moved on to the next version of the project. Work is now continuing on 9.x version which already has a Alpha version released. So going forward, from what I have read, the releases will happen in the 9.x series.