Monday, October 30, 2017

Kafka with OpenSSL



UPDATE: This article has been updated on Nov 11th 2017 with the following details:

  • Updated to note that the issues linked in this article, related to WildFly OpenSSL, have all been fixed and the WildFly OpenSSL master branch now has the version which contains these fixes. The numbers that you see in this article, include those fixes.
  • Updated to add a section which lists the producer and consumer numbers when Java 9 runtime is used, for both the SSL engine shipped by JRE as well as the OpenSSL one. 
  • Updated to note that the version of Kafka used is 1.0.0 (which was released recently)


In one of the products I'm involved in, we use Kafka extensively. We have been using Kafka since 0.8.x days. If you follow the Kafka development, you might be aware that they are about to release their 1.0.0 version very soon. Kafka allows you to use SSL for both producing and consuming messages. Both the Kafka broker and the client libraries are configurable to specify the necessary SSL characteristics. Within our own product, we use Java client side libraries for consuming and producing messages. Their Java client side libraries have gone through a phase of API changes a while back in one of their releases. We use their "new" Java client APIs.

We started experimenting with using SSL for producing and consuming messages in Kafka, more than a year back. Our initial experiments showed that switching to SSL instead of using plaintext had a noticeable impact on performance. Given the way we use Kafka within our product, even some consistent (milli seconds) degradation in latency is almost noticeable. It's a acceptable and a known fact that you do incur certain performance impact when you are using SSL. However, the amount of degradation was to a point that we decided not to switch to SSL for a while. There have been discussions and JIRAs like this one where such impact has been tracked. Things have definitely improved since that JIRA (we are on 0.10.x release these days), but we didn't have enough time to get some numbers with SSL enabled within our environment.

A small detour

With that background, let me take a small detour from Kafka discussions. I also follow WildFly and various other projects in its ecosystem. Very recently, WildFly added support for using OpenSSL as a SSL provider instead of the one that's shipped as part of the JRE. As part of that support, they use WildFly OpenSSL project, which provides Java bindings (the implementation of interfaces necessary to use it as a SSLEngine in Java) for OpenSSL. Given that Java has a plugable mechanism for SSL providers and that fact that Kafka allows you to configure such SSL configurations, the WildFly OpenSSL project interested me.

Using Kafka with WildFly OpenSSL

I vaguely remember reading in some discussions that OpenSSL performs better compared to the SSL provider shipped in Java. So I decided to experiment with using WildFly OpenSSL with Kafka and compare it with the SSL provider shipped in Java. My goals of this experiment were pretty much these:
  • Use SSL for producing and consuming messages in Kafka
  • Compare OpenSSL against the SSL provider shipped in Java
  • Use the default settings that's shipped in Kafka for this performance testing. In fact, Kafka developers encourage you to use the defaults in performance testing as much as possible.
  • Use the tools shipped within Kafka for testing this performance. Kafka ships both producer and consumer performance testing tool, which is good enough for what we are after. Using their own tools, rules out any issues that I might end up with in the tool that I write for these tests (I did in fact write one of my own, just for the sake of it, but decided to stick with the ones shipped in Kafka since it pretty much ended up being similar both in terms of code and the output it produced)

I would like to note that it wasn't ever a goal for me, in these experiments to compare plain text and SSL numbers. This experiment is solely to see how different SSL providers are performing.

Setting up the system for the tests

 

Kafka installation

I decided to use the latest version of Kafka. The latest released version currently is 1.0.0 and I downloaded it from their downloads page. Kafka installation is straightforward, you just extract the downloaded archive and can straightaway boot it up and start consuming and producing messages. I won't go into any of the installation details of Kafka since that's out of the scope of this installation. I will go into the configurations that I used as we go along.

WildFly OpenSSL installation

I use MacOS for development and will be using this for my tests. In order to use WildFly OpenSSL, I went ahead and cloned the github repo. Ran into a build issue, but it was a straightforward fix for which there's now a pull request with a fix. I then setup Kafka to use WildFly OpenSSL and started experimenting. This set of experiments was just to make sure that it's usable without impacting any functionality. I did run into an issue which turned out to be an issue in WildFly OpenSSL which is now reported here. This issue has now been fixed and pushed to the WildFly OpenSSL upstream repo.

The README of that project already has the necessary instructions to build it, so I won't get into those details.

Java installation

I use the Oracle JRE 1.8 version for these tests:


java version "1.8.0_131"

Java(TM) SE Runtime Environment (build 1.8.0_131-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

 

OpenSSL installation

In this experiments I'm going to use 1.1 version of OpenSSL. I installed it through homebrew and it's available at /usr/local/opt/openssl@1.1/bin/openssl. The exact version is:


/usr/local/opt/openssl@1.1/bin/openssl

OpenSSL> version

OpenSSL 1.1.0f  25 May 2017

Putting it all together


  • I built my WildFly OpenSSL libraries and copied over the java/target/wildfly-openssl-java-1.0.3.Final-SNAPSHOT.jar and macosx-x86_64/target/wildfly-openssl-macosx-x86_64-1.0.3.Final-SNAPSHOT.jar into the /libs/ folder. These are the 2 jars that contain the necessary WildFly OpenSSL support (depending on what OS you are on, you might need a different jar).
  • Given that I was going to use WildFly OpenSSL in multiple different tools which have their own different "main" classes, I decided to write a extremely basic Java agent which would just register WildFly OpenSSL as a provider. I then just pass -javaagent: as a JVM option to each of the tools/scripts I use for these tests. The code in my Java agent is pretty straight forward:

public class OpenSSLEnabler {

 public static void premain(final String agentArgs) throws Exception {
  try {
   org.wildfly.openssl.OpenSSLProvider.register();
  } catch(Exception e) {
   System.err.println("Failed to register WildFly OpenSSL provider");
   e.printStackTrace();
  }
 }
}


  • I setup the following as an environment variable to make sure this Java agent is picked up as well as OpenSSL 1.1 is used for these tests (when I enable OpenSSL as the provider):

    export KAFKA_OPTS="-javaagent:/opt/installations/kafka/1.0.0.RC4/kafka_2.12-1.0.0/libs/wildfly-openssl-javaagent-1.0.0-SNAPSHOT.jar -Dorg.wildfly.openssl.path=/usr/local/opt/openssl@1.1/lib/"

Note: The wildfly-openssl-javaagent-1.0.0-SNAPSHOT.jar is the jar containing the Java agent class that I explained above and the -Dorg.wildfly.openssl.path system property is to ensure that WildFly OpenSSL uses this specific OpenSSL installation (when I enable OpenSSL as the provider).

Performance tests details

The first round of testing will be using the SSL provider shipped in Java. We will use the kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh scripts that are shipped by Kafka itself (they are available in the bin directory of your Kafka installation).

The second round of testing will be using OpenSSL provider backed by WildFly OpenSSL. In these tests too we will use kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh scripts.

Kafka Broker and topics

In my test I'm going to create 3 topics, each with a replication factor of 1 and with partition count 1. The topics will be called kafka-ssl-perf-test-1k, kafka-ssl-perf-test-10k and kafka-ssl-perf-test-500k:

./kafka-topics.sh --create --topic kafka-ssl-perf-test-1k --partitions=1 --replication-factor=1 --zookeeper=localhost:2181

./kafka-topics.sh --create --topic kafka-ssl-perf-test-10k --partitions=1 --replication-factor=1 --zookeeper=localhost:2181

./kafka-topics.sh --create --topic kafka-ssl-perf-test-500k --partitions=1 --replication-factor=1 --zookeeper=localhost:2181

Producer tests

The producer will run with --record-size of 1024, 10240 and 512000 in 3 separate runs. Each run will generate --num-records 10000 with the respective size. Our usage of Kafka typically generates messages of lesser than 10K, so I decided to not stretch the tests for too large messages. In my test, I will use the kafka-ssl-perf-test-1k topic for 1024 sized messages, kafka-ssl-perf-test-10k for 10240 sized messages and kafka-ssl-perf-test-500k for 512000 sized messages.

Consumer tests

The consumer will be consuming (all) 10000 messages of each of these topics, in 3 separate runs. I used `--new-consumer` option for these tests since that's what we use in our application, through the Java client APIs.

Java default SSL run

As noted, by default, Kafka uses the SSL provider shipped in JRE. So it isn't necessary to configure the provider specifically. However, we will configure a few other configurations to enable SSL itself (by default Kafka uses plain text). So the broker configs that I added/changed are these, in $KAFKA_HOME/config/server.properties:

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
ssl.keystore.location=/opt/kafka-experiments/ssl-certs/keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=/opt/kafka-experiments/trust-certs.jks
ssl.truststore.password=password
ssl.protocol=TLSv1.2

The rest of the configurations are unchanged and the ones default shipped by Kafka. As you see above, the main configurations are enabling SSL and using 9093 as the port for SSL communication and using TLSv1.2 as the SSL protocol.

Start Zookeeper and Kafka broker

cd /bin/
nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &
nohup ./kafka-server-start.sh ../config/server.properties &

Run the producer perf test script

We'll pass the following producer configs (through a kafka-jre-ssl-producer.properties) to these runs:

bootstrap.servers=localhost:9093
security.protocol=SSL
ssl.protocol=TLSv1.2
ssl.truststore.location=/opt/kafka-experiments/trust-certs.jks
ssl.truststore.password=password

These configurations just enable SSL (and by default uses the JRE shipped SSL provider) with TLSv1.2 as the protocol on 9093 port.

1024 sized message

./kafka-producer-perf-test.sh --record-size 1024 --num-records 10000   --topic kafka-ssl-perf-test-1k --producer.config ./kafka-jre-ssl-producer.properties --throughput -1 > producer-jre-ssl-1k.txt

10240 sized message

./kafka-producer-perf-test.sh --record-size 10240 --num-records 10000   --topic kafka-ssl-perf-test-10k --producer.config ./kafka-jre-ssl-producer.properties --throughput -1 > producer-jre-ssl-10k.txt

512000 sized message

./kafka-producer-perf-test.sh --record-size 512000 --num-records 10000   --topic kafka-ssl-perf-test-500k --producer.config ./kafka-jre-ssl-producer.properties --throughput -1 > producer-jre-ssl-500k.txt

Note: I have the performance numbers in a table, later in this blog.

Run the consumer perf test script

We'll pass the following consumer configs (through a kafka-jre-ssl-consumer.properties) to these runs:

bootstrap.servers=localhost:9093
security.protocol=SSL
ssl.protocol=TLSv1.2
ssl.truststore.location=/opt/kafka-experiments/trust-certs.jks
ssl.truststore.password=password

Just like for the producer, these configurations enable SSL and use the default JRE provider with TLSv1.2. We will be consuming of 3 separate topics, each having messages of different sizes that we produced above. Each run uses a different and unique consumer group id.

1024 sized message

./kafka-consumer-perf-test.sh --topic kafka-ssl-perf-test-1k --new-consumer --messages 10000 --broker-list localhost:9093  --consumer.config ./kafka-jre-ssl-consumer.properties --group jre-ssl-1k > consumer-jre-ssl-1k.txt

10240 sized message

./kafka-consumer-perf-test.sh --topic kafka-ssl-perf-test-10k --new-consumer --messages 10000 --broker-list localhost:9093  --consumer.config ./kafka-jre-ssl-consumer.properties --group jre-ssl-10k > consumer-jre-ssl-10k.txt

512000 sized message

./kafka-consumer-perf-test.sh --topic kafka-ssl-perf-test-500k --new-consumer --messages 10000 --broker-list localhost:9093  --consumer.config ./kafka-jre-ssl-consumer.properties --group jre-ssl-500k > consumer-jre-ssl-500k.txt

Just like the producer numbers, I've noted these consumer numbers in a section later in this blog.

WildFly OpenSSL run

Now that we are done with the producer and consumer runs with default JRE SSL, we'll now reconfigure the Kafka broker to use OpenSSL as the provider. We won't be doing any other configuration changes to the broker configs and the producer, consumer configs we use for testing. To give this run the similar characteristics as that of our previous run, I deleted the Kafka and Zookeeper directories that store the Kafka topics. Essentially, this run is going to be from a clean slate. As noted previously, to enable WildFly OpenSSL, I configured the following environment property:

export KAFKA_OPTS="-javaagent:/opt/installations/kafka/1.0.0.RC4/kafka_2.12-1.0.0/libs/wildfly-openssl-javaagent-1.0.0-SNAPSHOT.jar -Dorg.wildfly.openssl.path=/usr/local/opt/openssl@1.1/lib/"

Here's what the relevant broker configs (in server.properties) look like now for OpenSSL:

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
ssl.keystore.location=/opt/kafka-experiments/ssl-certs/keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=/opt/kafka-experiments/trust-certs.jks
ssl.truststore.password=password
ssl.provider=openssl
ssl.protocol=TLSv1.2

As you'll notice the only additional configuration here is the ssl.provider=openssl.

We then start zookeeper and the Kafka broker as previously. Remember, we (intentionally) deleted the Kafka directories that held the topics. So we will recreate the necessary topics as we did previously.

Run the producer perf test script

We'll pass the following producer configs (through a kafka-openssl-producer.properties) to these runs:

bootstrap.servers=localhost:9093
security.protocol=SSL
ssl.protocol=TLSv1.2
ssl.provider=openssl
ssl.truststore.location=/opt/kafka-experiments/trust-certs.jks
ssl.truststore.password=password

It's the same as what we used in our previous run, except that we use ssl.provider=openssl.

1024 sized message

./kafka-producer-perf-test.sh --record-size 1024 --num-records 10000   --topic kafka-ssl-perf-test-1k --producer.config ./kafka-openssl-producer.properties --throughput -1 > producer-openssl-1k.txt

Note: You'll see the following log message, which indicates that WildFly OpenSSL is rightly picked up, which then uses the natively installed 1.1.0f version of OpenSSL:

Oct 29, 2017 8:35:25 PM org.wildfly.openssl.SSL init
INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0f  25 May 2017

10240 sized message

./kafka-producer-perf-test.sh --record-size 10240 --num-records 10000   --topic kafka-ssl-perf-test-10k --producer.config ./kafka-openssl-producer.properties --throughput -1 > producer-openssl-10k.txt

512000 sized message

./kafka-producer-perf-test.sh --record-size 512000 --num-records 10000   --topic kafka-ssl-perf-test-500k --producer.config ./kafka-openssl-producer.properties --throughput -1 > producer-openssl-500k.txt

Run the consumer perf test script

We'll pass the following consumer configs (through a kafka-openssl-consumer.properties) to these runs:

bootstrap.servers=localhost:9093
security.protocol=SSL
ssl.protocol=TLSv1.2
ssl.provider=openssl
ssl.truststore.location=/opt/kafka-experiments/trust-certs.jks
ssl.truststore.password=password

It's the same as what we used for our consumer run with JRE SSL, except that we set the ssl.provider=openssl in this case.

1024 sized message

./kafka-consumer-perf-test.sh --topic kafka-ssl-perf-test-1k --new-consumer --messages 10000 --broker-list localhost:9093  --consumer.config ./kafka-openssl-consumer.properties --group open-ssl-1k > consumer-openssl-1k.txt

Just like the producer run with OpenSSL, you should see the following log message which confirms that WildFly OpenSSL was picked up for this run:

Oct 29, 2017 8:35:25 PM org.wildfly.openssl.SSL init
INFO: WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.0f  25 May 2017

10240 sized message

./kafka-consumer-perf-test.sh --topic kafka-ssl-perf-test-10k --new-consumer --messages 10000 --broker-list localhost:9093  --consumer.config ./kafka-openssl-consumer.properties --group openssl-10k > consumer-openssl-10k.txt

512000 sized message

./kafka-consumer-perf-test.sh --topic kafka-ssl-perf-test-500k --new-consumer --messages 10000 --broker-list localhost:9093  --consumer.config ./kafka-openssl-consumer.properties --group openssl-500k > consumer-openssl-500k.txt

Final numbers (Java 8)

So let's now jump to the numbers, that we captured, from the above runs. The following is producer and consumer numbers for various message sizes that we tried above with JRE SSL and OpenSSL:

Producer Stats:


Producer Stats Message size 1024 Message size 10240 Message size 512000
JRE SSL OpenSSL JRE SSL OpenSSL JRE SSL OpenSSL
Records/sec 10857.76 14306.15 2232.64 2645.50 181.41 430.45
MB/sec 10.60 13.97 21.80 25.83 88.58 210.19
Avg. Latency (ms) 337.01 222.05 776.62 659.71 361.02 151.50
Max. Latency (ms) 568.00 387.0 1050.00 887.00 618.00 282.00
50th % latency (ms) 351 236 814 690 356 146
95th % latency (ms) 548 369 933 808 381 183
99th % latency (ms) 565 384 1016 870 522 235
99.9th % latency (ms) 568 387 1046 885 561 262


Consumer Stats:


Consumer Stats Message size 1024 Message size 10240 Message size 512000
JRE SSL OpenSSL JRE SSL OpenSSL JRE SSL OpenSSL
Data consumed MB 9.7656 9.7656 97.6563 97.6563 4882.8125 4882.8125
MB/sec 16.5239 24.5986 50.4423 97.2672 86.8797 250.4263
Total consumed messages 10000 10000 10000 10000 10000 10000
Num messages/sec 16920.4738 25188.9169 5165.2893 9960.1594 177.9296 512.8731
Rebalance time (ms) 29 17 31 16 29 17
Fetch time (ms) 562 380 1905 988 56173 19481
Fetch MB/sec 17.3766 25.6990 51.2631 98.8424 86.9245 250.6449
Fetch messages/sec 17793.5943 26315.7895 5249.3438 10121.4575 178.0215 513.3207


Summary (Java 8)

The above tables show that OpenSSL (backed by WildFly OpenSSL) out-performs the SSL provider shipped in the JRE, in both producer and consumer metrics recorded by the Kafka performance scripts. This by no means is a fine tuned performance testing or any kind of benchmark. The whole goal of this exercise was to see if it was worth the efforts to try and use OpenSSL (backed by WildFly OpenSSL) with Kafka. If the numbers/differences weren't as prominent as they are here, it wouldn't have been worth it. But as you see, the numbers show drastic improvements with WildFly OpenSSL and are promising enough to let us experiment more with OpenSSL.


Performance when using Java 9


I (and few other folks) were curious what kind of numbers we get when this same test was run with Java 9 as the runtime environment. Java 9 has some known performance improvements around SSL (like this), so I ran the entire set of tests (producer and consumer with both JRE shipped SSLEngine and WildFly OpenSSL) with Java 9 runtime. Just like for Java 8, I used the out-of-the-box settings for Kafka as well as Java 9 itself. The same set of instructions, noted previously in this article, were followed as for Java 8 to run these tests. The exact Java 9 version that was used is:


java version "9.0.1"

Java(TM) SE Runtime Environment (build 9.0.1+11)

Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

Producer Stats (Java 9)

Producer Stats Message size 1024 Message size 10240 Message size 512000
JRE-9 SSL OpenSSL JRE-9 SSL OpenSSL JRE-9 SSL OpenSSL
Records/sec 11481.05 14265.33 2480.77 2760.14 437.34 494.07
MB/sec 11.21 13.93 24.23 26.95 213.55 241.25
Avg. Latency (ms) 341.45 227.38 702.48 636.20 148.82 132.03
Max. Latency (ms) 529.00 403.00 1009.00 854.00 691.00 269.00
50th % latency (ms) 358 236 709 634 138 122
95th % latency (ms) 513 389 885 775 198 193
99th % latency (ms) 526 401 987 834 310 228
99.9th % latency (ms) 529 402 1007 853 645 249


Consumer Stats (Java 9)

Consumer Stats Message size 1024 Message size 10240 Message size 512000
JRE-9 SSL OpenSSL JRE-9 SSL OpenSSL JRE-9 SSL OpenSSL
Data consumed MB 9.7656 9.7656 97.6563 97.6563 4882.8125 4882.8125
MB/sec 13.3593 24.1723 66.6141 89.5108 233.0253 247.7076
Total consumed messages 10000 10000 10000 10000 10000 10000
Num messages/sec 13679.8906 24752.4752 6821.2824 9165.9028 477.2358 507.3052
Rebalance time (ms) 30 18 31 18 28 17
Fetch time (ms) 701 386 1435 1073 20926 19695
Fetch MB/sec 13.9310 25.2995 68.0531 91.0123 233.3371 247.9214
Fetch messages/sec 14265.3352 25906.7358 6968.6411 9319.6645 477.8744 507.7431


Summary (Java 9)

In the above numbers you'll notice that:
  • Both for producer and consumer, there's a drastic improvement in the JRE shipped SSLEngine numbers, in almost all metrics, in Java 9 as compared to its counterpart in Java 8. It's especially prominent in messages with higher sizes.
  • There's not much difference in the numbers for WildFly OpenSSL, in Java 9, as compared to its Java 8 counterpart. In fact, the consumer performance numbers of WildFly OpenSSL in Java 9 have dropped slightly when compared to Java 8. The producer performance in Java 9 with WildFly OpenSSL have however improved slightly when compared to Java 8.
  • When the numbers of producer and consumer metrics of WildFly OpenSSL with Java 9 runtime are compared with the JRE shipped SSL engine in Java 9, WildFly OpenSSL still out-performs the one shipped in JRE.
All the configurations, the Java agent code and the output of the runs are available in my github repo here

Saturday, December 27, 2014

Remote JMX access to WildFly (or JBoss AS7) using JConsole

One of the goals of JBoss AS7 was to make it much more secure by default, when compared to previous versions. One of the areas which was directly impacted by this goal was that you could no longer expect the server to expose some service on a port and get access to it without any authentication/authorization. Remember that in previous versions of JBoss AS you could access the JNDI port, the JMX port without any authentication/authorization, as long as those ports were opened for communication remotely. Finer grained authorizations on such ports for communications, in JBoss AS7, allows the server to control who gets to invoke operations over that port.

Of course, this is not just limited to JBoss AS7 but continues to be the goal in WildFly (which is the rename of JBoss Application Server). In fact, WildFly has gone one step further and now has the feature of "one single port" for all communication.


JMX communication in JBoss AS7 and WildFly


With that background, we'll now focus on JMX communication in JBoss AS7 and WildFly. I'll use WildFly (8.2.0 Final) as a reference for the rest of this article, but the same details apply (with minor changes) to other major versions of JBoss AS7 and WildFly, that have been released till date.

WildFly server is composed of "subsystems", each of which expose a particular set of functionality. For example, there's the EE subsystem which supports the Java EE feature set. Then there's the Undertow subsystem which supports web/HTTP server functionality. Similarly, there's a JMX subsystem which exposes the JMX feature set on the server. As you all are aware, I'm sure, JMX service is standardly used for monitoring and even managing Java servers and this includes managing the servers remotely. The JMX subsystem in WildFly allows remote access to the JMX service and port 9990 is what is used for that remote JMX communication.

JConsole for remote JMX access against JBoss AS7 and WildFly


Java (JDK) comes bundled with the JConsole tool which allows connecting to local or remote Java runtimes which expose the JMX service. The tool is easy to use, all you have to do is run the jconsole command it will show up a graphical menu listing any local Java processes and also an option to specify a remote URL to connect to a remote process:

# Start the JConsole
$JAVA_HOME/bin/jconsole


Let's assume that you have started WildFly standalone server, locally. Now when you start the jconsole, you'll notice that the WildFly Java process is listed in the local running processes to which you can connect to. When you select the WildFly Java instance, you'll be auto connected to it and you'll notice MBeans that are exposed by the server. However, in the context of this article, this "local process" mode in JConsole isn't what we are interested in.

Let's use the "Remote process" option in that JConsole menu which allows you to specify the remote URL to connect to the Java runtime and username and password to use to connect to that instance. Even though our WildFly server is running locally, we can use this "Remote process" option to try and connect to it. So let's try it out. Before that though, let's consider a the following few points:

  1. Remember that the JMX subsystem in WildFly allows remote access on port 9990
  2. For remote access to JMX, the URL is of the format - service:jmx:[vendor-specific-protocol]://[host]:[port]. The vendor specific protocol is the interesting bit here. In the case of WildFly that vendor-specific-protocol is http-remoting-jmx.
  3. Remember that WildFly is secure by default which means that just because the JMX subsystem exposes 9990 port for remote communication, it doesn't mean it's open for communication to anyone. In order to be allowed to communicate over this port, the caller client is expected to be authenticated and authorized. This is backed by the "ManagementRealm" in WildFly. Users authenticated and authorized against this realm are allowed access to that port.

Keeping those points in mind, let's first create a user in the Management Realm. This can be done using the add-user command line script (which is present in JBOSS_HOME/bin folder). I won't go into the details of that since there's enough documentation for that. Let's just assume that I created a user named "wflyadmin" with an appropriate password in the Management Realm. To verify that the user has been properly created, in the right realm, let's access the WildFly admin console at the URL http://localhost:9990/console. You'll be asked for username and password for access. Use the same username and password of the newly created user. If the login works, then you are good. If not, then make sure you have done things right while adding the new user (as I said I won't go into the details of adding a new user since it's going to just stretch this article unnecessarily long).

So at this point we have created a user named "wflyadmin" belonging to ManagementRealm. We'll be using this same user account for accessing the JMX service on WildFly, through JConsole. So let's now bring up the jconsole as usual:



$JAVA_HOME/bin/jconsole


On the JConsole menu let's again select the "Remote process" option and use the following URL in the URL text box:

service:jmx:http-remoting-jmx://localhost:9990

Note: For JBoss AS 7.x and JBoss EAP 6.x, the vendor specific protocol is remoting-jmx and the port for communication is 9999. So the URL will be service:jmx:remoting-jmx://localhost:9999

In the username and password textboxes, use the same user/pass that you newly created. Finally, click on Connect. What do you see? It doesn't work! The connection fails. So what went wrong?

Why isn't the JConsole remote access to WildFly not working?


You did all the obvious things necessary to access the WildFly JMX service remotely but you keep seeing that JConsole can't connect to it. What could be the reason? Remember, in one of those points earlier, I noted that the "vendor specific protocol" is an interesting bit? We use http-remoting-jmx and that protocol internally relies on certain WildFly/JBoss specific libraries, primarily for remote communication and authentication and authorization. These libraries are WildFly server specific and hence aren't part of the standard Java runtime environment. When you start jconsole, it uses a standard classpath which just has the relevant libraries that are part of the JDK/JRE.

To solve this problem, what you need to do is bring in the WildFly server specific libraries into the classpath of JConsole. Before looking into how to do that, let's see which are the WildFly specific libraries that are needed. All the necessary classes for this to work are part of the jboss-cli-client.jar which is present in JBOSS_HOME/bin/client/ folder. So all we need to do in include this jar in the classpath of the jconsole tool. To do that we use the -J option of jconsole tool which allows passing parameters to the Java runtime of jconsole. The command to do that is:

$JAVA_HOME/bin/jconsole -J-Djava.class.path=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/jconsole.jar:/opt/wildfly-8.2.0.Final/bin/client/jboss-cli-client.jar

(Note that for Windows the classpath separator is the semi-colon character instead of the colon)

Note, the server specific jar for JBoss AS 7.x and JBoss EAP 6.x is named jboss-client.jar and is present at the same JBOSS_HOME/bin/client directory location.

So we are passing -Djava.class.path as the parameter to the jconsole Java runtime, using the -J option. Notice that we have specified more than just our server specific jar in that classpath. That's because, using the -Djava.class.path is expected to contain the complete classpath.  We are including the jars from the Java JDK lib folder that are necessary for JConsole and also our server specific jar in that classpath.

Running that command should bring up JConsole as usual and let's go ahead and select the "Remote process" option and specify the same URL as before:

service:jmx:http-remoting-jmx://localhost:9990

and the same username and password as before and click Connect. This time you should be able to connect and should start seeing the MBeans and others services exposed over JMX.


How about providing a script which does this necessary classpath setup?


Since it's a common thing to try and use JConsole for remote access against WildFly, it's reasonable to expect to have a script which sets up the classpath (as above) and you could then just use that script. That's why WildFly ships such a script. It's in the JBOSS_HOME/bin folder and is called jconsole.sh (and jconsole.bat for Windows). This is just a wrapper script which internally invokes the jconsole tool present in Java JDK, after setting up the classpath appropriately. All you have to do is run:

$JBOSS_HOME/bin/jconsole.sh

What about using JConsole from a really remote machine, against WildFly?


So far we were using the jconsole tool that was present on the same machine as the WildFly instance, which meant that we have filesystem access to the WildFly server specific jars present in the WildFly installation directory on the filesystem. This allowed us to setup the classpath for jconsole to point to the jar on the local filesystem?

What if you wanted to run jconsole from a remote machine against a WildFly server which is installed and running on a different machine. In that case, your remote client machine won't be having filesystem access to the WildFly installation directory. So to get jconsole running in such a scenario, you will have to copy over the JBOSS_HOME/bin/jboss-cli-client.jar to your remote client machine, to a directory of your choice and then setup the classpath for jconsole tool as explained earlier and point it to that jar location. That should get you access to JMX services of WildFly from jconsole on a remote machine.

 

More questions?

If you still have problems getting this to work or have other questions, please start a discussion in the JBoss community forums here https://developer.jboss.org/en/wildfly/content.

Sunday, December 07, 2014

WildFly 8.2.0.Final release - Quick overview of the changes

It's been a while since I last wrote on this blog. Although I have had some topics that I wanted to blog about, I just haven't found enough time to do it. I finally decided to write this up today after I saw a mail from one of the JBoss community members, checking up on why there haven't been any updates here lately (thanks for checking, Bhaskar! :)).

Before I move on to some technical things, a quick personal update - It's now been more than a year now since I changed jobs. I no longer work at Red Hat, JBoss. My (almost) 5 years at JBoss have been very fruitful and I enjoyed being part of the (JBoss AS/WildFly) application server development team. Last year, I decided to move on to something different and the right opportunity came along and I decided to take it up. Some of you know that I've been involved with the JBoss community for longer than the 5 years that I had been employed at Red Hat. I have been a JBoss community member since around 2004/2005, so even though I have moved on from Red Hat, I am still active in the JBoss forums.

Now that you all know what I've been upto, let's move on to some technical things.

WildFly 8.2.0.Final released!


The WildFly team just released the 8.2.0.Final version of WildFly some days back. As usual, it's available for download on the project's download page http://wildfly.org/downloads/. This is mainly a bug fix (plus some features) release in the 8.x series. I for one, was pleased to see this release happen because it allows the community to receive bug fixes on top of 8.1.0.Final release, which has been tested/used in the community for quite some time now. The WildFly team has indicated that this will be the last release in the 8.x series which sounds reasonable, given that the development team has already moved on to work on the 9.x series. It's never easy to work/maintain more than one major version of the code, especially in the context of bug fixes and backward compatibility.

What does WildFly 8.2.0.Final contain?


The complete overview of changes in available in this announcement on Jason's blog http://wildfly.org/news/2014/11/20/WildFly82-Final-Released/. CDI spec upgrade and improved WebSocket support are the main items in terms of feature set. There's also this note in the release notes:
- EJBs in WARs now inherit the WAR security domain

There was a recent forum thread, where one of the users asked what that really means. Here's some background to that change https://issues.jboss.org/browse/WFLY-3102. As noted in that JIRA, this was feature request that was raised in the context of EJBs packaged in .war deployments. Most of you, I guess, will be aware that Java EE spec allows EJBs to be deployed as part of the .war deployment. What this means is that you can place your EJB classes within the .war/WEB-INF/classes or within a jar in .war/WEB-INF/lib. Although, this sounds straightforward for the end users, there are some technical implications to this (given the way Java EE "components" and "modules" are defined and configured within the server ecosystem). One such detail, is the way one configures the EJBs that are part of the .war deployment. Remember that if this EJB was part of a separate EJB module (within a .jar packaging outside of the .war) then one would use the ejb-jar.xml (and the WildFly specific jboss-ejb3.xml) as the deployment descriptors to configure it. This applies to EJBs deployed in a .war deployment too. i.e. one can use those same files for configuring EJBs. Now since these EJBs are part of a .war, the .war itself can/will have a deployment descriptor if its own (the web.xml and jboss-web.xml).

With that context, consider a case where the you have EJBs within the .war deployment and your .war deployment descriptor (the jboss-web.xml) configures a specific security domain for that deployment. Now remember that the EJBs too can configure a security domain (in jboss-ejb3.xml) and if none is configured and security is enabled on some EJBs, then the default "other" security domain gets used. So let's say your .war deployment, in which the EJBs reside, states that it wants to use "foo-bar" security domain and the EJBs, within that deployment, don't specify any specific security domain. So what should one expect in such scenario? Should the EJBs use the security domain configured at the .war level or should they default to the "other" security domain (since the EJB deployment descriptors don't specify any specific security domain configuration). The previous versions of WildFly had decided to use the default "other" security domain for the EJBs in such a case. Of course, the EJBs could use a jboss-ejb3.xml to set a different security domain, one which matches the jboss-web.xml. So that JIRA which I linked to earlier requested for a better, smarter and a more logical default in such cases. So starting this 8.2.0.Final version of WildFly, if you have a .war containing the EJBs and the EJBs don't define a security domain, then the security domain for any secured EJBs in that deployment *defaults* to the one that's defined at the .war deployment level. If the .war deployment doesn't set any specific security domain, then it ultimately, defaults to the "other" security domain. A good and logical change IMO. This will reduce some of the "surprises" that users have reported with previous version of WildFly, when it came to the security domain usage of EJBs in .war deployments.

For more about this, you can read the discussion here https://developer.jboss.org/thread/250375 and ask any questions you have around this, in that thread.

What are the other notable things in WildFly 8.2.0.Final?


Although, not specific to 8.2.0.Final, the WildFly release contain a "patch" distribution which you can use if you already are using WildFly 8.1.0.Final and just want to "upgrade" to this new release. WildFly 8 has patch management built in and one can apply this patch on top of an existing 8.1.0.Final version.

What's next for WildFly?


As noted earlier, the WildFly development team has moved on to the next version of the project. Work is now continuing on 9.x version which already has a Alpha version released. So going forward, from what I have read, the releases will happen in the 9.x series.