Monday, May 14, 2018

Apache Ivy 2.5.0-rc1 released - Now allows timeouts on resolvers

A few weeks back, we released the 2.5.0-rc1 version of Apache Ivy. Apache Ivy is a dependency management build tool, which usually is a used in combination with Apache Ant. The download is available on the project download page

This release is significant since the last release of Apache Ivy was way back in December 2014. So it's more than 3 years since the last official years. During these past few years, the project development stalled for a while. I use Apache Ivy in some of our projects and have been pretty happy with the tool. It's never a good sign to see one of your heavily used tools to be no longer under development or even have bug fixes. So a year or so back, I decided to contribute some bug fixes to the project. Over time, the project management committee invited me to be part of the team.

We decided that the first obvious, immediate goal would be to revive the project and do a formal release with bug fixes. This 2.5.0-rc1 is the result of that effort which started almost a year back. A lot of changes have gone into this release and also a good number of enhancements have made it into this release. This release has been a result of contributions from various different members from the community. The complete list of release notes is available here

We intentionally named this release 2.5.0-rc1 (release candidate) since it's been a while we have done an official release and also given the nature of changes. Please give this release a try and let us know how it goes. Depending on the feedback, we will either release 2.5.0 or 2.5.0-rc2. As usual, some of us from the development team keep an active watch in the ivy user mailing list. So if you have any feedback or questions, please do drop a mail to us, there.

Now coming to one of the enhancements in this release - there's been more than one. One of the issues I personally had was if the repository, backing a dependency resolver configured for Ivy, had some connectivity issues, the build would just hang. This was due to the inability to specify proper timeouts for communicating with these repositories through the resolver. As of this release, Ivy now allows you to configure timeouts for resolvers. This is done through the use of (the new) timeout-constraints element in your Ivy settings file. More details about it are here. Imagine you have a url resolver which points to some URL. The URL resolver would typically look something like:

<url name="foo">
  <ivy pattern=.../>
  <artifact pattern=.../>
  <artifact pattern=.../>

Let's now try and configure a connection timeout for this resolver. The first thing you would do is define a named timeout-constraint, like below:

        <timeout-constraint name="timeout-1" connectionTimeout="60000" />

The value for the name attribute can be anything of your choice. The value for connectionTimeout attribute is represented as a timeout in milli seconds. In the above example, we configure the "timeout-1" timeout-constraint to be of 1 minute. You can even specify a readTimeout which too is in milli seconds. More about this element can be found in the documentation.

As you might notice, we have just defined a timeout-constraint here but haven't yet instructed Ivy to use this constraint for some resolver. We do that in the next step, where we set the "timeoutConstraint" attribute on the URL resolver that we had seen before:

<url name="foo" timeoutConstraint="timeout-1">
  <ivy pattern=.../>
  <artifact pattern=.../>
  <artifact pattern=.../>

Notice that the value of "timeoutConstraint" attribute now points to "timeout-1" which we defined to have a 1 minute connection timeout. With this, when this URL resolver gets chosen by Ivy for dependency resolution, this connection timeout will be enforced and if the connections fails to be established within this timeout, then an exception gets thrown instead of the build hanging forever.

Although the example uses a URL resolver to setup the timeout constraint, this feature is available for all resolvers that are shipped out of the box by Ivy. So you can even use it with the ibiblio resolver (which communicates with Maven central) too.

Like I noted earlier, please do give this release a try and let us know how it goes.

Friday, May 11, 2018

VMWare vijava - The curious case of "incorrect user name or password" exception

In one of the projects I have been involved in, we use yavijava (which is a fork of vijava) library to interact with vCenter which hosts our VMs. vCenter exposes various APIs through their webservice endpoints which are invoked through HTTP(s). The yavijava library has necessary hooks which allows developers to use a HTTP client library of their choice on the client side to handle invocations to the vCenter.

In our integration, we plugged in the Apache HTTP client library, so that the yavijava invocations internally end up using this HTTP library for interaction. Things mostly worked fine and we were able to invoke the vCenter APIs. I say mostly, because every once in a while we kept seeing exceptions like:

InvalidLogin : Cannot complete login due to an incorrect user name or password.

This was puzzling since we were absolutely sure that the user name and password we use to interact with the vCenter was correct. Especially since all of the previous calls were going through fine, before we started seeing these exceptions.

The exception stacktrace didn't include anything more useful and neither did any other logs. So the only option that I was left with was to go look into the vCenter (server side) event logs to see if I can find something. Luckily, I had access to a setup which had a vSphere client, which I then used to connect to the vCenter. The vSphere client allows you to view the event logs that were generated on the vCenter.

Taking a look at the logs, showed something interesting and useful. Every time, we had run into this "incorrect user name or password" exception on the client side, there was a corresponding event log on the vCenter server side at INFO level which stated "user cannot logon since user is already logged on". That event log was a good enough hint to give an idea of what might be happening.

Based on that hint, the theory I could form was, somehow for an incoming (login) request, vCenter server side notices something on the request which gives it an impression that the user is already logged in. Given my background with Java EE technologies, the immediate obvious thing that came to mind was that the request was being attached with a "Cookie" which the server side uses to associate requests against a particular session. Since I had access to the client side code which was issuing this login request, I was absolutely sure that the request did not have any explicitly set Cookie header. So that raised the question, who/where the cookie was being associated with the request. The only place that can happen, if it's not part of the request we issued, is within the HTTP client library. Reading up the documentation of Apache HTTP client library confirmed the theory that the HTTP client was automagically associating a (previously generated) Cookie against the request.

More specifically, the HTTP client library uses pooled connections. When a request is made, one of the pooled connections (if any) gets used. What was happening in this particular case  was that, a previous login would pick up connection C1 and the login would succeed. The response returned from vCenter for that login request would include a Cookie set in the response header. The Apache HTTP client library was then keeping track of this Cookie against the connection that was used. Now when a subsequent login request arrived, if the same pooled connection C1 gets used for this request, then the HTTP client library was attaching the Cookie that it kept track against this connection C1, to this new request. As a result, vCenter server side ends up seeing that the incoming login request has a Cookie associated with it, which says that there's already a logged in session for that request. Hence, that INFO message in the event logs of vCenter. Of course, the error returned isn't that informative and in fact a bit misleading since it says the username/password is incorrect.

Now that we know what's going on, the solution was pretty straightforward. Apache HTTP client library allows you to configure Cookie policy management. Since in our case, we wanted to handle setting the Cookie explicitly on the request, we decided to go with the "ignoreCookies" policy which can be configured on the HTTP client. More about this can be found in the HTTP client library documentation (see the "Manual Handling of Cookies" section). Once we did this change, we no longer saw this exception anymore.

There isn't much information about this issue anywhere that I could find. The closest I could find was this forum thread It didn't have a conclusive solution, but it does appear that it's the same issue that the user there was running into (almost 7 years back!)