WSO2 ESB Tips & Tricks 05: Error Handling with Sequences and Proxy Services

The job of an Enterprise Service Bus is to act as the backbone of an organization’s SOA. It is the spine through which all the systems and applications within the enterprise communicate with each other. At times an ESB might even help to integrate internal systems with external applications so that complex cross cutting business tasks such as supply chain management and customer relationship management can be carried out with precision and in perfect synchronism. As such an ESB often has to deal with many wire level protocols, messaging standards and remote APIs. But applications and networks are full of errors. Applications often crash due to various bugs. Network routers and links often get into states where they cannot pass messages through with the expected efficiency. These error conditions are very likely to cause a fault or trigger a runtime exception in the ESB. A good ESB should provide simple yet powerful mechanisms for dealing with such errors. So today, I’m going to discuss a little bit about error handling options provided in WSO2 ESB.

WSO2 ESB gives you the concept of fault sequences. A fault sequence is just like any other sequence, a collection of mediators. A fault sequence can be associated with another sequence or a proxy service. When the sequence or the proxy service encounters an error during mediation or while forwarding a message out, the message which triggered the error will be delegated to the specified fault sequence. Using the available mediators we can log the erroneous message, forward it to a special error tracking service, send a SOAP fault back to the client indicating the error or even send a mail to the system admin. It is not mandatory to associate each sequence and proxy service with a fault sequence. In situations where a fault sequence is not specified explicitly, a default fault sequence will be used to handle errors. Sample 4 in WSO2 ESB documentation shows how to specify a fault sequence with a regular mediation sequence.

Whenever an error occurs in WSO2 ESB, the mediation engine attempts to provide as much information as possible on the error to the user. This is done by initializing a set of property values on the erroneous message. These properties are:

• ERROR_CODE
• ERROR_MESSAGE
• ERROR_DETAIL
• ERROR_EXCEPTION

Within the fault sequence we can access these property values using the get-property XPath function. Sample 4 uses the log mediator as follows to log the actual error message:

<log level="custom">
  <property name="text" value="An unexpected error occured"/>
  <property name="message" expression="get-property('ERROR_MESSAGE')"/>       
</log>

Note how the ERROR_MESSAGE property is being used to get the error message text. You can also write special custom mediators for error handling tasks. In such cases you can use the MessageContext API to retrieve above mentioned property values from the message.

String errorMsg = (String) messageContext.getProperty(“ERROR_MESSAGE”);

Exception e = (Exception) messageContext.getProperty(“ERROR_EXCEPTION”);

WSO2 ESB also makes it possible to send a SOAP fault back to the client whenever an exception occurs in the ESB. This is done using the makeFault mediator. The makeFault mediator converts the current message into a SOAP fault. The user can specify the error code and the reason statement to be included in the fault message. Once the SOAP fault is constructed it can be sent to the client using the send mediator. Sample 5 in WSO2 ESB documentation shows exactly how to do this. Here’s the makeFault mediator configuration from sample 5:

<makefault>
  <code value="tns:Receiver" xmlns:tns="http://www.w3.org/2003/05/soap-envelope"/>           
  <reason expression="get-property('ERROR_MESSAGE')"/>
</makefault>

Note how the ERROR_MESSAGE property is used to set the original error message as the reason phrase of the SOAP fault. Keep in mind that the makeFault mediator only converts the message into a fault message. To send it to a client or an endpoint you need to use the send mediator. In sample 5 we mark the message as a response using the property mediator and invoke the send mediator so that it is sent back to the client which sent the original request.

Assume you want to send an E-Mail to the system admin whenever the ESB encounters an error. How do you do it? You simply need to enable the mail transport sender in the ESB configuration (by editing the axis2.xml file or using the UI). Configure the transport sender with the appropriate mail server settings. Then in your fault sequence you can simply use the send mediator as follows to send an E-Mail out.

<property name="Subject" value="An Error Occurred in the ESB" scope="transport"/>
<property name="OUT_ONLY" value="true"/>
<send>   
  <endpoint>       
     <address uri=”mailto:admin@yourdomain.com”/>
  </endpoint>
</send>

So these are the error handling options available for sequences and proxy services in WSO2 ESB. It also provides some more error handling options for endpoints. That however is a different matter entirely and hence I’m leaving it for a future blog post.

Apache Attacks: A Real Eye Opener

It seems year 2010 is gradually turning into the year of cyber crimes. The year started off with news reports on a very sophisticated and targeted attack on Google corporate infrastructure. According to Google, the attackers were successful in stealing valuable intellectual property. Now after about four months from the Google incident, a massive attack has been carried out on the Apache Software Foundation. Attackers have exploited a previously unknown vulnerability in the Atlassian JIRA instance used by the ASF, to gain root access to the server hosting the JIRA instance. Attackers also messed up the JIRA instance to install some sniffers that can capture and log user passwords. According to the Apache infra team a whole bunch of user passwords have been compromised as a result of the attack. This attack was later followed by another attack, this time directly on the Atlassian IT infrastructure, which has also exploited the same security hole in JIRA.

The Apache infra team did a pretty amazing job to mitigate the threat and take control of the situation. They have also taken some additional security measures to prevent such disasters in the future while Atlassian has rolled out patches for the aforementioned security hole.

It is hard to imagine what makes somebody attack the ASF IT infrastructure. The whole world knows that we are just doing non-profit, voluntary work at ASF. Nobody gains a monetary benefit by attacking the ASF. May be it is just to compromise the passwords and get to the users/committers. Most people use the same password to login to multiple systems (Gmail, Facebook, Yahoo, Hotmail etc). So if the attacker can get the password for one system, he can gain access to all the other systems.

All in all, this incident is a real eye opener to all of us surfing the Internet. No system is 100% safe and no system is invulnerable. An attacker with sufficient patience and skill will always find a way in. It is up to the users to be careful and minimize the chance of something horrible happening.

Moral of the Story:

We should use multiple passwords to login to different systems as much as possible. We should use strong passwords at all times. And we should definitely change the passwords in a regular basis to mitigate the effects of a possible brute force attack. (We hear these stuff everyday but how many of us actually do it? That's the problem. We should actually put these guidelines into action.)

PS: If you are an Apache committer and did not change the JIRA password yet, please do it NOW!!!

WSO2 ESB Tips & Tricks 04: Troubleshooting JMS

WSO2 ESB works really well with JMS. The JMS transport adapter used by WSO2 ESB comes from the Apache Axis2 project and it makes use of JNDI to connect to various JMS brokers. Therefore WSO2 ESB can work with pretty much any JMS broker that offers JNDI support. The product ships with sample configurations for Apache ActiveMQ integration, but it can be easily modified for integration with a JMS broker like IBM MQ or SwiftMQ.

Enabling JMS in WSO2 ESB is a simple 2-step process.

1. Deploy the JMS client libraries to the repository/components/lib directory (these libraries will be used by the ESB to connect to the JMS broker)
2. Enable JMS transport receiver and the sender by modifying the axis2.xml file or using the management console

Having followed the above mentioned process you can start creating JMS endpoints to forward messages and JMS proxy services to receive messages. However while implementing certain JMS related scenarios you might run into some error conditions. The rest of this blog post focuses on a couple of such commonly encountered problems and how to get rid of them, once and for all.

CNF? NCDF?

If you ever run into any ClassNotFoundExceptions or NoClassDefFoundExceptions while using JMS support in WSO2 ESB, chances are you haven’t deployed all the required client libraries. Please inspect what you have copied into repository/components/lib directory and make sure the required class is available in one of the deployed jars.

Also note that WSO2 ESB ships with geronimo-jms library which contains the javax.jms packages. Therefore you do not have to deploy them again.

HTTP Header Conversion Problem

When forwarding HTTP traffic to a JMS queue using WSO2 ESB, you might at times run into an error similar to the one given below.

ERROR JMSSender Error creating a JMS message from the axis message context

javax.jms.MessageFormatException: MQJMS1058: Invalid message property name: Content-Type

This exception is thrown by the JMS client libraries used to connect with your JMS broker and the problem is specific to the JMS broker being used. I have encountered this issue mostly with IBM Websphere MQ, but I guess there might be other broker implementations that may display the same issue. So here’s what happens….

The incoming HTTP message contains a bunch of HTTP headers that contain the ‘-‘ character. Some notable examples are ‘Content-length’ header and the ‘Transfer-encoding’ header. When ESB attempts to forward the message over JMS it sets the headers of the incoming message to the outgoing JMS message as JMS properties. According to the JMS spec the ‘-‘ character is prohibited in JMS property names. Some JMS brokers like ActiveMQ do not strictly check for this and hence they won’t complain. But certain other brokers will throw exceptions similar to the one shown above.

Even though the problem sounds very complex and tricky the solution is almost trivial. We just need to get rid of the troublemaking HTTP headers from the message before it is delivered over JMS. You can make use of the property mediator as follows to achieve this:

<property action="remove" name="Content-Length" scope="transport">
<property action="remove" name="Accept-Encoding" scope="transport">
<property action="remove" name="User-Agent" scope="transport">
<property action="remove" name="Content-Type" scope="transport">

JMS Property Data Type Mismatch

Sometimes when the ESB attempts to forwards a message over JMS, the client libraries may complain, saying that the data type of a particular message property is invalid. You may often run into this problem if you are using the property mediator to manipulate property values set on the message. This is because certain implementations of JMS, have restrictions on data types of properties. The property mediator always sets property values as strings.

To overcome this problem you will have to revise your mediation sequences and avoid manipulating property values which should contain non-string values. If you must set a non-string property value you will have to write a simple custom mediator to get the job done. Starting from WSO2 ESB 3.0 (scheduled to go out this month), the property mediator is type aware. Therefore to get rid of this problem you can set properties with specific data types. For an example if you want to set a property named foo with the integer value 12345 use the property mediator as follows:

<property name="foo" value="12345" type="INTEGER" scope="transport/">

If you leave the type attribute out the property value will be set as a string.

Too Many Threads? Out of Memory?

With certain JMS brokers WSO2 ESB tends to spawn new worker threads indefinitely until it runs out of memory and crashes. I have observed this behavior with SwiftMQ. This problem is caused by a bug in the underlying Axis2 engine and there is a simple workaround to the problem. In your mediation sequence engage the property mediator as follows:

<property action="remove" name="transportNonBlocking" scope="axis2">

This will prevent the ESB from creating new worker threads unlimitedly and everything will be back to normal. Note that you can always use a JMX client like jconsole to monitor the active threads and memory consumption of the ESB.