Friday, November 20, 2009

WSO2 ESB Tips & Tricks 01: Sequences and Proxy Services

As promised, here is the first article of the “WSO2 ESB Tips & Tricks” series. The main objective of this post is to introduce two very important functional components of the ESB, namely sequences and proxy services. These are perhaps the most commonly used functional components of WSO2 ESB. They can be used to implement even the most complex of messaging systems and enterprise integration patterns using the service bus. During the course of this article I will briefly explain some of the use cases of sequences and proxy services along with some useful information on how to use them in WSO2 ESB.
WSO2 ESB supports four modes of operation.
  1. Service mediation
  2. Message mediation
  3. Task scheduling
  4. Eventing
In the service mediation mode, WSO2 ESB acts as a proxy to a real Web Service hosted in a Web Services container like Apache Axis2 or WSO2 WSAS. The ESB exposes a virtual Web Service which can accept requests from clients. The requests are processed, mediated and forwarded to the actual service implementation by the ESB. Any responses coming from the service implementation are mediated and forwarded to the clients. Proxy services are used to get the ESB operating in this mode.
In the message mediation mode, WSO2 ESB functions as a message router. It can filter, transform, drop messages or forward them to remote endpoints for further processing when operating in this mode. Sequences are used to define the message mediation behavior of the ESB. A sequence is a series of mediators, where each mediator is an entity that can accept messages and carry out a predefined task on them. WSO2 ESB provides you with a wide range of mediators designed to carry out various processing tasks on the messages. You can mix and match these available mediators to develop sequences. For instance a sequence comprising of the log mediator and the send mediator will act as a simple log-and-forward message flow. You can have any number of mediators in a sequence and a sequence can also dispatch messages to other sequences if needed.
A proxy service is a combination of three sequences and a target endpoint. The target endpoint is for the actual service implementation to which the messages will be forwarded after mediation. The three sequences are as follows.
  • In-sequence: All the incoming requests to the proxy service are dispatched to the in-sequence. This sequence defines how the requests should be processed before forwarding them to the target endpoint.
  • Out-sequence: All the responses coming back from the backend service implementation are dispatched to this sequence. It defines how the responses should be handled before sending them back to the clients. The out sequence can also forward the responses to a given service endpoint, thus effectively linking up multiple services. This way a single request to the proxy service will trigger multiple service invocations. The response from the 1st service is fed to the 2nd service as the request.
  • Fault sequence: If an error occurs during service mediation the faulty message is handed to the fault sequence for error handling work. Depending on our requirements the fault sequence can be used log the error, ignore it or send a SOAP fault back to the client indicating that something went wrong.
Now that you have a basic understanding of sequences and proxy services let’s take a look at some of the use cases where these components come in handy. Proxy services can be used to expose an existing service over a different schema (message format). In scenarios like this one, proxy services can transform the messages flowing back and forth. Also since proxy services can be exposed over multiple transports they can be used to expose an existing service over a different transport. For an example we can take a simple HTTP service and expose it over JMS by creating a proxy service for the HTTP service and exposing it over JMS. The ESB will take care of switching the communication protocols. In addition, proxy services are useful in adding QoS features (security, RM etc) to ordinary message flows. As an example, we can take an unsecured service, create a proxy service for it and engage security on the proxy service to add security to the backend service implementation.
Sequences can be used to audit, filter, control and transform messages flowing through a network. Certain advanced features like access control, load balancing and fail over are also possible with sequences. In WSO2 ESB all the incoming messages which are not destined to a proxy service are dispatched to the “main” sequence. The main sequence can further distribute them to other sequences and they in turns can call even more sequences. This is somewhat analogous to how a C program works. The application runtime invokes the main procedure and it in turns can call other procedures.
Enough with the theory! Let’s get practical and see how to create sequences and proxy services in WSO2 ESB. Here actually you have several options. You can either use the XML based Synapse configuration language to define sequences and proxy services, or use the set of graphical tools that come with WSO2 ESB. To create a sequence graphically, first start the ESB server and logon to the management console. (By default the web based console is accessible at https://localhost:9443/carbon) Sign in as an administrator (By default the username and the password are both “admin”). Click on the “Sequences” option in the left panel. This will show you a list of existing sequences. By default the “main” sequence and the global “fault” sequence is listed on this page.
To create a new sequence, click on “Add Sequence”. This will bring up the on-line sequence editor. Start by giving a name to the sequence. Then go ahead and start adding mediators.
To further customize the behavior of a mediator instance click on the mediator and a dialog box will appear just below the sequence editor panel. Once you are done creating the sequence click on “Save” to save and close the editor. The newly created sequence will now show up on the list of available sequences.
Ceating a proxy service is just as easy. Click on the Proxy Service link in the left panel to start the proxy service creation wizard.
Simply go through the 3-step wizard and specify an in-sequence, target endpoint, out-sequence and a fault sequence for the proxy service. When specifying these items you can either import an already existing sequence/endpoint or create one on the spot by selecting the “Anonymous” option.
Once you finish the wizard the new service will get created and deployed on the server. If you gave the name “FooProxy” to the service it will be exposed on the HTTP endpoint http://localhost:8280/services/FooProxy.
Now if you click on the “Synapse” link on the left navigation bar you can see that the overall ESB configuration has been updated with your changes.
The UI also allows you to enable statistics collection on sequences and proxy services, enable tracing on them and modifying their configuration. Play around with the various options available to you and get familiar with sequences, proxy services and the ESB management console. We will be using these a lot in the days to come.

Thursday, November 19, 2009

New Kids in Town

The team at WSO2 just released the latest version of WSO2 Carbon (v2.0.2) along with a whole bunch of Carbon based products. Log on to the WSO2 Oxygen Tank to lay your hands on the following smoking hot releases, right from the WSO2 oven ;)
  • WSO2 Enterprise Service Bus v2.1.2
  • WSO2 Web Services Application Server v3.1.2
  • WSO2 Business Process Server v1.1.0
  • WSO2 Mashup Server v2.0.1
  • WSO2 Identity Server v2.0.2
You will be particularly interested in our Business Process Server and Mashup Server releases since we haven't done any releases of them for a fairly long time. Needless to say that all new releases have many new features, bug fixes and enhancements over the previous versions. So download today and experience the power of SOA.

Sunday, November 15, 2009

Introducing "WSO2 ESB Tips and Tricks"

WSO2 ESB has kept me quite busy for the last couple of months. The ESB 2.1 release in July was soon followed by a 2.1.1 release and work is now underway for the 2.1.2 release. That’s way too much activity in such a short period, for an open source project. Anyway, since I’m spending a lot of time working on WSO2 ESB, I thought I might as well blog about my work in a regular basis. So in the next few months, I will be publishing a series of blog posts regarding WSO2 ESB and how to use it to implement real life integration scenarios. This series of blogs, which I have named “WSO2 ESB Tips and Tricks”, will surely benefit many folks who are either just learning SOA concepts or looking to integrate a bunch of apps using an ESB.
You can consider this post to be the 0th article of the “WSO2 ESB Tips and Tricks” series. So before I move on to the 1st post of the series I think I should briefly describe what WSO2 ESB is and why you should consider it as an option for enterprise integration. So here goes:
WSO2 ESB is a fast and lightweight, enterprise service bus. It is based on Apache Synapse, the lightweight ESB from the ASF (speaking of which, “Happy Birthday ASF”). It supports many messaging standards including SOAP, WS-* standards and REST as well as a variety of application and transport layer protocols like HTTP/S, Mail, JMS, VFS, AMQP, TCP and FIX. WSO2 ESB comes with a rich collection of mediators and other functional components which can be used out of the box to implement even the most complex integration patterns. Message routing, transformation, protocol switching, load balancing, clustering and service chaining are some of the common features supported by WSO2 ESB. Controlling and managing WSO2 ESB is also a trivial task thanks to the Web based management. Many of the common system administration tasks such as user management, certificate management and statistics collection are all available as inbuilt features of the server and the management console. WSO2 ESB is shipped with an embedded WSO2 Governance Registry, which makes it easy to store and manage SOA metadata. In addition to that the ESB can be configured to work with an externally hosted WSO2 Governance Registry if needed.
WSO2 ESB is also based on WSO2 Carbon, the OSGi based components framework for SOA. WSO2 Carbon makes it possible to easily install and configure additional features into the ESB runtime. Custom code and third part libraries can also be deployed into the server without any hassle.
Perhaps the best thing about WSO2 ESB (apart from it being super fast) is that it is totally free and open source. Binary and source distributions are available for download through the WSO2 Oxygen Tank. All artifacts are released under the business friendly Apache Software License 2.0. WSO2 also offers training and commercial support for users that require them.
As I have already mentioned a couple of times, WSO2 ESB is well known as a fast ESB. It can easily handle over 2500 transactions per second while maintaining constant memory usage. This level of performance is neither a coincidence nor an accident, but by clever and careful design. Two of the most significant features of WSO2 ESB architecture can be listed as follows:
  • Non-blocking HTTP transport : The HTTP server worker threads of the ESB do not get blocked over network I/O. The Apache HTTP Core-NIO based transport makes it possible to accept many concurrent connections and process more messages than any other Java HTTP transport implementation.
  • Pull Parsing and Streaming Model: WSO2 ESB uses Apache AXIOM, the StAX based XML infoset model. Therefore it does not build the object model for incoming messages, unless it has to. Incoming bytes are streamed through the ESB without touching the payload. This reduces memory usage significantly and also saves many valuable CPU cycles.
These architectural elements combine with the Carbon framework to deliver the best performance imaginable along with seamless modularization.
Some of the latest features that we have added to WSO2 ESB like rule based mediation and EDA (Eventing) support make WSO2 ESB suitable for implementing even the most complicated SOA platforms.
I think this explanation pretty much justifies why you should give WSO2 ESB a try. It is fast, feature rich and user friendly by all means. And don’t forget, it’s free and open source too. So grab a copy of the latest binary distribution today and get started. If you need any help we got tons of free documentation and articles on the WSO2 Oxygen Tank. This is in addition to our mailing lists and user forums.
I hope you will enjoy the “WSO2 ESB Tips and Tricks”.

Friday, September 18, 2009

A Tour to the Past

Something non-technical for a change today.....
I've always wanted to visit a museum and see some artifacts from the Jurassic age. I'm very enthusiastic about learning more about the giant lizards that roamed the earth millions of years ago and I've always wished that I could see those gigantic dinosuar fossils live, without a TV screen. Finally I got the opportunity when I was in Salt Lake City, Utah, a few weeks back. The Utah Museum of Natural History is a fascinating place with lots of things to see, learn and do. They exhibit some very rare artifacts illustrating the Jurassic ages, world history, American history, geography and plant and wildlife. What I liked most was that instead of just seeing stuff, the museum has lots of facilities where you can watch educational videos, read books and managazines and take part in interactive games and activities. The use of modern technology in the musuem is mind blowing.

The musuem has special admittance rates for University students (you just need to show your student ID) and they have a special gallery named "Toadally Frogs", dedicated to kids, featuring some rare species of toads and frogs.


It indeed was a wonderful and fulfilling experience vsiting the natural history musuem of Utah. If you are visiting Salt Lake City, make sure you allocate some time to go and visit this exceptional place. Oh and did I mention that they have a huge collection of natural minerals and rare stones along with a gallery full of live insects? You don't wanna miss those.


Saturday, September 12, 2009

Enterprise Messaging with Synapse, WSO2 ESB and WebSphere MQ

I've been recently playing a lot with Apache Synapse, WSO2 ESB and IBM WebShpere MQ (WMQ). My intention was to use WMQ as a JMS provider for Synapse and WSO2 ESB. IBM WebSphere MQ being a very popular and matured messaging solution, I figured that getting Synapse and WSO2 ESB to work with WMQ and document the integration process would really help the IBM, Apache and WSO2 communities. Originally I anticipated this application integration to be a very comlpex and tedious task. But for much of my delight, it turned out to be a very easy and simple process in the end. What was little complicated was to get WMQ properly installed and get its JMS features configured. But the integration with Synapse and WSO2 ESB was indeed a piece of cake.
So here I'm going to list the steps hat one should follow to get WMQ integrated with Synapse or WSO2 ESB. Start by downloading the required software. Binary distributions of Apache Synapse and WSO2 ESB can be downloaded from their respective websites. A trial version of the IBM WMQ V7.0 can be downloaded from here. In addition to these application you will also require Java 5 or higher along with Apache ANT to run some of the samples described here.
Installing Apache Synapse or WSO2 ESB doesn't require any additional steps. You simply need to extract the downloaded archives and that's all. However WMQ installation is not so easy. It is a fairly long and time consuming process which requires carefulness and patience. Anyway the installation process is well documented and so it shouldn't be a problem. For your convenience I'm describing the WMQ installation procedure for Linux here. The necessary commands to be executed at each step are given in italic font.

Installing WMQ
1. Create an empty directory and extract the downloaded WMQ archive into it
2. Add a new user group called 'mqm' to the system
groupadd mqm
3. Add a new user called 'mqm' to the system. Add the user to the 'mqm' group. Set the user's home directory to /var/mqm
useradd -d /var/mqm -g mqm mqm
4. Create the directory /var/mqm and change its owner to mqm
mkdir /var/mqm
chown mqm:mqm mqm
5. Create the directory /opt/mqm
mkdir /opt/mqm
6. Create another directory anywhere on the disk and set a symlink to it from /opt/mqm
mkdir /home/hiranya/mqm
ln -s /home/hiranya/mqm /opt/mqm
7. Install libstdc++5 library on the system (using Synaptic or apt-get on Debian/Ubuntu systems) if not already installed
8. As root go to the directory where WMQ is extracted and run the license display script
./mqlicense.sh
9. Install the MQ runtime and the server using rpm
rpm -ivh MQSeriesRuntime-7.0.0-0.i386.rpm MQSeriesServer-7.0.0-0.i386.rpm --nodeps
10. Install the MQ samples
rpm -ivh MQSeriesSamples-7.0.0-0.i386.rpm --nodeps
11. Install the MQ client
rpm -ivh MQSeriesClient-7.0.0-0.i386.rpm --nodeps
That wasn't too bad, was it? Now it's time to verify the installation. Here's what you got to do.

Verifying the Installation
1. Login as user mqm (ssh mqm@localhost)
2. Create a new queue manager
crtmqm -q venus.queue.manager
3. Start the queue manager
strmqm
4. Start MQSC program (a new shell will appear - without a prompt)
runmqsc
5. Define a local queue in the MQSC shell
define qlocal (orange.queue)
6. Terminate MQSC
end
7. Go to /opt/mqm/samp/bin and run the amqsput sample program to place a message on to the queue
./amqsput ORANGE.QUEUE
8. The above command should be followed up by some sample message text of your choice and a blank line
9. Run the following command to pull the message back from the queue
./amqsget ORANGE.QUEUE
10. So far so good! We got the server properly installed. Now on to the client!
11. Create a new queue manager
crtmqm -q saturn.queue.manager
12. Start the queue manager
strmqm
13. Start MQSC
runmqsc
14. Create a new queue
define qlocal (queue1)
15. Create a connection channel to the queue
define channel (channel1) chltype (svrconn) trptype (tcp) mcauser ('mqm')
16. Create a listener (specify a port)
define listener (listener1) trptype (tcp) control (qmgr) port (8585)
17. Start the listener
start listener (listener1)
18. Terminate MQSC
end
19. Create a system environment variable to point to the channel created above
export MQSERVER='CHANNEL1/TCP/localhost(8585)'
20. Go to /opt/mqm/samp/bin and run the amqsputc to place a message in the queue (using the client API)
./amqputc QUEUE1 saturn.queue.manager
21. Now run amqsgetc to receive the message
./amqgetc QUEUE1 saturn.queue.manager
22. Excellent! We got the client installed properly too :)
Now that we got WMQ installed let's try to get its JMS features enabled.

Setting Up JMS
1. Install WMQ classes for JMS
rpm -ivh MQSeriesJava-7.0.0-0.i386.rpm --nodeps
2. Start MQSC (This will start mqsc for the last created queue manager - saturn.queue.manager)
runmqsc
3. Define a JMS channel for the queue manager
define channel(java.channel) chltype(svrconn) trptypr(tcp)
4. Define a listener
define listener(listener.tcp) trptype(tcp) port(1414)
5. Start the listener
start listener(listener.tcp)
6. Setup the classpath variable and the MQ_JAVA_LIB_PATH to run the IVT programt
export CLASSPATH=/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/
export MQ_JAVA_LIB_PATH=/opt/mqm/java/lib
7. Go to /opt/mqm/java/bin and invoke the IVT sample program
./IVTRun -nojndi
./IVTRun -nojndi -client -m saturn.queue.manager -host localhost -channel JAVA.CHANNEL
8. If the IVT program can recieve and send JMS messages we are done! Now let's proceed to enabling JNDI support
9. Go to the /opt/mqm/java/bin directory and open the JMSAdmin.config file. Edit the PROVIDER_URL property to point to an empty directory of your choice on the file system. This directory will be used the JNDI provider source (eg: /var/mqm/jndi). If the specified directory does not exist in the file system create it.
10. Run the IVTSetup tool to create the default set of JNDI bindings
./IVTSetup
11. Now run the IVTRun tool as follows
./IVTRun -url "file:/var/mqm/jndi" -icf com.sun.jndi.fscontext.RefFSContextFactory
12. By now we have enabled and verified JNDI support. Let's use the JMSAdmin tool to make some modifications in the JNDI bindings
13. Fireoff /opt/java/bin/JMSAdmin and run the following commands:
ALTER QCF(ivtQCF) TRANSPORT(CLIENT)
ALTER QCF(ivtQCF) QMGR(saturn.queue.manager)
14. Run the IVTRun tool again as mentioned in step 11. This time messages will be sent to the saturn queue manager.
Now we are all set. It's time to get WMQ integrated with Synapse and WSO2 ESB. Let's start with Apache Synapse.

Synapse Integration
Follow the steps given below as the user 'mqm'. Trying to run Synapse as a different user caused some JMS security exceptions.
1. Copy the following jar files from /opt/mqm/java/lib to SYNAPSE_HOME/lib
  • com.ibm.mqjms.jar
  • fscontext.jar
  • providerutil.jar
  • com.ibm.mq.jmqi.jar
  • dhbcore.jar
2. Enable the JMS listener in the axis2.xml. Configure the default connection factory as follows.
<parameter name="default">
<parameter name="java.naming.factory.initial">com.sun.jndi.fscontext.RefFSContextFactory</parameter>
<parameter name="java.naming.provider.url">file:/var/mqm/jndi</parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName">ivtQCF</parameter>
<parameter name="transport.jms.ConnectionFactoryType" locked="false">queue</parameter>
<parameter name="transport.jms.Destination">ivtQ</parameter>
</parameter>
Note that we are using the JNDI configuration used by the IVT sample program. We are using the same queue connection factory and the JMS queue to define the default connection factory.

3. Create the following proxy service in the synapse.xml (this is a simplified version of sample 250)
<proxy name="JMSProxy" transports="jms">
<target>
<inSequence>
<log level="full"/>
<drop/>
</inSequence>
</target>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>application/xml</default>
</rules>
</parameter>
</proxy>
4. Start Synapse - go to SYNAPSE_HOME/bin and run ./synapse.sh
5. Go to SYNAPSE_HOME/samples/axis2Client/src/samples/userguide and open the GenericJMSClient.java source file. Make the following changes in the code.
  • Set the jms_dest property default value to "ivtQ" (line 45)
  • Set the java.naming.provider.url to "file:/var/mqm/jndi" (line 82)
  • Set the java.naming.factory.initial to "com.sun.jndi.fscontext.RefFSContextFactory" (line 85)
  • Set the lookup key to "ivtQCF" (line 89)
6. Now from the SYNAPSE_HOME/samples/axis2Client directory run the following command to run the JMS client application which will send messages to the ivtQ.
ant jmsclient -Djms_type=pox -Djms_payload=IBM

7. At this point Synapse will pick the messages from the queue and log them on the console.

WSO2 ESB 2.1 Integration
Follow the steps given below as user 'mqm'
1. Copy the Websphere MQ client jars (mentioned above under Synapse Integration) to ESB_HOME/repository/components/lib
2. Enable the JMS listener in axis2.xml and configure the default JMS connection factory as follows
<parameter name="default">              
<parameter name="java.naming.factory.initial" >com.sun.jndi.fscontext.RefFSContextFactory</parameter>
<parameter name="java.naming.provider.url" >file:/var/mqm/jndi</parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName" >ivtQCF</parameter>
<parameter name="transport.jms.ConnectionFactoryType" >queue</parameter>
<parameter name="transport.jms.Destination">BOGUSQ</parameter>
</parameter>
Note that we have specified a queue named 'BOGUSQ' as the default destination. This is because we need to use ivtQ for our proxy service only. If we use 'ivtQ' here all the services deployed in ESB (XKMS, echo, wso2carbon-sts) will start listening on the same queue. You also need to login to JMSAdmin and create the queue named 'BOGUSQ'. In JMSAdmin shell run the following commands.
DEFINE Q(BOGUSQ) QMGR(saturn.queue.manager)
ALTER Q(BOGUSQ) QUEUE(QUEUE1)
If the QUEUE1 does not exist, first login to MQSC and create it.
3. Add the proxy service we used with Synapse to ESB_HOME/conf/synapse.xml
4. Start WSO2 ESB
./wso2server.sh -DuseSynapseXML
5. Use the sample client we used in Synapse to send messages to the queue

Wednesday, September 9, 2009

WSO2 Carbon Shines in Open Source Space

Awesome news folks!
WSO2 Carbon, the open source SOA platform, the team of which I'm also a part of, has won an award in "InfoWorld Best of Open Source Software (Bossie) 2009". Bossie awards cover a wide range of software categories and WSO2 Carbon won the award in the "Platforms and Middleware" category. With this, WSO2 Carbon has been recognized by InfoWorld as one of top 40 open source products available for business and IT. Here I quote the official announcement from InfoWorld which declares the award winners.
"Two leading lights of open source SOA are MuleSource and WSO2. The Mule ESB takes our prize for the finest enterprise service bus in the open source pantheon, while WSO2 receives a Bossie for its Carbon framework, the basis of a completely componentized SOA platform."
Fantastic.... Given the fact that WSO2 Carbon is still a very new software platform, this is a great achivement indeed. It just shows how fast WSO2 Carbon has matured as a complete SOA platform and how fast its community has expanded in the last few months.
Read the complete publication from InfoWorld and the official news article from WSO2 for more details on this great achievement by the WSO2 Carbon team.