How the World's Fastest ESB was Made

A couple of years ago, at WSO2 we implemented a new HTTP transport for WSO2 ESB. Requirements for this new transport can be summarized as follows:

1. Ultra-fast, low latency mediation of HTTP requests.
2. Supporting a very large number of inbound (client-ESB) and outbound (ESB-server) connections concurrently (we were looking at several thousand concurrent connections).
3. Automatic throttling and graceful performance degradation in the presence of slow or faulty clients and servers.

The default non-blocking HTTP (NHTTP) transport from Apache Synapse, which we were also using in WSO2 ESB, supported the above requirements up to a certain extent but we wanted to do better. The default transport was very generic and it was designed to offer reasonable performance in all the integration scenarios the ESB could potentially participate in. However HTTP load balancing, HTTP URL routing (URL rewriting) and HTTP header-based routing are some of the most widely used integration patterns in the industry and to support these use cases well, we needed a specialized transport. 

The old NHTTP transport was based on a dual buffer model. Incoming message content was placed in a SharedInputBuffer and the outgoing message content was placed in a SharedOutputBuffer. Apache Axiom, Apache Axis2 and the Synapse mediation engine sit between the two buffers, reading from the input buffer and writing to the output buffer. This architecture is illustrated in the following diagram.

The key advantage of this architecture is that it enables the ESB (mediators) to intercept all the messages and manipulate them in any way necessary. The main downside is every message happens to go through the Axiom layer, which is not really necessary in cases like HTTP load balancing and HTTP header-based routing. Also the overhead of moving data from one buffer to another was not always justifiable in this model. So when we started working on the new HTTP transport we wanted to get rid of these limitations. We knew that this might result in a not-so-generic HTTP transport, but we were willing to pay that price at the time.

So after some very interesting brainstorming sessions, an exciting 1-week long hackathon followed by several months of testing, bug-fixing and refactoring we came up with what’s today known as the HTTP pass-through transport. This transport was based on a single buffer model and completely bypassed the Axiom layer. The resulting architecture is illustrated below.

The HTTP pass-through transport was first released in June 2011 along with WSO2 ESB 4.0. Back then it was disabled by default and the user had to enable it by uncommenting a few entries in the axis2.xml file. The performance numbers we were seeing with the new transport were simply remarkable. WSO2 also published some of these benchmarking results in a March 2012 article. However at this point the 2 main limitations in the new transport were starting to give us headaches.

1. Configuration overhead (Users had to explicitly enable the transport depending on their target use cases)
2. Cannot support any integration scenario that requires HTTP content manipulation (because Axiom was bypassed, any mediator attempting to access the message payload would not get anything useful to work with)

In addition to these technical issues there were other process related issues that we had to deal with. For instance maintaining two separate HTTP transports was twice as work for the developers and testers. We found that because the pass-through transport was not used as the default, it often lagged behind the default NHTTP transport in terms of features and stability. So after a few brainstorming sessions we decided to try and make the pass-through transport the default HTTP transport in Apache Synapse/WSO2 ESB. But this required making the content manipulation use cases (content aware use cases) work with the new transport. This implied bringing Axiom back into the picture, the very thing we wanted to avoid in our initial implementation. So in order to balance out our performance and heterogeneous integration requirements we came up with the idea of “on-demand message parsing in the mediation engine”.

In this new model, each mediator instance belongs to one of two classes.

1. Content-unaware mediators – Mediators that never access the message content in anyway (eg: drop mediator)
2. Content-aware mediators – Mediators that always access the message content (eg: xslt mediator)

We also identified a third class known as conditionally content-aware mediators. These mediators could be either content-aware or content-unaware depending on their exact instance configuration. For an example a simple log mediator instance, configured as <log/> is content-unaware. However a log mediator configured as <log level=”full”/> would be content-aware since it’s expected to log the message payload. Similarly a simple property mediator instance such as <property name=”foo” value=”bar”/> is content-unaware but <property name=”foo” expression=”/some/xpath”/> could be content-aware depending on what the XPath expression does. In order to capture this content-awareness characteristic of mediator instances at runtime, we introduced a new method (isContentAware) to the top level Mediator interface of Synapse. The default implementation in AbstractMediator class returns true by default so as to maintain backward compatibility. 

With this change in place we modified the mediation engine to check the content-awareness of property of each mediator at runtime before submitting a message to it. List mediators such as the SequenceMediator would run the check recursively on its child mediators to obtain the final value. Assuming that messages are always received through the pass-through HTTP transport, the mediation engine would invoke a special message parsing routine whenever a mediator is detected to be content-aware. It is in this special routine that we bring Axiom into the picture. Therefore if none of the mediators in a given flow or a service is content-aware, the pass-through transport works as it usually does without ever engaging Axiom. But whenever a content-aware mediator is involved, we bring Axiom in. This way we can reap the performance benefits of the pass-through transport while supporting all integration scenarios of the ESB. Since we engage Axiom on-demand we get the best possible outcome for all scenarios. For instance a simple pass through proxy would always work without any Axiom interactions. An XSLT proxy that transforms requests would engage Axiom only in the request flow. Response flow would operate without parsing the messages.

Another tricky problem we encountered was dealing with message parsing itself. For instance how do we parse a message and then send it out when there is only one buffer provided by the underlying pass-through transport? Ideally we need two buffers to read the incoming message from and write the outgoing message to. Also the fact that the Axis2 message builder framework can only handle streams posed a few problems. The buffer we maintained in the pass-through transport was a Java NIO ByteBuffer instance. So we needed to adapt the buffer into a stream implementation whenever the mediation engine engages Axiom. We solved the first problem by implementing our message builder routine to create a second output buffer whenever Axiom is dragged into the picture. The outgoing messages are serialized into this second buffer and the pass-through transport was modified to pick the outgoing content from the second buffer when it’s available. Writing an InputStream implementation that can wrap a ByteBuffer instance solved the second problem.

One last problem that needed to be solved was handling security. In Synapse/WSO2 ESB, security is handled by Apache Rampart, which runs as an Axis2 module that intercepts the messages before they hit the mediation engine. So on-demand parsing at the mediation engine doesn’t work in this scenario. We need to parse the messages before Rampart intercepts them. We solved this issue by introducing a new smart handler to the Axis2 handler chain, which intercepts every message and performs an early parse if security is engaged on the flow. The same solution can be extended to support other modules that require parsing message payload in the Axis2 handler chain.

The reason I decided to compile this blog is because WSO2 folks just released WSO2 ESB 4.6. And this release is based on the new model I’ve described here. Pass-through transport is what the users now get by default. The WSO2 team has also published some performance figures that clearly indicate what the new design is capable of. It turns out the latest release of WSO2 ESB outperforms all the major open source ESB vendors by a significant margin. This release also comes with a new XSLT mediator (Fast XSLT) that operates on the top of the pass-through model of the underlying transport and a new streaming XPath implementation based on Antlr.

The next step of this effort would be to get these improvements integrated into the Apache Synapse code base. This work is already underway and you can monitor its progress through SYNAPSE-913 and SYNAPSE-920.

Busting Synapse and WSO2 ESB Myths

Paul Fremantle, PMC chair of the Apache Synapse project and CTO of WSO2, has written a very interesting blog post addressing some of the myths concerning Apache Synapse and WSO2 ESB. As of now both projects are quite popular, mature and have a very large user base including some of the largest organizations in the world. Surprisingly there are still some people who believe that these projects do not fall under the category of ESB (Enterprise Service Bus) implementations. In his latest post, Paul gives a clear and complete answer to all these misbeliefs, and backs it up with a wide range of facts.

ESB is one of those things in the IT world which don't have a proper standard definition. The best definitions I've come across attempt to align the term along the following cues:

• An architectural construct that provides fundamental services to complex architectures
• An entity that acts as a hub connecting many diverse systems
• A central driver that facilitates Enterprise Application Integration (EAI)

Apache Synapse and WSO2 ESB pass with flying colors on all the above criteria. They provide an array of fundamental services to the systems and architectures that rely on them.  Some of these basic services are:

• Message passing, routing and filtering
• Message transformation
• Protocol conversion
• QoS enforcement (security, reliable delivery etc)
• Logging, auditing and monitoring

Because Synapse and WSO2 ESB do such a good job providing these fundamental services, they can be used to integrate a large number of heterogeneous systems in an enterprise setting. As Paul has also pointed out in his post, Synapse and WSO2 ESB are currently used in hundreds of production deployments all around the world to connect various applications, implemented using various technologies (both open source and proprietary) running on various platforms (Windows, Linux, .NET, J2EE, LAMP, cloud..you name it). In other words Synapse and WSO2 ESB are widely used as centralized drivers that facilitate EAI. The configuration model of Synapse and WSO2 ESB is so agile and powerful that practically any EAI pattern can be implemented on top of them. In fact there are tons of samples, articles and tutorials that explain how various well-known EAI patterns can be implemented using these 'ESB implementations'.

One thing that I've learnt from writing code to Synapse is that it has a very flexible enterprise messaging model. Support for any wire level protocol or any message format can be easily implemented on top of this model and can be deployed as a separate pluggable module. During the last few years, I myself have contributed to the implementation of following adapters/connectors on various occasions:

• FIX transport
• SAP transport (IDoc and BAPI support)
• MLLP transport and HL7 message formats
• CSV and various other office document formats
• Thrift connector
• Numerous other custom binary protocols based on TCP/IP

This is just a bunch of stuff that I've had the privilege of implementing for Synapse/WSO2 ESB. I know for a fact that other committers of Synapse and WSO2 ESB have been working on supporting dozens of other protocols, message formats and mediators. Thanks to all this hard work Synapse and WSO2 ESB are currently two of the most powerful and feature-complete ESB implementations anyone will ever come across. Also the existence of connectors for so many protocols and applications is a testament to the agility and flexibility that Synapse and WSO2 ESB can bring in as ESB products.

Another aspect of Synapse/WSO2 ESB that has been questioned many times is their ability to support RESTful integrations (Paul also addresses this issue in his post). This confusion stems from the fact that Synapse uses SOAP as its intermediary message format. Without going into too many technical details, I'd just like to point out that one of the largest online marketplace and auctioning providers in the world uses Synapse/WSO2 ESB to process several hundred millions of REST calls in a daily basis. The new API support we have implemented in Synapse makes it absolutely simple to design, implement and expose RESTful APIs on Synapse/WSO2 ESB. In fact I recently published an article which demonstrates through practical examples how powerful RESTful applications can be implemented using Synapse/WSO2 ESB while supporting advanced REST semantics such as HATEOAS. The recently released WSO2 API Manager product which supports exposing rich web APIs with support for API key management is also based on Synapse/WSO2 ESB.

I think I have made my point. Both Synapse and WSO2 ESB are two excellent ESB choices if you're looking to adopt SOA or enterprise integration within your organization. Their wide range of features is only second to the very high level of performance they offer in terms of high throughput and low resource utilization. Please also go through the post made by Paul, where he has explained some of the above issues with low level technical details. I particularly like his analogy concerning Heisenberg's principle of uncertainty :) 

Apache Synapse 2.0 Released

The Apache Synapse team reached a very important milestone last week. After a long but very busy 2 and half years, we announced the general availability of Apache Synapse version 2.0. The previous release of Synapse was version 1.2 which was released way back in year 2008. The Apache Synapse project has grown a lot in terms of code, features and community over the past 2 years and therefore you will find a horde of new features, improvements, bug fixes and samples in the latest release. Some of the noteworthy new features in this release are:

• New, fine-grained configuration model
• Hot deployment and hot update support for configuration artifacts
• Priority based mediation support
• Comprehensive eventing capabilities with WS-Eventing support
• Secure vault for encrypting passwords in configuration files
• File locking support in the VFS transport for concurrent polling
• URLRewrite mediator for fast and simple URL rewriting
• Synapse configuration observer API
• Multiple identity support in the HTTPS transport
• Enhanced JMX monitoring support for the NHTTP transport
• Dead letter channel implementation (experimental)
• Synapse XAR Maven plug-in for generating configuration artifacts

Apache Synapse is a lightweight Enterprise Service Bus (ESB) released under the Apache Software License. It is based on the tried and tested Apache Axis2 SOAP engine. It has excellent support for SOAP, REST, POX, JSON and a variety of wire level transports (HTTP/S, JMS, File transport, FIX...). The non-blocking HTTP transport of Synapse enables it to handle very high volumes of HTTP traffic over thousands of concurrent connections.

Apache Synapse also supports many WS-* standards including WS-Security, WS-ReliableMessaging, and WS-Policy. It can be easily linked up with any SOA registry for metadata management and governance purposes. Synapse has a very flexible configuration model built into it, which makes it one of the easiest ESB applications to learn and adopt.

If you are involved in any SOA projects or system integration activities, Synapse may have something to offer. So please feel free to grab the binary distro and take it for a spin. Feed back welcome on Synapse user list.

New Configuration Model of Synapse

If you are trying out a latest Apache Synapse build off the SVN trunk, you will notice that the Synapse configuration model has gone through some significant changes lately. In the past, Synapse was programmed to load the entire mediation configuration (sequences, endpoints, proxy services, tasks etc) from a single XML file, named synapse.xml. Synapse would parse this XML file at startup and construct an object model known as the SynapseConfiguration which contains the definitions of all active mediation components at runtime. While this approach was simple and clean, it had a number of limitations:

• Configuration management becomes a nightmare as the size of the synapse.xml grows. A typical heavy duty configuration would consist of dozens of proxy services, sequences and endpoints. When all these are packed into a single XML file, it becomes difficult to locate a single configuration item quickly.
• With all configuration artifacts stored in a single flat file, it is almost impossible to develop satisfactory tooling support for configuration development and maintenance.
• A team of developers cannot work on configuring a single Synapse instance at the same time.
• Hot deployment is a feature that Synapse has been lacking for years. Being able to hot deploy a proxy service or a sequence into Synapse without having to restart the service bus is a great convenience at development time. But a configuration model based on a single XML file is not at all capable of handling such requirements.

Considering these drawbacks, I implemented a new configuration model for Synapse, known as the multi XML configuration builder. The new model, which is now the default configuration model in Synapse, loads the configuration from a structured file hierarchy, instead of loading the mediation configuration from a single XML file. With this model in place, each endpoint, sequence and proxy service has to be defined in separate XML files. The directory structure for storing these individual configuration files is as follows:

synapse-config
   |
   +-- registry.xml
   +-- endpoints
   +-- event-sources
   +-- local-entries
   +-- proxy-services
   +-- sequences
   `-- tasks

As you can see there are separate dedicated directories for each type of artifacts. Each of these directories can house zero or more XML configuration files. Each file must have the .xml extension to be recognized by the Synapse configuration builder. If you take a look at the source code you will notice that I have enforced this restriction using a Java FileFilter:

    private static FileFilter filter = new FileFilter() {
       public boolean accept(File pathname) {
           return (pathname.isFile() && pathname.getName().endsWith(".xml"));
       }
   };

   private static void createProxyServices(SynapseConfiguration synapseConfig, String rootDirPath)
           throws XMLStreamException {

       File proxyServicesDir = new File(rootDirPath, PROXY_SERVICES_DIR);
       if (proxyServicesDir.exists()) {
           if (log.isDebugEnabled()) {
               log.debug("Loading proxy services from : " + proxyServicesDir.getPath());
           }
           File[] proxyDefinitions = proxyServicesDir.listFiles(filter);
           for (File file : proxyDefinitions) {
               try {
                   OMElement document = parseFile(file);
                   ProxyService proxy = SynapseXMLConfigurationFactory.defineProxy(
                           synapseConfig, document);
                   proxy.setFileName(file.getName());
                   SynapseArtifactDeploymentStore.getInstance().addArtifact(
                           file.getAbsolutePath(), proxy.getName());
               } catch (FileNotFoundException ignored) {}
           }
       }
   }

The mediation registry is defined in the registry.xml file which should be placed at the top level of the file hierarchy.

So does the multi XML configuration builder solve the problems in the old configuration model? Let’s consider the facts:

• With Synapse configuration broken down into smaller, manageable pieces the whole configuration becomes easier to manage and keep track of. As long as the XML files are named appropriately, it is extremely easy to quickly locate a particular configuration item. We recommend using the artifact names to name the corresponding XML files. For an example the file containing the definition of the FooProxy can be named FooProxy.xml.
• With the multi XML configuration builder, developing powerful and elegant tools for creating pieces of the service bus configuration becomes a trivial task. Also one can use conventional configuration management tools and version controlling systems such as Subversion to store and manage the configuration artifacts.
• A team of developers can now work on configuring Synapse. Each developer in the team can work on his own configuration file or set of files.
• Supporting hot deployment is now feasible. As a matter of fact, Ruwan implemented hot deployment and hot update support for Synapse based on the multi XML configuration builder a few weeks back. This feature is now available in the Synapse trunk and will be available for the next Synapse release.

So there you go. Target achieved. There is one little glitch in this approach though. That is how do we handle backward compatibility with older Synapse versions? For instance how can a Synapse 1.2 user, who has a single synapse.xml file, migrate to a new Synapse version? We have provided a solution for that as well. In the new Synapse configuration file hierarchy you can place a synapse.xml file at the top level (alongside with registry.xml). All the mediation components defined in this synapse.xml will be loaded to the service bus at startup along with any other components defined inside the individual directories. So a Synapse 1.2 user can simply copy the existing synapse.xml file to the synapse-config directory in a new Synapse distribution, and it will be picked up by the service bus. In addition to this convenience feature, we are planning on developing some migration tools that can help users to easily migrate an old Synapse configuration file onto a newer version of Synapse.

As far as Synapse is concerned each configuration builder should be associated with a corresponding configuration serializer implementation. Serializers are used to convert the SynapseConfiguration object model back to the textual/XML form. So for the multi XML configuration builder I developed a matching multi XML configuration serializer which can save a SynapseConfiguration object model to a file hierarchy. Similar to the configuration builder this implementation was also heavily dependent on Java file IO APIs. However, after a while we realized that the serializer is not working as expected on certain platforms; most notably on Windows. After running some debug sessions and doing some on-line reading I realized that the Java file IO operations are not consistent on every platform. As a result sometimes the serializer would encounter trouble creating a new file or moving an existing file on Windows.

At this point, my colleague, Rajika suggested using Apache Commons IO API for file manipulation. Commons IO API provides a nice layer of abstraction on top of the standard Java IO APIs. It handles all file IO operations in a consistent and platform independent manner. So I got rid of almost all the Java file IO code in the multi XML configuration serializer and replaced them with corresponding calls to the Commons IO API. Some sample code fragments are shown below:

    private void cleanUpDirectory()  throws Exception {
       // If the target directory already exists and contains any files simply rename it to
       // create a backup - This method does not delete the target directory
       if (rootDirectory.exists() && rootDirectory.isDirectory() &&
               rootDirectory.listFiles().length > 0) {

           if (log.isDebugEnabled()) {
               log.debug("The directory :" + rootDirectory.getPath() + " already exists. " +
                       "Creating a backup.");
           }

           backupDirectory = new File(rootDirectory.getParentFile(), "__tmp" +
                   new GregorianCalendar().getTimeInMillis());
           FileUtils.moveDirectory(rootDirectory, backupDirectory);
       }

       // Create a new target directory
       FileUtils.forceMkdir(rootDirectory);
   }

   private void writeToFile(OMElement content, File file) throws Exception {
       File tempFile = File.createTempFile("syn_mx_", ".xml");
       OutputStream out = new FileOutputStream(tempFile);
       XMLPrettyPrinter.prettify(content, out);
       out.flush();
       out.close();

       FileUtils.copyFile(tempFile, file);
       FileUtils.deleteQuietly(tempFile);
   }

FileUtils is the entry point for file IO operations in the Commons IO API. It provides you with a range of useful methods for managing and manipulating files.

Note that when writing to a file we first write the whole thing to a temporary file and once completed we copy the temp file to the final location. This was required to handle certain edge cases in the hot update implementation. Without that the hot updater will attempt to pickup and process half baked files (Again mostly on Windows – It seems Windows writes to files in chunks).

All in all, Apache Synapse is now equipped with a powerful new configuration model, a matching configuration serializer and hot deployment capabilities that leverage the new configuration model. Speaking of hot deployment, it is also based on the Axis2 hot deployment framework. Therefore it is configured in the axis2.xml file. We are yet to do a Synapse release with all these new exciting features but if you are itching to try them out feel free to grab a nightly build. Also be aware that WSO2 ESB 3.0, which is based on Synapse, has been released and that release contains all these new improvements.

Introduction to Apache Synapse @ Apache Asia 2009

Yesterday (3rd December 2009) I conducted a technical presentation titled “Introduction to Apache Synapse” at the Apache Asia Roadshow 2009 held at Colombo, Sri Lanka. For everyone’s benefit I have published the presentation slides on SlideShare. The presentation went well, and so far I have received pretty good feedback from the participants. There were a few very interesting questions from the audience at the end of the session too.

In this presentation I have addressed the areas like problems of enterprise integration, ESB pattern, key features of Synapse, Synapse architecture and the configuration model. I hope you will find it interesting and useful.

Introduction to Apache Synapse

View more documents from Hiranya Jayathilaka.

Introducing "WSO2 ESB Tips and Tricks"

WSO2 ESB has kept me quite busy for the last couple of months. The ESB 2.1 release in July was soon followed by a 2.1.1 release and work is now underway for the 2.1.2 release. That’s way too much activity in such a short period, for an open source project. Anyway, since I’m spending a lot of time working on WSO2 ESB, I thought I might as well blog about my work in a regular basis. So in the next few months, I will be publishing a series of blog posts regarding WSO2 ESB and how to use it to implement real life integration scenarios. This series of blogs, which I have named “WSO2 ESB Tips and Tricks”, will surely benefit many folks who are either just learning SOA concepts or looking to integrate a bunch of apps using an ESB.

You can consider this post to be the 0th article of the “WSO2 ESB Tips and Tricks” series. So before I move on to the 1st post of the series I think I should briefly describe what WSO2 ESB is and why you should consider it as an option for enterprise integration. So here goes:

WSO2 ESB is a fast and lightweight, enterprise service bus. It is based on Apache Synapse, the lightweight ESB from the ASF (speaking of which, “Happy Birthday ASF”). It supports many messaging standards including SOAP, WS-* standards and REST as well as a variety of application and transport layer protocols like HTTP/S, Mail, JMS, VFS, AMQP, TCP and FIX. WSO2 ESB comes with a rich collection of mediators and other functional components which can be used out of the box to implement even the most complex integration patterns. Message routing, transformation, protocol switching, load balancing, clustering and service chaining are some of the common features supported by WSO2 ESB. Controlling and managing WSO2 ESB is also a trivial task thanks to the Web based management. Many of the common system administration tasks such as user management, certificate management and statistics collection are all available as inbuilt features of the server and the management console. WSO2 ESB is shipped with an embedded WSO2 Governance Registry, which makes it easy to store and manage SOA metadata. In addition to that the ESB can be configured to work with an externally hosted WSO2 Governance Registry if needed.

WSO2 ESB is also based on WSO2 Carbon, the OSGi based components framework for SOA. WSO2 Carbon makes it possible to easily install and configure additional features into the ESB runtime. Custom code and third part libraries can also be deployed into the server without any hassle.

Perhaps the best thing about WSO2 ESB (apart from it being super fast) is that it is totally free and open source. Binary and source distributions are available for download through the WSO2 Oxygen Tank. All artifacts are released under the business friendly Apache Software License 2.0. WSO2 also offers training and commercial support for users that require them.

As I have already mentioned a couple of times, WSO2 ESB is well known as a fast ESB. It can easily handle over 2500 transactions per second while maintaining constant memory usage. This level of performance is neither a coincidence nor an accident, but by clever and careful design. Two of the most significant features of WSO2 ESB architecture can be listed as follows:

• Non-blocking HTTP transport : The HTTP server worker threads of the ESB do not get blocked over network I/O. The Apache HTTP Core-NIO based transport makes it possible to accept many concurrent connections and process more messages than any other Java HTTP transport implementation.
• Pull Parsing and Streaming Model: WSO2 ESB uses Apache AXIOM, the StAX based XML infoset model. Therefore it does not build the object model for incoming messages, unless it has to. Incoming bytes are streamed through the ESB without touching the payload. This reduces memory usage significantly and also saves many valuable CPU cycles.

These architectural elements combine with the Carbon framework to deliver the best performance imaginable along with seamless modularization.

Some of the latest features that we have added to WSO2 ESB like rule based mediation and EDA (Eventing) support make WSO2 ESB suitable for implementing even the most complicated SOA platforms.

I think this explanation pretty much justifies why you should give WSO2 ESB a try. It is fast, feature rich and user friendly by all means. And don’t forget, it’s free and open source too. So grab a copy of the latest binary distribution today and get started. If you need any help we got tons of free documentation and articles on the WSO2 Oxygen Tank. This is in addition to our mailing lists and user forums.

I hope you will enjoy the “WSO2 ESB Tips and Tricks”.

Enterprise Messaging with Synapse, WSO2 ESB and WebSphere MQ

I've been recently playing a lot with Apache Synapse, WSO2 ESB and IBM WebShpere MQ (WMQ). My intention was to use WMQ as a JMS provider for Synapse and WSO2 ESB. IBM WebSphere MQ being a very popular and matured messaging solution, I figured that getting Synapse and WSO2 ESB to work with WMQ and document the integration process would really help the IBM, Apache and WSO2 communities. Originally I anticipated this application integration to be a very comlpex and tedious task. But for much of my delight, it turned out to be a very easy and simple process in the end. What was little complicated was to get WMQ properly installed and get its JMS features configured. But the integration with Synapse and WSO2 ESB was indeed a piece of cake.

So here I'm going to list the steps hat one should follow to get WMQ integrated with Synapse or WSO2 ESB. Start by downloading the required software. Binary distributions of Apache Synapse and WSO2 ESB can be downloaded from their respective websites. A trial version of the IBM WMQ V7.0 can be downloaded from here. In addition to these application you will also require Java 5 or higher along with Apache ANT to run some of the samples described here.

Installing Apache Synapse or WSO2 ESB doesn't require any additional steps. You simply need to extract the downloaded archives and that's all. However WMQ installation is not so easy. It is a fairly long and time consuming process which requires carefulness and patience. Anyway the installation process is well documented and so it shouldn't be a problem. For your convenience I'm describing the WMQ installation procedure for Linux here. The necessary commands to be executed at each step are given in italic font.

Installing WMQ

1. Create an empty directory and extract the downloaded WMQ archive into it

2. Add a new user group called 'mqm' to the system

groupadd mqm

3. Add a new user called 'mqm' to the system. Add the user to the 'mqm' group. Set the user's home directory to /var/mqm

useradd -d /var/mqm -g mqm mqm

4. Create the directory /var/mqm and change its owner to mqm

mkdir /var/mqm

chown mqm:mqm mqm

5. Create the directory /opt/mqm

mkdir /opt/mqm

6. Create another directory anywhere on the disk and set a symlink to it from /opt/mqm

mkdir /home/hiranya/mqm

ln -s /home/hiranya/mqm /opt/mqm

7. Install libstdc++5 library on the system (using Synaptic or apt-get on Debian/Ubuntu systems) if not already installed

8. As root go to the directory where WMQ is extracted and run the license display script

./mqlicense.sh

9. Install the MQ runtime and the server using rpm

rpm -ivh MQSeriesRuntime-7.0.0-0.i386.rpm MQSeriesServer-7.0.0-0.i386.rpm --nodeps

10. Install the MQ samples

rpm -ivh MQSeriesSamples-7.0.0-0.i386.rpm --nodeps

11. Install the MQ client

rpm -ivh MQSeriesClient-7.0.0-0.i386.rpm --nodeps

That wasn't too bad, was it? Now it's time to verify the installation. Here's what you got to do.

Verifying the Installation

1. Login as user mqm (ssh mqm@localhost)

2. Create a new queue manager

crtmqm -q venus.queue.manager

3. Start the queue manager

strmqm

4. Start MQSC program (a new shell will appear - without a prompt)

runmqsc

5. Define a local queue in the MQSC shell

define qlocal (orange.queue)

6. Terminate MQSC

end

7. Go to /opt/mqm/samp/bin and run the amqsput sample program to place a message on to the queue

./amqsput ORANGE.QUEUE

8. The above command should be followed up by some sample message text of your choice and a blank line

9. Run the following command to pull the message back from the queue

./amqsget ORANGE.QUEUE

10. So far so good! We got the server properly installed. Now on to the client!

11. Create a new queue manager

crtmqm -q saturn.queue.manager

12. Start the queue manager

strmqm

13. Start MQSC

runmqsc

14. Create a new queue

define qlocal (queue1)

15. Create a connection channel to the queue

define channel (channel1) chltype (svrconn) trptype (tcp) mcauser ('mqm')

16. Create a listener (specify a port)

define listener (listener1) trptype (tcp) control (qmgr) port (8585)

17. Start the listener

start listener (listener1)

18. Terminate MQSC

end

19. Create a system environment variable to point to the channel created above

export MQSERVER='CHANNEL1/TCP/localhost(8585)'

20. Go to /opt/mqm/samp/bin and run the amqsputc to place a message in the queue (using the client API)

./amqputc QUEUE1 saturn.queue.manager

21. Now run amqsgetc to receive the message

./amqgetc QUEUE1 saturn.queue.manager

22. Excellent! We got the client installed properly too :)

Now that we got WMQ installed let's try to get its JMS features enabled.

Setting Up JMS

1. Install WMQ classes for JMS

rpm -ivh MQSeriesJava-7.0.0-0.i386.rpm --nodeps

2. Start MQSC (This will start mqsc for the last created queue manager - saturn.queue.manager)

runmqsc

3. Define a JMS channel for the queue manager

define channel(java.channel) chltype(svrconn) trptypr(tcp)

4. Define a listener

define listener(listener.tcp) trptype(tcp) port(1414)

5. Start the listener

start listener(listener.tcp)

6. Setup the classpath variable and the MQ_JAVA_LIB_PATH to run the IVT programt

export CLASSPATH=/opt/mqm/java/lib/com.ibm.mqjms.jar:/opt/mqm/samp/jms/

export MQ_JAVA_LIB_PATH=/opt/mqm/java/lib

7. Go to /opt/mqm/java/bin and invoke the IVT sample program

./IVTRun -nojndi

./IVTRun -nojndi -client -m saturn.queue.manager -host localhost -channel JAVA.CHANNEL

8. If the IVT program can recieve and send JMS messages we are done! Now let's proceed to enabling JNDI support

9. Go to the /opt/mqm/java/bin directory and open the JMSAdmin.config file. Edit the PROVIDER_URL property to point to an empty directory of your choice on the file system. This directory will be used the JNDI provider source (eg: /var/mqm/jndi). If the specified directory does not exist in the file system create it.

10. Run the IVTSetup tool to create the default set of JNDI bindings

./IVTSetup

11. Now run the IVTRun tool as follows

./IVTRun -url "file:/var/mqm/jndi" -icf com.sun.jndi.fscontext.RefFSContextFactory

12. By now we have enabled and verified JNDI support. Let's use the JMSAdmin tool to make some modifications in the JNDI bindings

13. Fireoff /opt/java/bin/JMSAdmin and run the following commands:

ALTER QCF(ivtQCF) TRANSPORT(CLIENT)

ALTER QCF(ivtQCF) QMGR(saturn.queue.manager)

14. Run the IVTRun tool again as mentioned in step 11. This time messages will be sent to the saturn queue manager.

Now we are all set. It's time to get WMQ integrated with Synapse and WSO2 ESB. Let's start with Apache Synapse.

Synapse Integration

Follow the steps given below as the user 'mqm'. Trying to run Synapse as a different user caused some JMS security exceptions.

1. Copy the following jar files from /opt/mqm/java/lib to SYNAPSE_HOME/lib

• com.ibm.mqjms.jar
• fscontext.jar
• providerutil.jar
• com.ibm.mq.jmqi.jar
• dhbcore.jar

2. Enable the JMS listener in the axis2.xml. Configure the default connection factory as follows.

<parameter name="default">
  <parameter name="java.naming.factory.initial">com.sun.jndi.fscontext.RefFSContextFactory</parameter>
  <parameter name="java.naming.provider.url">file:/var/mqm/jndi</parameter>
  <parameter name="transport.jms.ConnectionFactoryJNDIName">ivtQCF</parameter>
  <parameter name="transport.jms.ConnectionFactoryType" locked="false">queue</parameter>
  <parameter name="transport.jms.Destination">ivtQ</parameter>
</parameter>

Note that we are using the JNDI configuration used by the IVT sample program. We are using the same queue connection factory and the JMS queue to define the default connection factory.

3. Create the following proxy service in the synapse.xml (this is a simplified version of sample 250)

<proxy name="JMSProxy" transports="jms">
  <target>
     <inSequence>                             
        <log level="full"/>
        <drop/>
     </inSequence>
  </target>
  <parameter name="transport.jms.ContentType">
       <rules>
           <jmsProperty>contentType</jmsProperty>
           <default>application/xml</default>
       </rules>
  </parameter>
</proxy>

4. Start Synapse - go to SYNAPSE_HOME/bin and run ./synapse.sh

5. Go to SYNAPSE_HOME/samples/axis2Client/src/samples/userguide and open the GenericJMSClient.java source file. Make the following changes in the code.

• Set the jms_dest property default value to "ivtQ" (line 45)
• Set the java.naming.provider.url to "file:/var/mqm/jndi" (line 82)
• Set the java.naming.factory.initial to "com.sun.jndi.fscontext.RefFSContextFactory" (line 85)
• Set the lookup key to "ivtQCF" (line 89)

6. Now from the SYNAPSE_HOME/samples/axis2Client directory run the following command to run the JMS client application which will send messages to the ivtQ.

ant jmsclient -Djms_type=pox -Djms_payload=IBM

7. At this point Synapse will pick the messages from the queue and log them on the console.

WSO2 ESB 2.1 Integration

Follow the steps given below as user 'mqm'

1. Copy the Websphere MQ client jars (mentioned above under Synapse Integration) to ESB_HOME/repository/components/lib

2. Enable the JMS listener in axis2.xml and configure the default JMS connection factory as follows

<parameter name="default">              
        <parameter name="java.naming.factory.initial" >com.sun.jndi.fscontext.RefFSContextFactory</parameter>                      
        <parameter name="java.naming.provider.url" >file:/var/mqm/jndi</parameter>
        <parameter name="transport.jms.ConnectionFactoryJNDIName" >ivtQCF</parameter>
        <parameter name="transport.jms.ConnectionFactoryType" >queue</parameter>
        <parameter name="transport.jms.Destination">BOGUSQ</parameter>
</parameter>

Note that we have specified a queue named 'BOGUSQ' as the default destination. This is because we need to use ivtQ for our proxy service only. If we use 'ivtQ' here all the services deployed in ESB (XKMS, echo, wso2carbon-sts) will start listening on the same queue. You also need to login to JMSAdmin and create the queue named 'BOGUSQ'. In JMSAdmin shell run the following commands.

DEFINE Q(BOGUSQ) QMGR(saturn.queue.manager)

ALTER Q(BOGUSQ) QUEUE(QUEUE1)

If the QUEUE1 does not exist, first login to MQSC and create it.

3. Add the proxy service we used with Synapse to ESB_HOME/conf/synapse.xml

4. Start WSO2 ESB

./wso2server.sh -DuseSynapseXML

5. Use the sample client we used in Synapse to send messages to the queue

Amplify Your SOA with WSO2 ESB 2.1

If you have been following my blog, then you already know that WSO2 Carbon 2.0 and a host of other Carbon based WSO2 SOA products were released last week. WSO2 Enterprise Service Bus 2.1, which is one of the released products, must be highlighted as a high quality, feature rich and extremely user friendly piece of SOA middleware for a number of reasons. Like all its predecessors, this version of WSO2 ESB is also based on Apache Synapse, the lightweight, ultra-fast ESB. WSO2 ESB is generally popular among SOA enthusiasts because of the following set of key features provided by the ESB.

• Proxy services - facilitating synchronous/asynchronous transport, interface (WSDL/Schema/Policy), message format (SOAP 1.1/1.2, POX/REST, Text, Binary), QoS (WS-Addressing/WS-Security/WS-RM) and optimization switching (MTOM/SwA).
• Non-blocking HTTP/S transports based on Apache HttpCore for ultrafast execution and support for thousands of connections at high concurreny with constant memory usage.
• Built in Registry/Repository, facilitating dynamic updating and reloading of the configuration and associated resources (e.g. XSLTs, XSD, WSDL, Policies, JS, Configurations ..)
• Easily extendable via custom Java class (mediator and command)/Spring mediators, or BSF Scripting languages (Javascript, Ruby, Groovy, etc.)
• Built in support for scheduling tasks using the Quartz scheduler.
• Load-balancing (with or without sticky sessions)/Fail-over, and clustered Throttling and Caching support
• WS-Security, WS-Reliable Messaging, Caching & Throttling configurable via (message/operation/service level) WS-Policies
• Lightweight, XML and Web services centric messaging model
• Support for industrial standards (Hessian binary web service protocol/ Financial Information eXchange protocol and optional Health Level-7 protocol)
• Enhanced support for the VFS(File/FTP/SFTP)/JMS/Mail transports with optional TCP/UDP transports and transport switching for any of the above transports
• Support for message splitting & aggregation using the EIP and service callouts
• Database lookup & store support with DBMediators with reusable database connection pools
• WS-Eventing support with event sources and event brokering
• Rule based mediation of the messages using the Drools rule engine
• Transactions support via the JMS transport and Transaction mediator for database mediators
• Internationalized GUI management console with user/permission management for configuration development and monitoring support with statistics, configurable logging and tracing
• JMX monitoring support and JMX management capabilities like, Gracefull/Forcefull shutdown/restart

Wow! That's a lot of features for a software product developed openly and distributed free of charge under Apache Software License 2.0. As you would imagine, it is mainly the performance and the lightweight operation model of WSO2 ESB which makes it stand out from the rest. In addition to the above mentioned key features the latest ESB 2.1 release brings you the following set of new features.

• Rule based mediation via Drools
• Fine grained authorization for services via the Entitlement mediator
• Reliable-Messaging specification 1.1 support
• Enhanced WS-Eventing support and Event Sources making it an even broker
• Enhanced AJAX based sequence, endpoint and proxy service editors
• Enhanced transport configuration management through the graphical console
• Enhanced integrated registry and search functionalities with versioning, notifications, rating of resources, and commenting
• Enhanced remote registry support
• Default persistence to the registry for the configuration elements
• Enhanced permission model with the user management
• Enhanced REST/GET and other HTTP method support
• P2 based OSGi feature support, for optional features like service management, runtime governance

The coolest thing about WSO2 ESB 2.1 is that it is 100% OSGi based (thanks to the Carbon platform of course!). All the features are packed into OSGi bundles and therefore adding new features and removing unnecessary features cannot get any easier. The newly introduced provisioning support based on Equinox P2 makes it particularly easy to deploy new features and third party libraries into the ESB. With WSO2 ESB 2.1, you can deploy only the features you want and only them. You are not forced to load any features/libraries that you never use. Why waste memory and other resources on features never used, right?

With ESB 2.1 registry integration support has improved vastly. WSO2 ESB 2.1 comes with an embedded WSO2 G-Reg instance but you can easily point the ESB to a remotely hosted registry instance in a matter of seconds. ESB 2.1 also has the ability to export its entire configuration to the registry and reload the configuration back from the registry at the server startup.

User interfaces and context sensitive help system have gone through lot of rework. You will find that most of the web interfaces are now fully AJAX compliant making it easy and fun to work with WSO2 ESB. All UIs are fully internationalized and can be even separated from the backend system to be hosted as different web application.

WSO2 ESB 2.1 is a giant step forward by the WSO2 folks to make their ESB even more elegant and enterprise ready. It gives you a combination of high performance, modularity and user friendliness. It is completely free and open source, with a very active and supportive community of developers to back up all the development work. WSO2 also offers user training, development support and production support to any party which requires such facilities.

If you are looking for a robust mediation solution to power your enterprise SOA or if you are tired of trying out expensive proprietary ESB solutions, it’s high time you give WSO2 ESB 2.1 a spin. It will be totally worth it!!!

A 'Good' ESB Should....

As technologies like SOA and Web Services continue to become more and more dominant mechanisms for implementing complex distributed systems, the demand for efficient and reliable enterprise service bus middleware is becoming larger and larger. There are dozens of potential ESB solutions out there, some open source and some proprietary, but the cold hearted truth is most of these products have a number of short comings, which makes them totally useless in production environments. Production environments don't require fast ESBs. They need ultra fast ESBs that can handle thousands of concurrent user requests. such deployments require middleware which can satisfy scalability and availability requirements through features such as load balancing, clustering and fail over support. A good ESB should also be capable of dealing with many communication protocols, transport mechanisms and messaging standards.

Fortunately we are not totally out of hope. There are some really good products out there which give you all the above mentioned features and deliver 100% in production deployments. WSO2 ESB is certainly one of those middleware solutions which is fast, reliable and feature rich. It is based on Apache's tried and tested Apache Synapse light weight ESB and starting from version 2.0 it is also based on WSO2's revolutionary Carbon framework.

WSO2 folks recently published a cool flash presentation which walks us through all the basic features of WSO2 ESB within a few minutes. Have a look and see whether your ESB delivers at least half of the things the WSO2 ESB provides.

WSO2 ESB 2.0.2 Released...Come and Get it

The WSO2 ESB team is pleased to announce the release of version 2.0.2 of the Open Source Enterprise Service Bus (ESB). This is a bug fix release of the 2.0 release of WSO2 ESB

WSO2 ESB is a lightweight and easy-to-use Open Source Enterprise Service Bus (ESB) available under the Apache Software License v2.0. WSO2 ESB allows administrators to simply and easily configure message routing, intermediation, transformation, logging, task scheduling, etc.. The runtime has been designed to be completely asynchronous, non-blocking and streaming based on the Apache Synapse core.

WSO2 ESB 2.0.2 is developed on top of the revolutionary Carbon platform (Middleware a' la carte), and is based on the OSGi framework to achieve the better modularity for your SOA architecture. This also contains a lots of new features and many other optional components to customize the behavior of the server. Further, if you do not want any of the built in features, you can uninstall those features without any trouble. In other words, this ESB can be customized to your SOA needs.

You can download this distribution from http://wso2.org/downloads/esb and give it a try.

How to Run

1. Extract the downloaded zip
2. Go to the bin directory in the extracted folder
3. Run the wso2server.sh or wso2server.bat as appropriate
4. Point your browser to the URL https://localhost:9443/carbon
5. Use "admin", "admin" as the username and password to login as an admin and create a user account
6. Assign the required permissions to the user through a role
7. If you need to start the OSGi console with the server use the property -DosgiConsole when starting the server
8. Samples configurations can be specified by passing the property -Desb.sample=${SAMPLE_NUMBER}

Key Features

• Proxy services - facilitating transport, interface (WSDL/Schema/Policy), message format (SOAP 1.1/1.2, POX/REST, Text, Binary), QoS (WS-Addressing/WS-Security/WS-RM) and optimization switching (MTOM/SwA).
• Non-blocking HTTP/S transports based on Apache HttpCore for ultrafast execution and support for thousands of connections at high concurreny with constant memory usage.
• Built in Registry/Repository, facilitating dynamic updating and reloading of the configuration and associated resources (e.g. XSLTs, XSD, JS, ..)
• Easily extended via custom Java class (mediator and command)/Spring mediators, or BSF Scripting languages (Javascript, Ruby, Groovy, etc.)
• Built in support for scheduling tasks using the Quartz scheduler.
• Load-balancing (with or without sticky sessions) /Fail-over, and clustered Throttling and Caching support
• WS-Security, WS-Reliable Messaging, Caching and Throttling configurable via (message/operation/service level) WS-Policies
• Lightweight, XML and Web services centric messaging model
• Support for industrial standards (Hessian binary web service protocol/Financial information exchange protocol)
• Enhanced support for the VFS/JMS/Mail transports
• Support for message splitting and aggregation using the EIP
• Database lookup and store support with DBMediators with reusable database connection pools
• JMX monitoring support

New Features of the WSO2 ESB 2.0 (2.0.2)

• This ESB release is based on Carbon "Middleware a' la carte" which is an OSGi based SOA platform by WSO2 Inc.
• Transactional JMS transport and the Transaction mediators
• Integrated graphical user management
• Integrated graphical key store management
• Configurable logging through the management console
• Graphical data source declaration
• WS-Eventing support and Event Sources making it an event broker
• Enhanced sequence and proxy service editor
• Module management capability
• Transport configuration management through the graphical console
• Graceful/Forced shutdown/restart
• Enhanced integrated registry and search functionalities
• User permissions support
• Enhanced monitoring tools for statistics and tracing
• Try-It tool to try an existing service
• Graphical policy editor
• Administration console fully internationalized
• Better modularity and extendability through OSGi component architecture

How You Can Contribute....

Mailing Lists

Join our mailing list and correspond with the developers directly.

• Developer List : esb-java-dev@wso2.org
• User List : esb-java-user@wso2.org

Reporting Issues

WSO2 encourages you to report issues and your enhancement requests for the WSO2 ESB using the public JIRA.

You can also watch how they are resolved, and comment on the progress..

Discussion Forums

Alternatively, questions could be raised using the forums available.

WSO2 ESB Forum : Discussion forum for WSO2 ESB developers/users

Training

WSO2 Inc. offers a variety of professional Training Programs, including training on general Web services as well as WSO2 ESB, Apache Synapse, Apache Axis2 and number of other products.

For additional support information please refer to http://wso2.com/training/course-catalog/

Support

WSO2 Inc. offers a variety of development and production support programs, ranging from Web-based support up through normal business hours, to premium 24x7 phone support.

For additional support information please refer to http://wso2.com/support/

For more information on WSO2 ESB, visit the WSO2 Oxygen Tank (http://wso2.org)

We welcome your feedback on this implementation. Thank you for your interest in WSO2 ESB.

-- The WSO2 ESB Team --

Get Ready to Get MOINCed

Few weeks back me and my team managed to finalize developing the first prototype of our dream project, MOINC. If you are clueless as to what project MOINC is all about, it is an attempt to combine the Web Services paradigm with grid computing. The goal of the project is to be able to deploy enterprise Web Services for high availability and high scalability using commodity hardware. So far we have done three formal presentations describing each of the major components of the MOINC platform and we have written three research papers which we hope to publish very soon. In addition to the presentations we also conducted a formal demonstration of MOINC at University of Moratuwa, using 6 computers which effectively gave a preview of MOINC in action to a panel headed by our project supervisor Dr. Sanjiva Weerawarana and our senior lecturer Dr. Chathura De Silva. Presentations were attended by Dr. Sanath Jayasena, Dr. Chandana Gamage, Dr. Chathura De Silva and Mr. Shantha Fernando of the department of CSE, University of Moratuwa.

We got pretty good feedback from all those who attended the presentations and the demonstration, clearly indicating that we are on the right track to make MOINC into a useful software solution for business organizations worldwide. Some of the core features of the MOINC platform that we demonstrated are listed below.

1. Deploy, manage and undeploy service artifacts (Service artifacts are uploaded as Axis archives - *.aar files)
   
2. Track down idling computers in the local network and add them to the MOINC grid dynamically
   
3. Download service artifacts into idling computers from a centralized registry/repository
   
4. Action script based screen saver for client PCs
   
5. Load balance the incoming service requests among all the active nodes connected to the MOINC grid (Powered by Apache Synapse)
   
6. Monitor the grid via an AJAX based Web interface (Powered by WSO2 WSF/AJAX)
   
7. Collect statistics related to computers connected to the MOINC grid and use them in the intelligent load balance mode
   
8. MOINC community portal and forums (backed by WSO2 Registry and JForum)
   

That’s certainly a lot of features for a mere prototype. No wonder we got pretty good feedback. However there is certainly lot more work to be done. We need to improve the overall security of the platform. Currently there is a couple of security loop holes in MOINC SMM that we need to close off. Also Dr. Sanjiva suggested writing a Java security manager for MOINC client agent which ensures the security of PCs connected to the MOINC server. We need to start working on that soon as well. Also we have to finish our Maven2 integration stuff and get the release artifacts for MOINC 0.1-alpha out soon. We still haven’t cut the release artifacts for all the components of MOINC but the full source code of the prototype can be checked out from our SVN. Also checkout the developer resources section on our website for all the slides we used for the above mentioned three presentations.

I will also make arrangements for all of you humble readers to have a sneak peak at our research papers before we actually publish them. I promise. Meanwhile enjoy the presentations and other design documents :)

FIX Support in WSO2 ESB

WSO2 ESB is an ultra fast, lightweight, open source ESB based on Apache Synapse. The FIX transport implementation of Apache Synapse is fully operational in WSO2 ESB 1.7. Asanka Abeysinghe from WSO2 has recently published a comprehensive article on using the FIX transport in WSO2 ESB. He starts by giving a short introduction to the FIX protocol and goes onto explaining a number of interesting usecases for the FIX transport in WSO2 ESB.

If you are looking for a powerful SOA based solution to deal with your FIX transactions, this is a must read for you. While you are at it don't forget to take a peek at the article titled 'Apache Synapse FIX'ed', and article I wrote few months back introducing the FIX transport implementation of Apache Synapse.

Fixing the FIX Transport

The 1.2 version of Apache Synapse, the open source lightweight ESB was released on June 2008. One of the most striking features of this release was the FIX transport implementation, mainly due to the fact that most ESBs in the world still do not support the FIX protocol. The Synapse FIX transport implementation which is only two months old at the moment has already made a lot of hype in the world of Software Engineering and SOA. It seems we already got a few interested people (potential clients?) who are at the moment testing and playing with the transport.

Synapse 1.2 ships with two samples that demonstrate the FIX transport module. One of them demonstrates how two FIX endpoints can be bridged using Apache Synapse. The other samples shows how to bridge an HTTP client with a FIX endpoint. Over the last couple of months the Synapse community has worked really hard to further improve the FIX transport implementation and also to identify more exciting usecases for the transport module. As a result we managed to add four new samples demonstrating the FIX transport to the Synapse documentation. These samples are currently available in the Apache Synapse Snapshot and will be most likely included in the next release.

The first of the four newly added samples (sample 259) shows how to bridge a FIX client with an HTTP endpoint. The sample effectively demonstrates how Banzai, the sample FIX blotter that ships with Quickfix/J can be used to send order requests via Synapse to a service listening on HTTP. The sample uses the XSLT mediator to convert the FIX messages into a SOAP messages.

The second of the new samples (sample 260) shows how Synapse can be used to bridge a FIX endpoint with an AMQP endpoint. Here once the FIX message is converted into XML it will be bound to a JMS payload and sent to an AMQP consumer. Since AMQP is used widely in business applications, I believe that this sample will open up the door way to a ton of really cool usecases.

The third sample (sample 261) demonstrates how Synapse can be used to switch between FIX sessions with different versions (BeginString values). The sample successfully bridges a FIX 4.0 session with a FIX 4.1 session but the underlying concepts can be used to bridge virtually any two FIX sessions. One thing worth mentioning here is that the FIX transport implementation of Synapse initially did not support bridging FIX sessions of different versions. But considering the potential usecases, we implemented that feature very recently.

The fourth newly added sample (sample 262) demonstrates how CBR (Content Based Routing) can be done with FIX messages using Apache Synapse. The sample configuration causes Synapse to accept FIX messages over a session, read a certain symbol in the messages and based on the symbol value route the messages to different endpoints.

We also added namespace support to the FIX transport module so that it can properly parse and validate XML based FIX messages with namespaces. Another recent feature addition was improving the way FIX sessions are initialized in Synapse. The initial implementation lazy initializes all the FIX sessions for outgoing FIX messages. That means an outgoing session will not be created until a message arrives for that particular session. However since this leads to fairly large delays we improved the transport module so that the outgoing sessions are also initialized at the startup along with the incoming sessions. (the old way of initializing sessions is also supported)

Currently we are working on adding support for FIX repeated groups. (we already have a feature request for this on the JIRA from one of our users) All in all it seems that the FIX transport module for Synapse is increasingly becoming a very powerful and matured piece of software in a blistering rate.

WSO2 ESB 1.7 Released

The WSO2 Enterprise Service Bus (ESB) team is pleased to announce the release of its version 1.7 of the Open Source ESB.

The WSO2 ESB is an ultra fast, light-weight and versatile Enterprise Service Bus based on the Apache Synapse ESB. It allows you to Connect, Manage and Transform service interactions between Web services, REST/POX services and Legacy systems. You can easily switch transports between http/s, JMS, File Systems, Mail, FIX etc, or read/write from Databases, split, aggregate or clone messages and support declarative enforcement of QoS aspects such as WS-Security, WS-Reliable Messaging etc, and also switch between message formats such as SOAP 1.1/1.2, PoX/REST, Hessian, Text, Binary, MTOM and SwA.

The WSO2 ESB is released under the Apache Software License v2.0, and ships with a graphical management and administration console and enhanced JMX management/monitoring support, and integrates seamlessly with the WSO2 Registry.

Webinar series introducing the WSO2 ESB v1.7:
In this Webinar series Paul Fremantle, CTO of WSO2, will introduce the new features and capabilities of the WSO2 ESB. The first session will recap on the overall approach and benefits of the WSO2 ESB solution and the underlying Apache Synapse project, and then go into the added functionality and benefits of the 1.7 release. The series will include details of the newly released support for Hessian, FIX, AMQP and also discuss the improvements in performance and stability.

* For more details on the Webinar series, and to register,
visit http://wso2.com/about/news/esb-webinar-june-17/


Core features of the WSO2 ESB includes:
* Proxy services / Service mediation and Message mediation
* Support for Non-blocking http/s, JMS, FIX, Apache VFS (s/ftp, file,
zip/tar/gz, webdav, cifs..), POP3/IMAP/SMTP, AMQP transports
* Support for SOAP 1.1/1.2, PoX/REST, Hessian, Text and Binary payloads
* Support for scheduled task execution and management
* Support for custom extensions in Java through custom mediators, POJO
Classes and Java Command classes
* Support for Apache BSF Scripting languages such as (Javascript, Ruby,
Groovy..etc)
* Support for clustered deployment with pinned services and tasks
* Throttling, Caching, Load balancing and Failover support
* Support for declarative WS-Reliable Messaging, WS-Security and
WS-Policy attachment
* Integrated WSO2 Registry with support for external Registries
* Ability to stop, re-start and gracefully shutdown the ESB through JMX
* Cluster aware sticky load balancing support

New features of the v.1.7 release includes:
* Support for Hessian binary messages
* FIX (Financial Information eXchange) protocol transport
* WS-Reliable Messaging support with WSO2 Mercury
* Ability to stop, re-start and gracefully shutdown the ESB through JMX
* Integrated WSO2 Registry shipped, with ability to connect to a remote
WSO2 Registry
* Support for re-usable database connection pools for DB report/lookup
mediators
* Support for GZip encoding and HTTP 100 continue
* Natural support for dual channel messaging with WS-Addressing
* Cluster aware sticky load balancing support
* Non-blocking streaming of large messages at high concurreny with
constant memory usage
* Support for an ELSE clause for the Filter mediator
* Ability to specify XPath expressions relative to the envelope or body
* Support for separate policies for incoming/outgoing messages
* Support for a mandatory sequence before mediation
* New Router mediator
* Ability to re-deploy proxy services

Useful Links
Download WSO2 ESB - http://wso2.org/downloads/esb/
Quickstart Guide
Installation Guide
Administration Guide
Samples Guide
Documentation Index

Contribute to WSO2 ESB
SVN: http://wso2.org/repos/wso2/trunk/esb/java/
JIRA: http://wso2.org/jira/browse/ESBJAVA
User list: esb-java-user@wso2.org
Developer list: esb-java-dev@wso2.org
Web Forum: http://wso2.org/forum/187

Training
WSO2 Inc. offers a variety of professional Training Programs, including training on general Web services as well as WSO2 ESB, Apache Synapse and Axis2, Data Services and a number of other products. For additional support information please refer to http://wso2.com/training/course-catalog/

Support
WSO2 Inc. offers a variety of development and production support programs, ranging from Web-based support up through normal business hours, to premium 24x7 phone support. For additional support information please refer to http://wso2.com/support/ For more information on WSO2 ESB visit the WSO2 Oxygen Tank (http://wso2.org)

Apache Synapse 1.2 Released

The Apache Synapse team is pleased to announce the release of version 1.2 of the Open Source Enterprise Service Bus (ESB).

Apache Synapse is an lightweight and easy-to-use Open Source Enterprise Service Bus (ESB) available under the Apache Software License v2.0. Apache Synapse allows administrators to simply and easily configure message routing, intermediation, transformation and logging task scheduling, etc.. The runtime has been designed to be completely asynchronous, non-blocking and streaming.

The Apache Synapse project and the 1.2 release can be found here:
http://synapse.apache.org

Apache Synapse offers connectivity and integration with a range of legacy systems, XML-based services and SOAP Web Services. It supports non-blocking HTTP and HTTPS using the Apache HTTPCore (http://hc.apache.org) components, as well as supporting JMS (v1.0 and higher) and a range of file systems and FTP sources including SFTP, FTP, File, ZIP/JAR/TAR/GZ via the Apache VFS project (http://commons.apache.org/vfs/filesystems.html).

At the same time Synapse 1.2 release adds the support for the Financial Information eXchange (FIX) an industry driven messaging standard through QuickFixJ, Hessian binary web service protocol, as well as other functional, stability and performance improvements. Synapse supports transformation and routing between protocols without any coding via configurable virtual services.

Synapse provides first class support for standards such as WS-Addressing, Web Services Security (WSS), Web Services Reliable Messaging (WSRM), Throttling and caching, configurable via WS-Policy upto message level, as well as efficient binary attachments (MTOM/XOP).

The 1.2 release contains a set of enhancements based on feedback from the user community, including:

* Support for Hessian binary web service protocol
* FIX (Financial Information eXchange) protocol for messaging
* WS-Reliable Messaging support with WSO2 Mercury
* Support for re-usable database connection pools for DB report/lookup mediators
* Support for GZip encoding and HTTP 100 continue
* Natural support for dual channel messaging with WS-Addressing
* Cluster aware sticky load balancing support
* Non-blocking streaming of large messages at high concurreny with constant memory usage
* Support for an ELSE clause for the Filter mediator
* Ability to specify XPath expressions relative to the envelope or body
* Support for separate policies for incoming/outgoing messages
* Support for a mandatory sequence before mediation

The combination of XML streaming and asynchronous support for HTTP and HTTPS using Java NIO ensures that Synapse has very high scalability under load. Performance tests show that Synapse can scale to support thousands of concurrent connections with constant memory on standard server hardware.

Apache Synapse ships with over 50 samples (http://synapse.apache.org/Synapse_Samples.html) designed to demonstrate common integration patterns "out-of-the-box", along with supporting sample services, and service clients that demonstrate these scenarios. Apache Synapse is configured using a straightforward XML configuration syntax
(http://synapse.apache.org/Synapse_Configuration_Language.html).

Apache Synapse is openly developed by a community that welcomes all forms of input, ranging from suggestions and bug reports to patches and code contributions. Your comments and feedback on the project and release are welcome.

The Apache Synapse code and binaries are available from the website at http://synapse.apache.org

FIXing Synapse!!!

Recently I got the opportunity to get involved in developing a new transport module for Apache Synapse to support the FIX protocol. Apache Synapse which is a light weight mediation framework for Web services, has had support for a number of application layer protocols like HTTP/S, SMTP, JMS and VFS. The development of the FIX transport module took nearly one and a half month and now this module is available in the Apache Synapse SVN trunk along with couple of samples and some documentation.

FIX protocol or the Financial Information eXchange is a messaging standard developed specifically to facilitate securities transactions. Strangely enough this protocol which is being used by hundreds of banks, stock exchanges and broker dealers all around the world is still not very popular in the Web services world (See here for a list of FIX users). The protocol has been in existence since 1992 and there are six major versions of the specification at the moment of writing (4.0, 4.1, 4.2, 4.3, 4.4, 5.0). The specs are owned by the FIX Protocol Limited (FPL) but it is essentially a free and open standard.

FIX specifications focus on two layers of the OSI reference model, namely the application layer and the session layer. Any application that wishes to communicate with another application using the FIX protocol must first establish a FIX session. A FIX session can exist among only two parties where one party is the acceptor and the other party is the initiator. Initiator is the one who starts the conversation by sending out the initial login request.

FIX messages are essentially a series of key-value pairs where each key-value pair is known as a field. Fields are separated by using the ASCII Start of Header (0001) character as the delimiter. The key of a field is simply a positive integer. But these integers have meanings and they are defined in the specifications. A typical FIX message might appear to be as follows.

8=FIX.4.09=10235=D34=1649=BANZAI
52=20080314-05:01:4756=SYNAPSE11=1205470907396
21=138=540=154=155=IBM59=0
10=078

A FIX message can be logically separated into a header, body and a trailer. The fields that should appear in each of these portions are clearly specified in the FIX specs. For an example the BeginString (8) field is a header field. The Checksum (10) is a trailer field. The content of a FIX message can vary greatly depending on the type of the message.

We used an open source FIX engine known as Quickfix/J to develop the FIX transport module for Apache Synapse. It currently supports five out of the six major versions of the FIX specification. Quickfix/J provides a very simple API to develop FIX based applications and applications developed on Quickfix/J are highly configurable. In addition to that Quickfix/J offers powerful message parsing, validation and logging.

Quickfix/J project is driven by a very active development team and a very enthusiastic user base. Quickfix/J uses Apache MINA and hence is based on Java NIO asynchronous network communications system.

All the transport modules of Apache Synapse are developed using the Apache Axis2 transport framework. Any transport module developed on the Axis2 transport framework must have two main elements, namely the transport listener and the transport receiver. The Axis2 transport framework provides the necessary interfaces and the base classes to implement these elements. The transport listener implementation is basically responsible for accepting in bound messages. For each accepted incoming message the transport listener should create an Axis2 message context, populate the message context accordingly and hand it over to the Axis2 kernel for further processing. The transport sender implementation is used by Axis2 kernel to send out messages. This implementation should be capable of processing Axis2 message contexts and converting the SOAP messages embedded in message contexts into messages that can be sent over the wire. Also depending on the nature of the protocol transport sender may also have to handle incoming response messages.

The implementations of the transport listener and the transport sender for the FIX transport module are named FIXTransportListener and FIXTransportSender respectively. The FIXTransportListener makes use of a FIXSessionFactory class which takes care of creating, storing and managing FIX sessions. The class FIXIncomingMessageHandler is where Apache Synapse binds with Quickfix/J. This class implements the quickfix.Application interface. For each accepted FIX message the transport module forks off a new thread from the thread pool associated with the transport listener implementation. This thread then converts the FIX message into XML using the Apache AXIOM API.

The FIX message converted into XML is then placed in a SOAP envelope. When converting FIX messages into XML, CDATA tags are used, basically as a precaution because theoretically FIX messages can have any kind of data in the fields. Fields can even contain XML or binary data. The transport do not change any of the field values while converting the FIX messages into XML. However if binary data is found in the message then necessary action will be taken to put the binary data in an Axis2 message context as a binary attachment. Finally the SOAP envelopes holding the FIX messages will be handed over to Axis2 kernel for further processing.

Another possible approach we could have taken here was to embed the FIX messages in SOAP envelopes without converting them into XML. But converting the FIX messages into XML has many advantages. Once converted into XML the Synapse user have more control over the FIX message content. Technologies like XPath and XQuery can be used to manipulate the FIX message content within the Synapse core.

One major problem we faced during the development of the new transport module was implementing in order message delivery. In order message delivery and processing is a characteristic feature of the FIX protocol. But since Synapse uses a separate thread to handle each incoming message we could notice that messages are not sent out in the order they were received. This is due to the thread switching that takes place while Synapse is performing the mediation. As a solution to this issue we introduced an application level sequence numbering mechanism to the transport module. Each and every incoming FIX message will be given a sequence number by the transport listener. A sequence number is unique for a given FIX session. A string that uniquely identifies the session is also associated with the messages. These information are specified in the SOAP envelope it self as attributes of the message element. The FIX transport sender implementation looks at these values and sends the messages in the exact order they were received. However having these attributes in the SOAP envelope is optional. They are required only if the user wants the FIX transport sender to send out messages in the order they were received.

While developing the FIX transport module we tried our best to provide all the options and choices Quickfix/J normally provides to the users. Apart from the logging features provided by Apache Synapse, users can enable logging at transport level (before messages are converted into XML) using Quickfix/J. Quickfix/J offers a number of options when it comes to message logging. You can either log messages on to the console, into a file or into a database. Same thing can be done with Synapse as well. Also all the message store implementations that come with Quickfix/J are available with Synapse too. By default Synapse will try to use the memory based message store implementation with acceptors and initiators. But the users can use other implementations (file, jdbc etc) if they want.

All in all developing the FIX transport module for Apache Synapse turned out to be a huge success and it indeed was a great learning opportunity for me. Not much ESBs in the world currently support the FIX protocol so it really improved the value and marketability of Apache Synapse. This module is still somewhat in its child state so we expect the contributions from the developers and FIX experts out there to improve it.