Sunday, November 13, 2011

ESB: The Swiss Army Knife of SOA

My talk at WSO2Con 2011 is now available on YouTube. There I discussed why an ESB is essential for SOA, what kind of problems it can solve and what it is not.

Saturday, October 22, 2011

JMS Proxy Services with Fail-Over Support

Rajika has written a very comprehensive article about setting up WSO2 ESB with JMS in a fail-over scenario.
The concept of fail-over endpoints is integral to WSO2 ESB. The user can define a fail-over endpoint in the ESB which in turns consists of two or more child endpoints. ESB will select one of the child endpoints to be the primary endpoint and will keep forwarding all messages to this primary endpoint. If the ESB detects some error while communicating with the primary endpoint, it will automatically fail-over to one of the available backup endpoints. User can also configure the endpoint error codes upon which to trigger a fail-over, endpoint suspend durations and timeout durations. However this is not what Rajika is talking about in his article. His article is about setting up a JMS proxy service with fail-over support in the ESB .
For an example imagine a situation where you have setup a proxy service to receive messages from a JMS queue. The ESB proxy service polls messages out of the queue and forwards them to a backend application for further processing. Now what if the ESB instance which hosts the above mentioned proxy service happens to fail (say due to a hardware failure). Actually in this case there's both good news and bad news. Good news is that no messages will get lost due to the persistent behavior of JMS messaging. If some messages were in flight within the ESB when the failure occurred those messages might get lost. But even that can be avoided by using something like JTA. The bad news however is that the backend application will not receive further requests to be processed. So as far as your overall system is concerned, your applications and services will stay idle while messages keep piling up on the JMS queue.
Rajika discusses a possible architecture to avoid such system downtime by having a backup proxy service (ideally deployed in a separate ESB instance) which listens on the same JMS queue. The backup proxy will be disabled by default to prevent contention between the two proxy services. His architecture also utilizes a scheduled task deployed on the backup ESB which sends periodic heartbeats to the other ESB node. If the backup ESB detects that the primary ESB instance is not responding, it will activate the backup proxy service which will start polling for JMS messages and resume operations in the system.
This is actually fairly simple to implement in WSO2 ESB and can be even further customized to suit the exact enterprise requirements. All the source code and relevant configurations are provided in the article itself.

Tuesday, October 18, 2011

Rediscover SOA with WSO2 Carbon and WS-Discovery

WS-Discovery is a standard protocol for discovering services and endpoints in an SOA based environment. Using WS-Discovery, servers can publish service metadata to a centralized repository known as the discovery proxy. Clients and other interested parties can query the discovery proxy to find a required service or an endpoint.
WSO2 Carbon platform has exceptional support for WS-Discovery. Service hosting products like WSO2 Application Server can publish their services to a user defined discovery proxy. WSO2 Governance Registry can be used as this discovery proxy and products like WSO2 ESB can query a discovery proxy to locate backend service endpoints.
I recently wrote an article explaining how WS-Discovery is supported in the Carbon platform. The article starts by explaining the basics of WS-Discovery and then describes how to build a complete practical solution based on the discovery protocol. The sample solution described in the article, an inventory management system for Trader Paul's department store, uses 4 WSO2 products. Almost all the core business logic of the solution resides in WSO2 Application Server and WSO2 Data Services Server. The web application which provides the UI layer of the solution uses WS-Discovery to dynamically discover the business services and data services. WSO2 Governance Registry acts as the discovery proxy where all the service metadata are stored. The solution also uses WSO2 ESB to expose some of the core services to the public through a secured channel. The article describes how to use the discovery support in ESB to create these proxy services. Further it describes an advanced functionality of the ESB (discovery mediation extensions), using which 100% dynamic proxy services can be created. These dynamic proxy services use the WS-Discovery protocol to discover the backend service endpoints at runtime.
This article shows the real power of the Carbon framework and what we can achieve with it when the appropriate components/products are combined into a single distributed solution. I certainly enjoyed a lot putting up all the samples and writing content for this article. I hope you will get a similar feeling reading it. All the code and configurations used for the samples are also provided so you can run the Trader Paul's Inventory Management System on your own and try it out. I'm currently working on porting this sample to the StratosLive environment. So it will be soon available on the cloud as well. If you are just looking to learn about WS-Discovery, or just trying to understand how to build solutions with WSO2 products, this article would be a great starting point.

Sunday, October 16, 2011

WSO2 ESB Documentation Revamped

No software application is complete without documentation. According to Ian Sommerville (author of the well known textbook on Software Engineering), software is not just the computer programs, but also all associated documentation and configuration data that is needed to make the programs operate correctly. According to this definition documentation is not only a key component of software, but it is one of the factors which differentiates software from computer programs. But unfortunately many open source projects are getting hammered for their poorly structured and less comprehensive documentation. WSO2 ESB documentation also used to suffer from a number of such strictures which made it look slightly obfuscated and disorganized to the reader. While the samples and other configuration related documentation was fairly well organized, the overall documentation offering had several gaps in terms of content, user friendliness and ability to search.
However I'm pleased to announce that those days are now behind us. WSO2 ESB now has a brand new documentation package with a whole lot of new content, an exquisite look and feel and a ton of new features. You will find that almost all the old content has been rewritten, reformatted and reshaped to make it more comprehensive and easier to follow. In contrast to the old documentation, the new offering contains many diagrams, sample configurations and much easier navigation capabilities between different sections. Samples are no longer a separate section in the documentation but they are baked into the relative explanatory sections in the documentation. This way the reader can seamlessly move between content and relevant samples.
Perhaps the most significant feature in the new documentation package is its ability to search for content. All the content items have been properly labeled and indexed so the reader can find out what he's looking for by doing a search on the documentation site. Try out following search links to see for yourself.
As of now you will find documentation related to WSO2 ESB 3.0.1 and WSO2 ESB 4.0.0 in the new documentation site. Please note that this is still very much an ongoing effort. We are continuously working on making our documentation even more comprehensive and easier to follow. Feel free to help us out by pointing out mistakes, limitations and better yet providing new content that we can include in our documentation offering.

Saturday, October 15, 2011

WSO2: The Mission, Vision and the Future

That was without a doubt a perfect week. I'm of course talking about the WSO2Con 2011 which was held last month at Colombo, Sri Lanka. We are still trying to cope up with the continuous flow of complements, well wishes, congratulations and thank you notes. There's no doubt in my mind that it was the most elegant and most successful technical event Sri Lanka (and perhaps even Asia) has ever witnessed. We had over 20 technical presentations conducted by people from various countries and organizations. We had participants from all corners of the world; USA, UK, Russia, India, China, New Zealand, name it. If you are doing anything related to web services or SOA that was the place to be. (If you missed it this year, start planning for next year right now)
The opening keynote of WSO2Con delivered by Dr. Sanjiva Weerawarana is now available on-line. This speech is about WSO2, WSO2 history and WSO2 vision. It's about what we have achieved, how we operate and what we intend to conquer in the years to come.
Within just a matter of 6 years, WSO2 has managed to develop a comprehensive SOA middleware platform which consists of around 15 products and a very powerful cloud platform. WSO2 products are used by some of the largest organizations in the world to handle billions of business transactions. WSO2 offers the only complete SOA PaaS in the cloud which is capable of hosting web services, web applications, data services, mashups, business processes, gadgets, integration flows and more. And WSO2 has managed to do all this while adhering to true open source principles and open standards.
This speech is about these great achievements, what motivated us to do what we do and how we are going to scale the game up in the future.

Monday, September 12, 2011

SOA & Business: Thoughts from a Pro

Most developers and architects today understand the fundamentals of SOA. SOA concepts and related technologies such as WS-* have been around for several years now and they are not exactly the new kids in town. But it's surprising to see that many business organizations are still not harnessing the true power and potential of SOA. SOA can help make a business more agile and flexible to change while reducing lot of development and maintenance overheads. It can also address many IT management problems by implanting proper SOA governance practices and processes within an organization. So then why many organizations have yet failed to realize these goals with SOA?
While many companies have been struggling to implement proper SOA, eBay, the world's largest on-line market place, has done wonders with it. Most of eBay's IT infrastructure is based on SOA principles. They have also developed a fantastic SOA platform called eBay Turmeric, which they open sourced a few months ago (this framework uses WSO2 Governance Registry inside). Sastri Malladi, who is a distinguished architect at eBay has been a driving force behind these SOA adoption efforts. In a couple of days time he's going to deliver a keynote speech titled "Service Orientation - Why is it good for business?" at WSO2Con. If you are wondering why business organizations should consider SOA or why they are failing to implement proper SOA, this is the session to attend. I'm sure it's going to be a very exciting session.

Saturday, September 10, 2011

SOA & Security: Keeping the Evil Away

SOA is about opening up. Instead of writing tightly coupled code, you implement your business logic and data access logic as reusable services which can be accessed over a network. Applications, clients and other stakeholders consume these services in varying manners to interact with your business. But more data and business logic you open up, more insecure and vulnerable you organization becomes to various types of attacks. Therefore you need to think about the security aspects of your SOA deployment right from the beginning. Security should be an integrated part of your solution and not a patched up layer that lives elsewhere. We all know what happens to systems with poorly designed security aspects. Need I remind you the well known case of Cablegate?
Fortunately for developers and architects, there are tons of security mechanisms and technologies out there. However selecting the right set of security technologies is usually a quite cumbersome task. You need to carefully weigh in many factors including level of security required, performance, hardware availability and user experience. You should also take into account the strengths and weaknesses of each security mechanism before deploying any of them in your organization.
Prabath Siriwardena, one of our security Gurus, is preparing to deliver a speech titled “Security in Practice” next week at WSO2Con. He will be explaining some of the above mentioned challenges associated with SOA security and how to overcome them using widely accepted security technologies such as WS-Security, WS-Trust, SAML, XACML, OpenID and OAuth. Prabath will discuss each of these technologies in detail, while exploring the pros and cons of each approach. If you are looking to develop a system based on SOA or just looking around to learn the security best practices in the industry, this is the session to attend. So don’t miss this opportunity of a lifetime.

Thursday, September 8, 2011

SOA & ESB: Food for Thought

With the immense popularity of Web Services and SOA, the term "Enterprise Service Bus" has become a well known technical buzz word. Many people consider Enterprise Service Bus (ESB) to be a magic wand by which any problem in enterprise integration and SOA can be solved at a whim. But what is the real technical definition of ESB? What are the problems it's supposed to solve? What are the fundamental services and facilities provided by an ESB? What are its limitations and what are the problems it's not supposed to solve? Many people don't really know the answers to these important questions and end up using the wrong tool to solve the wrong problem.
Some people consider ESB as a shortcut to implementing SOA. And why not? Most ESB solutions out their (including our own WSO2 ESB) provide excellent support for SOAP, WSDL and a plethora of other WS-* standards. But does adding an ESB into a solution architecture really bring any SOA aspects into it? On the other hand is it possible to implement a SOA without using an ESB? Food for thought...
Now don't get me wrong. ESB is a fantastic piece of technology. The number of problems it can solve and the number of use cases it can support is mind boggling. Having worked with a number of customers in the past, I've learnt that regardless of the application of SOA principles, it's almost impossible to implement a useful integration solution without an ESB. However as developers and architects we must have a good understanding of what an ESB is and what it is capable of so we can put the ESB technology to use in a more effective manner. Knowing the answers to above questions will help us make better choices when planning out an integration project and selecting middleware for a solution implementation.
I'm going to make an attempt at answering some of the above mentioned questions in my WSO2Con talk scheduled for next Tuesday. It would be interesting to listen to the feedback other developers and architects in the audience have to offer regarding these all important issues. So if you're keen on learning the fundamentals of enterprise integration and ESB feel free to drop in to my talk titled "ESB: The Swiss Army Knife of SOA". If you're not interested in this particular topic, please do join us at WSO2Con anyway. We have 20+ other professionals lined up to deliver some amazing tech talks. I'm sure you will find it interesting and useful.

Wednesday, September 7, 2011

WSO2Con 2011... An Event Not to Miss

WSO2Con 2011 is happening next week at Colombo. It's going to be a parade of tech talks conducted by 20+ brilliant speakers from all around the world. WSO2 also has plenty of tutorials, hands-on technical sessions and entertainment items planned for the attendees. This is definitely not an event to miss.

Sunday, July 24, 2011

No Business Like Show Business

Some highlights from my visit to the Wax Museum at Hollywood, LA....

Monday, June 20, 2011

The Fantastic 4

As usual I'm late with this, but better later than never :)
After almost an year worth of planning, developing, bug fixing and testing, WSO2 ESB 4.0 is finally out. This is the first WSO2 ESB release for year 2011 and follows the 3.0.1 release which went out in September 2010. WSO2 ESB 4.0 comes with an array of new features targeting enterprise users:
  • Message stores and processors (for implementing advanced QoS levels such as in-order delivery and exactly once delivery)
  • Concept of functions for the ESB configuration language (define sequences and endpoints as parameterized reusable entities)
  • Deployment synchronizer (easily synchronize clusters of ESB nodes via the registry)
  • Built-in Qpid broker (can be used as a JMS provider or an event broker for pub-sub messaging)
  • LDAP based user store powered by Apache DS
  • HTTP relay transport (a new non-blocking HTTP transport that supports pure streaming)
In addition you will find the following new mediators in this release:
  • URL rewrite mediator (rewrite and modify URL strings)
  • Event mediator (fire events to predefined topics)
  • Conditional router mediator (implement complex routing scenarios and rules easily)
Apart from these improvements, our UI gurus have given a complete make over to the ESB management console with collapsible menus, improved input validation, and new styling. You will also find that many of the recurring bugs in the 3.x series have been fixed in this latest release.
If you are looking for a high performance, user friendly solution for enterprise integration, then WSO2 ESB is for you. It's completely free and open source under the ASL 2.0. Download today and let us know how it goes. You can get in touch with us over mail, forums and issue tracker.

Friday, June 10, 2011

WSO2 ESB Tips & Tricks 07: Cron Legacy

SOA architects and system integrators often come across situations where they want to schedule a particular task to be executed periodically. It could be a task which polls a message queue or a database at regular intervals to see whether any new data has been received. It could be a task which invokes a web service to retrieve a live feed or to check the status of an external system. Or it could be a task which performs regular house keeping activities such as cleaning up log files, updating configuration files and obtaining data backups.
WSO2 ESB provides a simple but powerful framework based on Quartz for scheduling and managing periodic tasks. The user can program tasks using Java and deploy them to the ESB as jar files. Such custom developed tasks must implement the Task interface of Apache Synapse which consists of a single method named execute(). Within this method user can implement his/her business logic which will then get executed periodically by the ESB. The compiled jar files containing the task implementation classes should be placed in the repository/components/lib directory of the ESB. These jar files will get converted into OSGi bundles at server startup and get deployed into the ESB runtime.
When it comes to scheduling tasks, the user has two options. The user can configure a simple trigger by simply specifying the number of times the task should be executed along with the period between successive runs. The following example shows how the task named foo has been scheduled to be executed every 5 seconds for 1000 times. Here the task implementation class is named org.wso2.testing.tasks.FooTask (to keep the task running until the server is shutdown, just get rid of the count attribute).
<task class="org.wso2.testing.tasks.FooTask" name="foo">
<trigger interval="5" count="1000"/>
Alternatively user can configure tasks using the well known ‘Cron’ syntax. This way it is possible to specify more complex timing requirements. The following example shows how the task foo has been scheduled to be executed every Friday at 5pm. (To learn more about Cron syntax refer this tutorial)
<task class="org.wso2.testing.tasks.FooTask" name="foo">
<trigger cron="0 0 17 ? ? FRI"/>
WSO2 ESB ships with a sample task implementation class named MessageInjector. This can be used to inject a message into the service bus periodically. Injected messages will be handled by the ‘main’ sequence so they can be routed and mediated as if they were received from a remote client. The same technique can be extended to implement a scheduled task which periodically invokes a web service. Sample 300 of WSO2 ESB documentation shows how to put the MessageInjector task to use.
<task class="org.apache.synapse.startup.tasks.MessageInjector" name="CheckPrice">
<property name="to" value="http://localhost:9000/services/SimpleStockQuoteService"/>
<property name="soapAction" value="urn:getQuote"/>
<property name="message">
<m0:getQuote xmlns:m0="http://services.samples">
<trigger interval="5"/>
Note how a set of properties are passed into the task implementation class from the XML configuration. This can be done with user developed custom tasks as well. Let’s assume the task implementation class has a private attribute named ‘studentName’. If you want to pass a value for this attribute from the task configuration you should implement a getter method and a setter method for the above attribute following the standard Java bean conventions (ie getStudentName and setStudentName). Then you can have the following property definition under the task configuration to initialize the studentName attribute.
<property name="studentName" value="Hiranya"/>
This technique can be used to initialize any primitive type attribute of the task implementation class.
WSO2 ESB scheduled tasks also support a concept called ‘pinned servers’. This comes in handy when deploying scheduled tasks in a cluster of ESB servers. In such an environment it might be required that the task get deployed only on some of the nodes and not all the nodes. But since nodes in a cluster usually share all the configuration files this becomes little tricky. The solution is to specify the pinned servers attribute in the task configuration. With that the task will get deployed only on the specified set of servers. Basically this feature allows you to pin a scheduled task to a subset of the nodes in a cluster.
<task class="org.wso2.testing.tasks.FooTask" name="foo" pinnedServers=",">
<trigger cron="0 0 17 ? ? FRI"/>
In the above example the task will get deployed only if the server hostnames match the ones specified in the pinnedServers attribute.
WSO2 ESB management console can be used to create and manage scheduled tasks. The ESB artifact uploader UI (added in ESB 3.x) can be used to upload jar files containing tasks to the ESB without restarting the server. Unfortunately current releases of WSO2 ESB do not enable the user to temporarily suspend the execution of a task. To do that user must remove the task from the ESB configuration. This limitation will be addressed in a future release.

Sunday, March 20, 2011

Best Train Ride Ever

I was in Europe with a colleague, working with some customers and we got the rare opportunity to travel from Switzerland to Austria by train. The initial plan was to take a flight, but my CEO, Dr. Sanjiva Weerawarana suggested that we take the train and enjoy the scenery along the way. Boy oh boy, wasn’t he right about that?
Starting from Zurich, the train runs several hundred kilometers parallel with the Alps, until it turns South at Salzburg to finally cross the magnificent mountain range to reach the city of Graz, our destination. We had to make two connections along the way and the trip took almost 10 hours, but it totally was worth it and we enjoyed every minute of the journey. A couple of days later, we again crossed the Alps by train, this time to get to the city of Vienna.
I just downloaded all the pictures from my camera to the computer and apparently I have taken several hundred photographs along the way. I have a hard time picking the best of them, and therefore I’m just going to upload a few random photographs here…

Wednesday, February 16, 2011

WSO2 ESB Tips & Tricks 06: Error Handling Semantics

Sequence error handling is one of the most useful yet undervalued features of WSO2 ESB. Sequence error handling brings Java try-catch like semantics into the ESB mediation flow. However, error handling semantics of WSO2 ESB slightly varies, depending on how the sequences are setup in the ESB. This seems to generate lot of confusion around the concept of ESB error handling. For those who find it difficult to understand the error handling capabilities of WSO2 ESB, following guidelines might help:
  1. If a sequence explicitly defines a fault handler using the ‘onError’ attribute, the specified fault handler will be invoked, whenever an error occurs in the sequence. This is true even if the sequence is invoked by a proxy service.
  2. If a request arrives through the main sequence and if it happens to fail within a sequence which does not explicitly define a fault handler, the default ‘fault’ sequence will be invoked.
  3. If a request arrives through a proxy service and if it happens to fail within a sequence which does not explicitly define a fault handler, the fault sequence of the proxy service will be invoked. If the proxy service does not define a fault sequence, then no fault handler will be invoked.
  4. When there is a fault handler engaged at proxy service level, and another error handler engaged at the sequence level, the sequence level error handler gets invoked in case of an error (as per rule 1). In this case the proxy service fault sequence is ignored.
To learn more about WSO2 ESB error handling, please refer the previous installment of the tips and tricks series.

Sunday, January 23, 2011

Among Stars and Planets

From my visit to NASA Ames Research Center:

It's a great place to revise your old lessons on sun, moon, earth and the other planets. If you ever visit this place, check out the cool 3D visual model of the solar system they have on the giant screen. It is controlled by a Wi-Fi operated joystick. They sometimes let the visitors operate the model for a while. I was among the lucky few :)

Saturday, January 15, 2011

Apache Synapse 2.0 Released

The Apache Synapse team reached a very important milestone last week. After a long but very busy 2 and half years, we announced the general availability of Apache Synapse version 2.0. The previous release of Synapse was version 1.2 which was released way back in year 2008. The Apache Synapse project has grown a lot in terms of code, features and community over the past 2 years and therefore you will find a horde of new features, improvements, bug fixes and samples in the latest release. Some of the noteworthy new features in this release are:
  • New, fine-grained configuration model
  • Hot deployment and hot update support for configuration artifacts
  • Priority based mediation support
  • Comprehensive eventing capabilities with WS-Eventing support
  • Secure vault for encrypting passwords in configuration files
  • File locking support in the VFS transport for concurrent polling
  • URLRewrite mediator for fast and simple URL rewriting
  • Synapse configuration observer API
  • Multiple identity support in the HTTPS transport
  • Enhanced JMX monitoring support for the NHTTP transport
  • Dead letter channel implementation (experimental)
  • Synapse XAR Maven plug-in for generating configuration artifacts
Apache Synapse is a lightweight Enterprise Service Bus (ESB) released under the Apache Software License. It is based on the tried and tested Apache Axis2 SOAP engine. It has excellent support for SOAP, REST, POX, JSON and a variety of wire level transports (HTTP/S, JMS, File transport, FIX...). The non-blocking HTTP transport of Synapse enables it to handle very high volumes of HTTP traffic over thousands of concurrent connections.
Apache Synapse also supports many WS-* standards including WS-Security, WS-ReliableMessaging, and WS-Policy. It can be easily linked up with any SOA registry for metadata management and governance purposes. Synapse has a very flexible configuration model built into it, which makes it one of the easiest ESB applications to learn and adopt.
If you are involved in any SOA projects or system integration activities, Synapse may have something to offer. So please feel free to grab the binary distro and take it for a spin. Feed back welcome on Synapse user list.

Sunday, January 2, 2011

From Fleet Week Air Show

Here are some pictures from the Fleet Week air show held on last October at San Francisco port. The main attraction of the event were the renowned Blue Angels.
(Sorry about the sloppy pictures. I’m not a pro photographer and taking still pictures of some fighter jets travelling close to the speed of sound is probably not the easiest thing in the world.)