Recently I got the opportunity to get involved in developing a new transport module for Apache Synapse to support the FIX protocol. Apache Synapse which is a light weight mediation framework for Web services, has had support for a number of application layer protocols like HTTP/S, SMTP, JMS and VFS. The development of the FIX transport module took nearly one and a half month and now this module is available in the Apache Synapse SVN trunk along with couple of samples and some documentation.
FIX protocol or the Financial Information eXchange is a messaging standard developed specifically to facilitate securities transactions. Strangely enough this protocol which is being used by hundreds of banks, stock exchanges and broker dealers all around the world is still not very popular in the Web services world (See here for a list of FIX users). The protocol has been in existence since 1992 and there are six major versions of the specification at the moment of writing (4.0, 4.1, 4.2, 4.3, 4.4, 5.0). The specs are owned by the FIX Protocol Limited (FPL) but it is essentially a free and open standard.
FIX specifications focus on two layers of the OSI reference model, namely the application layer and the session layer. Any application that wishes to communicate with another application using the FIX protocol must first establish a FIX session. A FIX session can exist among only two parties where one party is the acceptor and the other party is the initiator. Initiator is the one who starts the conversation by sending out the initial login request.
FIX messages are essentially a series of key-value pairs where each key-value pair is known as a field. Fields are separated by using the ASCII Start of Header (0001) character as the delimiter. The key of a field is simply a positive integer. But these integers have meanings and they are defined in the specifications. A typical FIX message might appear to be as follows.
A FIX message can be logically separated into a header, body and a trailer. The fields that should appear in each of these portions are clearly specified in the FIX specs. For an example the BeginString (8) field is a header field. The Checksum (10) is a trailer field. The content of a FIX message can vary greatly depending on the type of the message.
We used an open source FIX engine known as Quickfix/J to develop the FIX transport module for Apache Synapse. It currently supports five out of the six major versions of the FIX specification. Quickfix/J provides a very simple API to develop FIX based applications and applications developed on Quickfix/J are highly configurable. In addition to that Quickfix/J offers powerful message parsing, validation and logging.
Quickfix/J project is driven by a very active development team and a very enthusiastic user base. Quickfix/J uses Apache MINA and hence is based on Java NIO asynchronous network communications system.
All the transport modules of Apache Synapse are developed using the Apache Axis2 transport framework. Any transport module developed on the Axis2 transport framework must have two main elements, namely the transport listener and the transport receiver. The Axis2 transport framework provides the necessary interfaces and the base classes to implement these elements. The transport listener implementation is basically responsible for accepting in bound messages. For each accepted incoming message the transport listener should create an Axis2 message context, populate the message context accordingly and hand it over to the Axis2 kernel for further processing. The transport sender implementation is used by Axis2 kernel to send out messages. This implementation should be capable of processing Axis2 message contexts and converting the SOAP messages embedded in message contexts into messages that can be sent over the wire. Also depending on the nature of the protocol transport sender may also have to handle incoming response messages.
The implementations of the transport listener and the transport sender for the FIX transport module are named FIXTransportListener and FIXTransportSender respectively. The FIXTransportListener makes use of a FIXSessionFactory class which takes care of creating, storing and managing FIX sessions. The class FIXIncomingMessageHandler is where Apache Synapse binds with Quickfix/J. This class implements the quickfix.Application interface. For each accepted FIX message the transport module forks off a new thread from the thread pool associated with the transport listener implementation. This thread then converts the FIX message into XML using the Apache AXIOM API.
The FIX message converted into XML is then placed in a SOAP envelope. When converting FIX messages into XML, CDATA tags are used, basically as a precaution because theoretically FIX messages can have any kind of data in the fields. Fields can even contain XML or binary data. The transport do not change any of the field values while converting the FIX messages into XML. However if binary data is found in the message then necessary action will be taken to put the binary data in an Axis2 message context as a binary attachment. Finally the SOAP envelopes holding the FIX messages will be handed over to Axis2 kernel for further processing.
Another possible approach we could have taken here was to embed the FIX messages in SOAP envelopes without converting them into XML. But converting the FIX messages into XML has many advantages. Once converted into XML the Synapse user have more control over the FIX message content. Technologies like XPath and XQuery can be used to manipulate the FIX message content within the Synapse core.
One major problem we faced during the development of the new transport module was implementing in order message delivery. In order message delivery and processing is a characteristic feature of the FIX protocol. But since Synapse uses a separate thread to handle each incoming message we could notice that messages are not sent out in the order they were received. This is due to the thread switching that takes place while Synapse is performing the mediation. As a solution to this issue we introduced an application level sequence numbering mechanism to the transport module. Each and every incoming FIX message will be given a sequence number by the transport listener. A sequence number is unique for a given FIX session. A string that uniquely identifies the session is also associated with the messages. These information are specified in the SOAP envelope it self as attributes of the message element. The FIX transport sender implementation looks at these values and sends the messages in the exact order they were received. However having these attributes in the SOAP envelope is optional. They are required only if the user wants the FIX transport sender to send out messages in the order they were received.
While developing the FIX transport module we tried our best to provide all the options and choices Quickfix/J normally provides to the users. Apart from the logging features provided by Apache Synapse, users can enable logging at transport level (before messages are converted into XML) using Quickfix/J. Quickfix/J offers a number of options when it comes to message logging. You can either log messages on to the console, into a file or into a database. Same thing can be done with Synapse as well. Also all the message store implementations that come with Quickfix/J are available with Synapse too. By default Synapse will try to use the memory based message store implementation with acceptors and initiators. But the users can use other implementations (file, jdbc etc) if they want.
All in all developing the FIX transport module for Apache Synapse turned out to be a huge success and it indeed was a great learning opportunity for me. Not much ESBs in the world currently support the FIX protocol so it really improved the value and marketability of Apache Synapse. This module is still somewhat in its child state so we expect the contributions from the developers and FIX experts out there to improve it.
FIX protocol or the Financial Information eXchange is a messaging standard developed specifically to facilitate securities transactions. Strangely enough this protocol which is being used by hundreds of banks, stock exchanges and broker dealers all around the world is still not very popular in the Web services world (See here for a list of FIX users). The protocol has been in existence since 1992 and there are six major versions of the specification at the moment of writing (4.0, 4.1, 4.2, 4.3, 4.4, 5.0). The specs are owned by the FIX Protocol Limited (FPL) but it is essentially a free and open standard.
FIX specifications focus on two layers of the OSI reference model, namely the application layer and the session layer. Any application that wishes to communicate with another application using the FIX protocol must first establish a FIX session. A FIX session can exist among only two parties where one party is the acceptor and the other party is the initiator. Initiator is the one who starts the conversation by sending out the initial login request.
FIX messages are essentially a series of key-value pairs where each key-value pair is known as a field. Fields are separated by using the ASCII Start of Header (0001) character as the delimiter. The key of a field is simply a positive integer. But these integers have meanings and they are defined in the specifications. A typical FIX message might appear to be as follows.
8=FIX.4.09=102 35=D 34=16 49=BANZAI
52=20080314-05:01:4756=SYNAPSE 11=1205470907396
21=138=5 40=1 54=1 55=IBM 59=0
10=078
A FIX message can be logically separated into a header, body and a trailer. The fields that should appear in each of these portions are clearly specified in the FIX specs. For an example the BeginString (8) field is a header field. The Checksum (10) is a trailer field. The content of a FIX message can vary greatly depending on the type of the message.
We used an open source FIX engine known as Quickfix/J to develop the FIX transport module for Apache Synapse. It currently supports five out of the six major versions of the FIX specification. Quickfix/J provides a very simple API to develop FIX based applications and applications developed on Quickfix/J are highly configurable. In addition to that Quickfix/J offers powerful message parsing, validation and logging.
Quickfix/J project is driven by a very active development team and a very enthusiastic user base. Quickfix/J uses Apache MINA and hence is based on Java NIO asynchronous network communications system.
All the transport modules of Apache Synapse are developed using the Apache Axis2 transport framework. Any transport module developed on the Axis2 transport framework must have two main elements, namely the transport listener and the transport receiver. The Axis2 transport framework provides the necessary interfaces and the base classes to implement these elements. The transport listener implementation is basically responsible for accepting in bound messages. For each accepted incoming message the transport listener should create an Axis2 message context, populate the message context accordingly and hand it over to the Axis2 kernel for further processing. The transport sender implementation is used by Axis2 kernel to send out messages. This implementation should be capable of processing Axis2 message contexts and converting the SOAP messages embedded in message contexts into messages that can be sent over the wire. Also depending on the nature of the protocol transport sender may also have to handle incoming response messages.
The implementations of the transport listener and the transport sender for the FIX transport module are named FIXTransportListener and FIXTransportSender respectively. The FIXTransportListener makes use of a FIXSessionFactory class which takes care of creating, storing and managing FIX sessions. The class FIXIncomingMessageHandler is where Apache Synapse binds with Quickfix/J. This class implements the quickfix.Application interface. For each accepted FIX message the transport module forks off a new thread from the thread pool associated with the transport listener implementation. This thread then converts the FIX message into XML using the Apache AXIOM API.
The FIX message converted into XML is then placed in a SOAP envelope. When converting FIX messages into XML, CDATA tags are used, basically as a precaution because theoretically FIX messages can have any kind of data in the fields. Fields can even contain XML or binary data. The transport do not change any of the field values while converting the FIX messages into XML. However if binary data is found in the message then necessary action will be taken to put the binary data in an Axis2 message context as a binary attachment. Finally the SOAP envelopes holding the FIX messages will be handed over to Axis2 kernel for further processing.
Another possible approach we could have taken here was to embed the FIX messages in SOAP envelopes without converting them into XML. But converting the FIX messages into XML has many advantages. Once converted into XML the Synapse user have more control over the FIX message content. Technologies like XPath and XQuery can be used to manipulate the FIX message content within the Synapse core.
One major problem we faced during the development of the new transport module was implementing in order message delivery. In order message delivery and processing is a characteristic feature of the FIX protocol. But since Synapse uses a separate thread to handle each incoming message we could notice that messages are not sent out in the order they were received. This is due to the thread switching that takes place while Synapse is performing the mediation. As a solution to this issue we introduced an application level sequence numbering mechanism to the transport module. Each and every incoming FIX message will be given a sequence number by the transport listener. A sequence number is unique for a given FIX session. A string that uniquely identifies the session is also associated with the messages. These information are specified in the SOAP envelope it self as attributes of the message element. The FIX transport sender implementation looks at these values and sends the messages in the exact order they were received. However having these attributes in the SOAP envelope is optional. They are required only if the user wants the FIX transport sender to send out messages in the order they were received.
While developing the FIX transport module we tried our best to provide all the options and choices Quickfix/J normally provides to the users. Apart from the logging features provided by Apache Synapse, users can enable logging at transport level (before messages are converted into XML) using Quickfix/J. Quickfix/J offers a number of options when it comes to message logging. You can either log messages on to the console, into a file or into a database. Same thing can be done with Synapse as well. Also all the message store implementations that come with Quickfix/J are available with Synapse too. By default Synapse will try to use the memory based message store implementation with acceptors and initiators. But the users can use other implementations (file, jdbc etc) if they want.
All in all developing the FIX transport module for Apache Synapse turned out to be a huge success and it indeed was a great learning opportunity for me. Not much ESBs in the world currently support the FIX protocol so it really improved the value and marketability of Apache Synapse. This module is still somewhat in its child state so we expect the contributions from the developers and FIX experts out there to improve it.
3 comments:
Paul Fremantle has made a blog post on this topic. Click here to check it out.
Great work... I've written a little bit about it on my blog.
cheers
Here is what Daniel Brum has to say.
Post a Comment