New Configuration Model of Synapse

If you are trying out a latest Apache Synapse build off the SVN trunk, you will notice that the Synapse configuration model has gone through some significant changes lately. In the past, Synapse was programmed to load the entire mediation configuration (sequences, endpoints, proxy services, tasks etc) from a single XML file, named synapse.xml. Synapse would parse this XML file at startup and construct an object model known as the SynapseConfiguration which contains the definitions of all active mediation components at runtime. While this approach was simple and clean, it had a number of limitations:

• Configuration management becomes a nightmare as the size of the synapse.xml grows. A typical heavy duty configuration would consist of dozens of proxy services, sequences and endpoints. When all these are packed into a single XML file, it becomes difficult to locate a single configuration item quickly.
• With all configuration artifacts stored in a single flat file, it is almost impossible to develop satisfactory tooling support for configuration development and maintenance.
• A team of developers cannot work on configuring a single Synapse instance at the same time.
• Hot deployment is a feature that Synapse has been lacking for years. Being able to hot deploy a proxy service or a sequence into Synapse without having to restart the service bus is a great convenience at development time. But a configuration model based on a single XML file is not at all capable of handling such requirements.

Considering these drawbacks, I implemented a new configuration model for Synapse, known as the multi XML configuration builder. The new model, which is now the default configuration model in Synapse, loads the configuration from a structured file hierarchy, instead of loading the mediation configuration from a single XML file. With this model in place, each endpoint, sequence and proxy service has to be defined in separate XML files. The directory structure for storing these individual configuration files is as follows:

synapse-config
   |
   +-- registry.xml
   +-- endpoints
   +-- event-sources
   +-- local-entries
   +-- proxy-services
   +-- sequences
   `-- tasks

As you can see there are separate dedicated directories for each type of artifacts. Each of these directories can house zero or more XML configuration files. Each file must have the .xml extension to be recognized by the Synapse configuration builder. If you take a look at the source code you will notice that I have enforced this restriction using a Java FileFilter:

    private static FileFilter filter = new FileFilter() {
       public boolean accept(File pathname) {
           return (pathname.isFile() && pathname.getName().endsWith(".xml"));
       }
   };

   private static void createProxyServices(SynapseConfiguration synapseConfig, String rootDirPath)
           throws XMLStreamException {

       File proxyServicesDir = new File(rootDirPath, PROXY_SERVICES_DIR);
       if (proxyServicesDir.exists()) {
           if (log.isDebugEnabled()) {
               log.debug("Loading proxy services from : " + proxyServicesDir.getPath());
           }
           File[] proxyDefinitions = proxyServicesDir.listFiles(filter);
           for (File file : proxyDefinitions) {
               try {
                   OMElement document = parseFile(file);
                   ProxyService proxy = SynapseXMLConfigurationFactory.defineProxy(
                           synapseConfig, document);
                   proxy.setFileName(file.getName());
                   SynapseArtifactDeploymentStore.getInstance().addArtifact(
                           file.getAbsolutePath(), proxy.getName());
               } catch (FileNotFoundException ignored) {}
           }
       }
   }

The mediation registry is defined in the registry.xml file which should be placed at the top level of the file hierarchy.

So does the multi XML configuration builder solve the problems in the old configuration model? Let’s consider the facts:

• With Synapse configuration broken down into smaller, manageable pieces the whole configuration becomes easier to manage and keep track of. As long as the XML files are named appropriately, it is extremely easy to quickly locate a particular configuration item. We recommend using the artifact names to name the corresponding XML files. For an example the file containing the definition of the FooProxy can be named FooProxy.xml.
• With the multi XML configuration builder, developing powerful and elegant tools for creating pieces of the service bus configuration becomes a trivial task. Also one can use conventional configuration management tools and version controlling systems such as Subversion to store and manage the configuration artifacts.
• A team of developers can now work on configuring Synapse. Each developer in the team can work on his own configuration file or set of files.
• Supporting hot deployment is now feasible. As a matter of fact, Ruwan implemented hot deployment and hot update support for Synapse based on the multi XML configuration builder a few weeks back. This feature is now available in the Synapse trunk and will be available for the next Synapse release.

So there you go. Target achieved. There is one little glitch in this approach though. That is how do we handle backward compatibility with older Synapse versions? For instance how can a Synapse 1.2 user, who has a single synapse.xml file, migrate to a new Synapse version? We have provided a solution for that as well. In the new Synapse configuration file hierarchy you can place a synapse.xml file at the top level (alongside with registry.xml). All the mediation components defined in this synapse.xml will be loaded to the service bus at startup along with any other components defined inside the individual directories. So a Synapse 1.2 user can simply copy the existing synapse.xml file to the synapse-config directory in a new Synapse distribution, and it will be picked up by the service bus. In addition to this convenience feature, we are planning on developing some migration tools that can help users to easily migrate an old Synapse configuration file onto a newer version of Synapse.

As far as Synapse is concerned each configuration builder should be associated with a corresponding configuration serializer implementation. Serializers are used to convert the SynapseConfiguration object model back to the textual/XML form. So for the multi XML configuration builder I developed a matching multi XML configuration serializer which can save a SynapseConfiguration object model to a file hierarchy. Similar to the configuration builder this implementation was also heavily dependent on Java file IO APIs. However, after a while we realized that the serializer is not working as expected on certain platforms; most notably on Windows. After running some debug sessions and doing some on-line reading I realized that the Java file IO operations are not consistent on every platform. As a result sometimes the serializer would encounter trouble creating a new file or moving an existing file on Windows.

At this point, my colleague, Rajika suggested using Apache Commons IO API for file manipulation. Commons IO API provides a nice layer of abstraction on top of the standard Java IO APIs. It handles all file IO operations in a consistent and platform independent manner. So I got rid of almost all the Java file IO code in the multi XML configuration serializer and replaced them with corresponding calls to the Commons IO API. Some sample code fragments are shown below:

    private void cleanUpDirectory()  throws Exception {
       // If the target directory already exists and contains any files simply rename it to
       // create a backup - This method does not delete the target directory
       if (rootDirectory.exists() && rootDirectory.isDirectory() &&
               rootDirectory.listFiles().length > 0) {

           if (log.isDebugEnabled()) {
               log.debug("The directory :" + rootDirectory.getPath() + " already exists. " +
                       "Creating a backup.");
           }

           backupDirectory = new File(rootDirectory.getParentFile(), "__tmp" +
                   new GregorianCalendar().getTimeInMillis());
           FileUtils.moveDirectory(rootDirectory, backupDirectory);
       }

       // Create a new target directory
       FileUtils.forceMkdir(rootDirectory);
   }

   private void writeToFile(OMElement content, File file) throws Exception {
       File tempFile = File.createTempFile("syn_mx_", ".xml");
       OutputStream out = new FileOutputStream(tempFile);
       XMLPrettyPrinter.prettify(content, out);
       out.flush();
       out.close();

       FileUtils.copyFile(tempFile, file);
       FileUtils.deleteQuietly(tempFile);
   }

FileUtils is the entry point for file IO operations in the Commons IO API. It provides you with a range of useful methods for managing and manipulating files.

Note that when writing to a file we first write the whole thing to a temporary file and once completed we copy the temp file to the final location. This was required to handle certain edge cases in the hot update implementation. Without that the hot updater will attempt to pickup and process half baked files (Again mostly on Windows – It seems Windows writes to files in chunks).

All in all, Apache Synapse is now equipped with a powerful new configuration model, a matching configuration serializer and hot deployment capabilities that leverage the new configuration model. Speaking of hot deployment, it is also based on the Axis2 hot deployment framework. Therefore it is configured in the axis2.xml file. We are yet to do a Synapse release with all these new exciting features but if you are itching to try them out feel free to grab a nightly build. Also be aware that WSO2 ESB 3.0, which is based on Synapse, has been released and that release contains all these new improvements.