White House API Standards, DX and UX

The White House recently published some standards for developing web APIs. While going through the documentation, I came across a new term - DX. DX stands for developer experience. As anybody would understand, providing a good developer experience is the key to the success of a web API. Developers love to program with clean, intuitive APIs. On the other hand clunky, non-intuitive APIs are difficult to program with and usually are full of nasty surprises that make the developer's life hard. Therefore DX is perhaps the single most important factor when it comes to differentiating a good API from a not-so-good API.

The term DX reminds me of another similar term - UX. As you would guess UX stands for user experience. A few years ago UX was one of the most exciting topics in the IT industry. For a moment there everybody was talking and writing about UX and how websites and applications should be developed with UX best practices in mind. It seems with the rise of the web APIs, cloud and mobile apps, DX is starting to generate a similar buzz. In fact I think for a wide range of application development, PaaS, web and middleware products DX would be way more important than UX. Stephen O'Grady was so right. Developers are the new kingmakers. 

On Premise API Management for Services in the Cloud

In some of my recent posts I explained how to install and start AppScale. I showed how to use AppScale command-line tools to manage an AppScale PaaS on virtualized environments such as Xen and IaaS environments such as EC2 and Eucalyptus. Then we also looked at how to deploy Google App Engine (GAE) apps over AppScale. In this post we are going to try something different.

Here I’m going to describe a possible hybrid architecture for deploying RESTful services in the cloud and exposing those services through an on-premise API management platform. This type of an architecture is most suitable for B2B integration scenarios where one organization provides a range of services and several other organizations consume them with their own custom use cases and SLAs. Both service providers and service consumers can greatly benefit from the proposed hybrid architecture. It enables the API providers to reap the benefits of the cloud with reduced deployment cost, reduced long-term maintenance overhead and reduced time-to-market. API consumers can use their own on-premise API management platform as a local proxy, which provides powerful access control, rate control, analytics and community features on top of the services already deployed in the cloud. 

To try this out, first spin up an AppScale PaaS in a desired cloud environment. You can refer my previous posts or go through the AppScale wiki to learn how to do this. Then we can deploy a simple RESTful web service in our AppScale cloud. Here I’m posting the source code for a simple web service called “starbucks” written in Python using the GAE APIs. The “starbucks” service can be used to submit and manage simple drink orders. It uses the GAE datastore API to store all the application data and exposes all the fundamental CRUD operations as REST calls (Creare = POST, Update = PUT, Read = GET, Delete = DELETE).

try:
  import json
except ImportError:
  import simplejson as json

import random
import uuid
from google.appengine.ext import db, webapp
from google.appengine.ext.webapp.util import run_wsgi_app

PRICE_CHART = {}

class Order(db.Model):
  order_id = db.StringProperty(required=True)
  drink = db.StringProperty(required=True)
  additions = db.StringListProperty()
  cost = db.FloatProperty()

def get_price(order):
  if PRICE_CHART.has_key(order.drink):
    price = PRICE_CHART[order.drink]
  else:
    price = random.randint(2, 6) - 0.01
    PRICE_CHART[order.drink] = price
  if order.additions is not None:
    price += 0.50 * len(order.additions)
  return price

def send_json_response(response, payload, status=200):
  response.headers['Content-Type'] = 'application/json'
  response.set_status(status)
  if isinstance(payload, Order):
    payload = {
      'id' : payload.order_id,
      'drink' : payload.drink,
      'cost' : payload.cost,
      'additions' : payload.additions
    }
  response.out.write(json.dumps(payload))

class OrderSubmissionHandler(webapp.RequestHandler):
  def post(self):
    order_info = json.loads(self.request.body)
    order_id = str(uuid.uuid1())
    drink = order_info['drink']
    order = Order(order_id=order_id, drink=drink, key_name=order_id)
    if order_info.has_key('additions'):
      additions = order_info['additions']
      if isinstance(additions, list):
        order.additions = additions
      else:
        order.additions = [ additions ]
    else:
      order.additions = []
    order.cost = get_price(order)
    order.put()
    self.response.headers['Location'] = self.request.url + '/' + order_id
    send_json_response(self.response, order, 201)

class OrderManagementHandler(webapp.RequestHandler):
    def get(self, order_id):
      order = Order.get_by_key_name(order_id)
      if order is not None:
        send_json_response(self.response, order)
      else:
        self.send_order_not_found(order_id)

    def put(self, order_id):
      order = Order.get_by_key_name(order_id)
      if order is not None:
        order_info = json.loads(self.request.body)
        drink = order_info['drink']
        order.drink = drink
        if order_info.has_key('additions'):
          additions = order_info['additions']
          if isinstance(additions, list):
            order.additions = additions
          else:
            order.additions = [ additions ]
        else:
          order.additions = []
        order.cost = get_price(order)
        order.put()
        send_json_response(self.response, order)
      else:
        self.send_order_not_found(order_id)

    def delete(self, order_id):
      order = Order.get_by_key_name(order_id)
      if order is not None:
        order.delete()
        send_json_response(self.response, order)
      else:
        self.send_order_not_found(order_id)

    def send_order_not_found(self, order_id):
      info = {
        'error' : 'Not Found',
        'message' : 'No order exists by the ID: %s' % order_id,
      }
      send_json_response(self.response, info, 404)

app = webapp.WSGIApplication([
    ('/order', OrderSubmissionHandler),
    ('/order/(.*)', OrderManagementHandler)
], debug=True)

if __name__ == '__main__':
  run_wsgi_app(app)

Before we go any further let’s take a few seconds and appreciate how simple and concise this piece of code is. With just about 100 lines of Python code we have developed a comprehensive webapp, which uses JSON as the data exchange format and also does database access and provides decent error handling. Imagine doing the same thing in a language like Java in a traditional servlet container environment. We will have to write lot more code and also bundle a ridiculous amount of additional dependencies to parse and construct JSON and perform database queries. But as seen here, GAE APIs make it absolutely trivial to develop powerful web APIs for the cloud with a minimum amount of code.

You can download the complete “starbucks” application from here. Simply extract the downloaded tar ball and you’re good to go. The webapp consists of just 2 files. The main.py contains all the source code of the app and app.yaml is the GAE webpp descriptor. No additional libraries or files are needed to make this work. Use AppScale-Tools to deploy the app in your AppScale cloud.

appscale-upload-app –-file /path/to/starbucks --keyname my_key_name

To try out the app, put the following JSON string into a file named order.json:

{
  "drink" : "Caramel Frapaccino",
  "additions" : [ "Whip Cream" ]
}

Now execute the following Curl request on your App:

curl –v –d @order.json –H “Content-type: application/json” http://host:port/order

Replace 'host' and 'port'  with the appropriate values for your AppScale PaaS. This request should return a HTTP 201 Created response with a Location header.

And now for the API management part. For this I’m going to use the open source API management solution from WSO2, a project that I was a part of a while ago. Download the latest WSO2 API Manager and install it on your local computer by extracting the zip archive. Go into the bin directory and execute wso2server.sh (or wso2server.bat for Windows) to start the API Manager. You need to have JDK 1.6 or higher installed to be able to do this.

Once the server is up and running, navigate to http://localhost:9763/publisher and sign in to the console using “admin” as both the username and the password. Go ahead and create an API for our “starbucks” service in the cloud. You can use http://host:port as the service URL where 'host' and 'port' should point to the AppScale PaaS. API creation process should be pretty straightforward. If you need any help, you can refer my past blog posts on WSO2 API Manager or go through the WSO2 documentation. Once the API is created and published, head over to the API Store at http://localhost:9763/store.

Now you can sign up at the API Store as an API consumer, generate an API key for the Starbucks API and start using it.

Submit Order:

curl –v –d @order.json –H “Content-type: application/json” –H “Authorization: Bearer api_key” http://localhost:8280/starbucks/1.0.0/order

Review Order:

curl –v –H “Authorization: Bearer api_key” http://localhost:8280/starbucks/1.0.0/order/order_id

Delete Order:

curl –v –X DELETE –H “Authorization: Bearer api_key” http://localhost:8280/starbucks/1.0.0/order/order_id

Replace 'api_key' with the API key generated by the API Store. Replace the 'order_id' with the unique identifier sent in the response for the submit order request.

There you have it. On-premise API management for services in the cloud. This looks pretty simple at first glimpse, but actually this is a quite powerful architecture. Note that all the critical components (service runtime, registry and consumer) are very well separated from each other, which allows maximum flexibility. The portions in the cloud can benefit from cloud specific features such as autoscaling to deliver the maximum throughput with optimal resource utilization. Since the API management platform is being controlled by individual consumer organizations, they can easily enforce their own custom policies, SLAs and optimize for their common access patterns.

The Era of Webapps is Over - Say Hello to Web APIs

I remember a time when developing a web application (webapp) was considered one of the coolest and most exciting feats a software developer could perform. It was not so long ago when a software product with no web application component was considered uncool, unpresentable and unmarketable. A wide range of dynamic web programming technologies and standards were born in this era and they continued to thrive thanks to the unprecedented growth and evolution of the Internet. However today if we take a closer look at how the Internet is being used in our day-to-day lives and in the business world, one thing becomes hauntingly obvious. The era of the webapps has passed. Now is the era of the web APIs.

First of all lets take a closer look at webapps. Webapps are designed to be directly consumed by human users. The user is aware of the URL through which the application can be accessed which he/she enters into a web browser. The web browser then interacts with the remote URL by making HTTP GET requests to pull content and HTTP POST requests to submit content. HTML is used as the primary data exchange format and how the content appears on the user’s web browser (style and formatting) is actually a huge deal. All the important information that should be communicated to the user must be embedded in the HTML payload as that’s the only thing that’s going to get rendered on the screen by the web browser. Things like HTTP status codes and headers do not play a big role in webapps, except perhaps when reporting an error (404 Not Found, 500 Internal Server Error etc) and performing a redirect (302 Found + Location header). 

But webapps are increasingly becoming less interesting to both developers and users. Today it’s all about web APIs. Interestingly most of the technologies that are used to develop webapps can also be used to create and expose web APIs. In fact it’s not wrong to say that web APIs are the next generation webapps and webapp development frameworks mutated into web API development frameworks as a consequence of the natural evolution of the web.

So how do web APIs differ from webapps? And what makes them cooler? Unlike webapps, web APIs are not designed to be directly consumed by human users. They are APIs, meaning they are designed to be consumed by other applications. Developers can use web APIs to construct other high level APIs and end-user applications. The end-user may or may not know the exact location or the URL with which the application is interacting with. Web APIs may use any content exchange format but JSON and XML are the most popular choices. A properly designed web API would use most of the available HTTP methods (at very least GET, POST, PUT, DELETE and OPTIONS) combined with the proper use of HTTP status codes and headers to pass critical control information. Things like layout and formatting don’t mean anything in the web API world but effective use of URL patterns, intuitiveness and simplicity of the APIs mean everything.

The end-user applications that are developed using web APIs can include desktop applications and more interestingly mobile applications. In my opinion this is the last nail in the coffin of the webapps. Simply put people are no longer browsing the web. Rather they use mobile apps. Latest reports and Internet usage surveys show that the amount of mobile Internet traffic is starting to bypass the amount of desktop Internet traffic (see the references section). This means the webapps are increasingly becoming obsolete. To point out a real world example lets take GitHub. GitHub is a pretty cool webapp, one of the best in my personal opinion. We can consume this app in a desktop environment using a traditional web browser such as Firefox. We can do the same with a smart phone, using the mini web browser the device is equipped with (eg: Safari for iPhones). But that’s not good enough for us. We need a native GitHub app for our smart phones. And as a result we now have the GitHub app for iPhone and Android platforms. Soon these mobile apps will become dominant traffic sources of GitHub. This has already happened to many of the popular social networking service providers such as Twitter and Facebook. The web API is a more critical component in these systems compared to their webapp counterparts. 

This transition from webapps to web APIs is a crucial one for technology driven companies. They have to carefully assess this trend and adjust their game plans accordingly. Many organizations have already realized the shift towards the web APIs and started to expose their business functionality as web APIs rather than as webapps. Web APIs open up businesses towards a larger and more diverse clientele with ample future proofing and more room for change. This massive push towards APIs has also given birth to the field of API management, which has now become a business of its own and starting to produce many lucrative business opportunities around the world. The growing number of on-premise and cloud API management solutions (Layer7, Apigee, Mashery etc) available is a testament to this fact.

Perhaps the most impacted by this change are the developers. They need to take a whole different stance in the way they think about software solutions that run over the web. They should learn to implement systems for other developers rather than for end users. They should learn to optimize systems for machine-to-machine interaction rather than for human computer interaction. They need to pay a lot of attention to little things like using proper HTTP codes, using proper HTTP headers, and adhering to open standards. Problems they are used to be messing with such as improving the browser compatibility of webapps and session management are becoming less important by day. They will have to learn to pay less attention to things like form authentication and adopt more HTTP-friendly security mechanisms such as BasicAuth and OAuth. API management is going to become a mainstream technology and developers will have to start treating it as primary development tool just like version controlling and issue tracking. Technology platform providers will also have to take this shift seriously. Developers no longer need webapp development frameworks. They need web API development frameworks. This is why platforms like Google App Engine has become so successful. Their staying power resides in their ability to facilitate the development of powerful web APIs with very little amount of code.

So does all this mean traditional webapps are dead (as in dead for good)? Well, not really. The need for webapps will always be there, at least for the foreseeable future. But we will see more dominant deployment and adoption of web APIs compared to webapps. Webapps will become the Cobol of 21st century; Thousands of lines of code are written every year, but nobody gives a damn. Newly implemented webapps will sit on a layer of web APIs making the webapp just one of the many frontends of a larger system. 

All in all, if you’re a developer, some very exciting times are ahead of you. So buckle up!

References

http://news.cnet.com/8301-1023_3-57556943-93/mobile-internet-traffic-gaining-fast-on-desktop-internet-traffic/

http://bits.blogs.nytimes.com/2011/10/12/mobile-accounts-for-7-percent-of-web-traffic-report-says/

http://www.kpcb.com/file/kpcb-internet-trends-2012

WSO2 API Manager: Designed for Scalability

Scalability is a tough nut to crack. When developing enterprise software and deploying them in mission critical environments, you need to think about the scalability aspects from day one. If you don’t, you may rest assured that a whole bunch of unpleasant surprises are heading your way. Some of the problems you may encounter are systems crashing inexplicably under heavy load, customers constantly rambling about the poor performance of the system and system administrators having to play watch dog to the deployed applications day in and day out. In addition to these possible mishaps, experience tells us that attempting to make a live production system scalable is hell of a lot more difficult and expensive. So it’s always wise to think about scalability before your solutions go live.

The crew at WSO2 have a firm grip on this reality. Therefore when designing and developing the WSO2 API Manager, we made scalability of the end product a top priority. We thought about how the overall solution is going to scale and how its individual components are going to scale. In general we thought about how the API Manager can scale under following circumstances.

• Growing number of API subscribers (growth of the user base)
• Growing number of APIs (growth of metadata and configurations)
• Growing number of API calls (growth of traffic)

Now let’s take a look at the architecture of WSO2 API Manager and how it can scale against the factors listed above. Following schematic provides a high level view of the major components of the product and their interactions.

When you download the WSO2 API Manager binary distribution, you get all the above components packaged as a single artifact. You can also run the entire thing in a single JVM. We call this the standalone or out-of-the-box setup. If you only have a few hundred users and a handful of APIs, then the standalone setup is probably sufficient to you. But if you have thousands and thousands of users and hundreds of APIs then you should start thinking about deploying the API Manager components in a distributed and scalable manner. Let’s go through each of the components in the above diagram and try to understand how we can make them scalable.

Databases

WSO2 API Manager uses 2 main databases - the registry database and the API management database. The registry database is used by the underlying registry components and governance components to store system and API related metadata. API management database is primarily used to store API subscriptions. In the standalone setup, these 2 databases are created in the embedded H2 server.

In a scalable setup, it will be necessary to create these databases elsewhere, ideally in a clustered and high available database engine. One may use a MySQL cluster, SQL Server cluster or an Oracle cluster for this purpose. As you may see in the next few sections of this post, in a scalable deployment we might cluster some of the internal components of the WSO2 API Manager. Therefore there will be more than one JVM involved. All these JVMs can share the same databases created in the same clustered database engine.

Settings for the registry database are configured in a file named registry.xml which resides in the repository/conf directory of the API Manager. API management database settings are configured in a file named api-manager.xml which also resides in the same directory. Additionally there’s also a master-datasources.xml file where all the different data sources can be defined and you have the option of reusing these data sources in registry.xml and api-manager.xml.

API Publisher and API Store

These 2 components are implemented as 2 web applications using Jaggery.js. However they require some of the underlying Carbon components to function – most notably the API management components, governance components and registry components. If your deployment has a large user base, then chances are both API Publisher and API Store will receive a large volume of web traffic. Therefore it’s advisable to scale these two web applications up.

One of the simplest ways to scale them up is by clustering the WSO2 API Manager. You can run multiple instances of the API Manager pointed at the same database. An external load balancer (a hardware load balancer, WSO2 Load Balancer or any HTTP load balancer) can distribute the incoming web traffic among the different API Manager nodes. Tomcat session replication can be enabled among the API Manager nodes so that the HTTP sessions established by the users are replicated across the entire cluster.

The default distribution of WSO2 API Manager has both API Publisher and API Store loaded into the same container. Therefore an out-of-the-box API Manager node plays a dual role. But you have the option of removing one of these components and making a node play a single role. That is a single node can act either as an API Publisher instance or as an API Store instance. Using this capability you can add a bit of traffic shaping into your clustered API Manager deployment. In a typical scenario there will be only a handful of people (less than 50) who create APIs but a large number of subscribers (thousands) who consume the published APIs. Therefore you can have a large cluster with many API Store nodes and a small cluster of API Publisher nodes (or even a single API Publisher node would do). Two clusters can be setup separately with their own load balancers.

Key Management

Key management component is responsible for generating and keeping track of API keys. It’s also in charge of validating API keys when APIs are invoked by subscribers. All the core functions of this component are exposed as web services.  The other components such as the API Store and API Gateway communicate with the key manager via web service calls. Therefore if your system has many consumers and if it receives a large number of API calls, then it’s definitely advisable to scale this component up.

Again the easiest way to scale this component is by clustering the API Manager deployment. That way we will get multiple key management service endpoints which can be put behind a load balancer. It’s also not a bad idea to have a separate dedicated cluster of Carbon servers that run as key management servers. An API Manager node can be stripped of its API Publisher, API Store and other unnecessary components to turn it into a dedicated key management server. 

User Management

This is the component against which all user authentication and permission checks are carried out. API Publisher and API Store frequently communicate with this component over a web service interface. In the standalone setup, a database in the embedded H2 server is used to store user profiles and roles. But in a real world deployment, this can be hooked up with a corporate LDAP or an Active Directory instance. To scale this component, we can again make use of simple clustering techniques. All the endpoints of the exposed user management services can be put behind a load balancer and exposed to the API Publisher and API Store.

API Gateway

This is the powerhouse where all the validating, throttling and routing of API calls take place. It mainly consists of WSO2 ESB components and hence can be easily clustered, just as how you would setup an ESB cluster. One of the gateway nodes will function as the primary node through which all API configuration changes are applied. API Publisher will communicate with the primary node via web service calls to deploy, update and undeploy APIs. Carbon’s deployment synchronizer can take care of propagating all the configuration changes from the primary node to rest of the nodes in the gateway cluster.

API Gateway also caches a lot of information related to API key validation in order to prevent having to query the key manager frequently. This information is stored in the built-in distributed cache of Carbon (based on Infinispan). Therefore in a clustered setup, information cached by a single gateway node becomes visible to other gateway nodes in the cluster. This further helps to reduce the load on the key manager and improves the response time of API invocations.

Usage Tracking

We use WSO2 BAM components to publish, analyze and display API statistics. BAM has its own scalability model. Thrift is used to publish statistics from API Gateway to a remote Cassandra cluster. Use of Thrift ensures that statistics can be published from API Gateway to the Cassandra store at a rapid rate. The BAM data publisher also employs its own queuing mechanism and thread pool so that data can be published asynchronously without having any impact on the messages routed through the API Gateway. Use of Cassandra enables fast read-write operations on enormous data sets. 

Once the data has been written to the Cassandra cluster, Hadoop and Hive are used to process the collected information. Analyzed data are then stored in a separate database from which API Manager (or any other monitoring application) can pull out the numbers and display in various forms of tables and charts.

Putting It All Together

As you can see WSO2 API Manager provides many options to scale up its individual components. However it doesn’t mean you should scale up each and every piece of it for the overall solution to be scalable. You should decide which components to scale up by looking at your requirements and the expected usage patterns of the solution. For instance, if you only have a handful of subscribers you don’t have to worry about scaling up API Store and API Publisher, regardless of how much traffic they are going to send. If you have thousands of subscribers, but only a handful of them are actually sending any traffic, then the scalability of API Store will be more important than scaling up the Gateway and statistics collection components.

WSO2 API Manager 1.0.0 Goes GA

Last Friday we released WSO2 API Manager 1.0. It is the result of months of hard work. We started brainstorming about a WSO2 branded API management solution back in mid 2011. Few months later, in October, I implemented API support for Apache Synapse, which was a huge step in improving the REST support in our integration platform (specially in WSO2 ESB). This addition also brought us several steps closer to implementing a fully fledged API management solution based on WSO2 Carbon and related components. Then somewhere around February 2012, a team of WSO2 engineers officially started working on the WSO2 API Manager product. Idea was simple - combine our existing components to offer a smooth and end-to-end API management experience while addressing a number of challenges such as API provisioning, API governance, API security and API monitoring. The idea of combining our mediation, governance, identity and activity monitoring components to build the ultimate API management solution was a fascinating one to think about even for us.

I officially joined the WSO2 API Manager team in late April. It's been 15 hectic weeks since then but at the same time it's been 15 enjoyable weeks. Nothing is more fulfilling than seeing a project evolving from a set of isolated components into a feature complete product with its own UIs, samples and documentation. The development team was also one of the best a guy could ask for with each member delivering his/her part to the fullest, consistently going beyond expectations. 

This release of WSO2 API Manager supports creating APIs, versioning them and then publishing them into an 'API Store' after a review process. API documentation, technical metadata and ownership information can also be collected and tracked through the solution. The built-in API Store allows API consumers to browse the published APIs, provide feedback on them, and ultimately obtain API keys required to access them. API security is based on OAuth bearer token profile and OAuth resource owner grant types are supported to allow end-user authentication for the APIs. The API gateway (runtime) publishes events and statistics to a remote BAM server which then runs a series of analyzers  to extract useful usage information and display them on a dashboard. 

We are currently working with a group of customers and independent analysts to evaluate the product and further improve it. Objective is to go into 'release early - release often' mode and do a series of patch releases, thereby driving the product into maturity quickly. You can also join the effort by downloading the product, trying out a few scenarios and giving us some feedback on our mailing lists. You can report any issues or feature requests on our JIRA. Please refer the on-line documentation if you need any clarifications on any features. Have fun!

WSO2 API Manager Community Features: The Social Side of Things

Ability to build, nurture and sustain a healthy community of subscribers (API consumers) is one of the most prominent features expected from an API management solution. However the ability of the solution to support a rich and growing user base never stands on its own. In fact it's always contingent upon many functional, usability and social aspects of the underlying software. Techees such as myself, usually do a good job when identifying and implementing the functional side of things, but we suck at identifying other non-technical nitty-grittys. Therefore when designing the social aspects of WSO2 API Manager, we went through a ton of API management related literature. We wanted to make sure that the solution we build fits the customer needs and industry expectations well. We read many articles, blogs and case studies that highlighted the community aspects expected in API management solutions and how various software vendors have adopted those principles. We also talked to a number of customers who were either looking to enter the business of API management or were already struggling with some API management solution. As a result of this exercise we were able to identify a long list of functional and non-functional requirements that has a direct impact on the social aspects of API management solutions. I'm listing some of the most important ones here:

1. Ability to associate documentation and samples with APIs
2. Overall user friendliness of the API store
3. Ability to tag APIs and powerful search capabilities
4. Ability to rate APIs
5. Ability to provider feedback on APIs
6. Ability to track API usage by individual subscribers

If you go through the WSO2 API Manager Beta release you will notice how we have incorporated some of the above requirements into the product design.  If you login to API Publisher as a user who has the “Create” permission, then you are given a set of options to create documents for each API.

Again we have taken into consideration the fact that some API providers might already have tons of documentation, managed by an external CMS. For such users, it is not required to import all the available documents from the CMS into the WSO2 API Manager. They can continue to use the CMS for managing files and simply manage file references (URLs) through the API Manager.

Once the APIs are published to the API Store, subscribers and potential subscribers can browse through the available documents.

Subscribers are also given options to rate and comment on APIs.

API providers can associate one or more tags with each API.

Subscribers can use these tags to quickly jump into the APIs they are interested in.

In the monitoring front, WSO2 API Manager allows API providers to track how often individual subscribers have invoked the APIs.

In general, all the above features combine to give a pretty sleek and smooth API management experience as well as a strong notion of a user community. Feel free to browse through the other related features offered by WSO2 API Manager and see how the end-to-end story fits together. Personally I don't think we are 100% there yet in terms of the social aspects of the product, but I think we are off to a great start (anybody who tells you that their solution is 100% complete in social aspects is full of crap).

One very crucial feature that we are currently lacking in this area is alerting and notifications. Ideally the API Manager should notify the API subscribers about any changes that may occur in an API (for an example, an API becoming deprecated). On the other hand it should alert API providers when an API is not generating enough hype or subscriptions. We are already brainstorming about ways to add these missing pieces into the picture. Idea is to take them up as soon as the API Manager 1.0.0 goes GA so hopefully we can have an even more compelling community features story by the 1.1 release.  

API Life Cycles with WSO2 API Manager

One of the key API management challenges that we're trying to address in WSO2 API Manager is governing the API life cycle. Once you have developed, tested and published an API, you need to maintain it. Time to time you will roll out newer versions of the API into production with bug fixes, functional improvements and performance enhancements. When a new version is rolled out, you need to deprecate previously published versions thereby encouraging the subscribers to use the latest stable version of the API. However you will have to keep supporting the old versions for a period of time, thus providing existing subscribers some concession time to migrate to the latest version. But at some point you will completely take the old API versions off production and retire them from service. At a glance all this looks very simple and straightforward. But when you add the usual DevOps overhead into the picture, managing the API life cycle becomes one of the most complicated, long running tasks that you need deal with. If this long term procedure isn't tedious enough, there are many short term maintenance activities you have to do on APIs which are already in production. More often than not, you would want to temporarily block the active APIs so the system administrators can perform the required maintenance work on them. A well thought out API life cycle design should take these long term as well as short term requirements into consideration.

We at WSO2 gave this problem a long, hard thought and came up with the following schematic which illustrates the major life cycle states of a typical API.

If you download the beta release of WSO2 API Manager, you will see how we have incorporated the above model into the product design. Not only the product supports different life cycle states, it makes it absolutely simple to switch between different states. It's quite fascinating to see how API Publisher, API Store and API Gateway adjust to the changes in perfect synchronism as you take an API through its complete life cycle. In general these are the semantics that we have enforced for each life cycle state.

• Created – A new API has been created and awaiting moderation. It hasn't been published to the API Store yet and hence cannot be consumed.
• Published – API has been moderated and published to the API Store and is available to subscribers. Subscribers can use them in applications, generate keys against them and consume them.
• Deprecated – API has been marked for retirement in near future. New subscriptions are not allowed on the API, but the existing subscriptions and keys will continue to work. However the existing subscribers are highly advised to upgrade to the latest version of the API.
• Blocked – The API has been temporarily disabled for maintenance.
• Retired – The API has been permanently taken off service.

Lets try some of this stuff out and see for ourselves. Extract the downloaded WSO2 API Manager Beta pack, go into the “bin” directory and start the server up. Launch your web browser and enter the URL https://localhost:9443/publisher. Login to the portal using the default admin credentials (user: admin, pass: admin). Go ahead and create a new API. You can create the “FindTweets” API described in one of previous posts.

Note that the API is initially in the “CREATED” state. While in this state, the API is not visible in the API Store. Point your web browser to http://localhost:9763/store to confirm this. Now go back to the API Publisher portal and click on the “FindTweets” API. Select the “Life Cycle” tab which contains the necessary controls to change the API life cycle state. 

To start with put the API into the “PUBLISHED” state and click “Update”. Go to the API Store and refresh the page. The “FindTweets” API should now be visible there. 

You can login to the API Store, click on it and subscribe to the API using the “DefaultApplication”. When done your “My Subscriptions” page will look something like the following.

Go ahead and generate a production key too. Using this you can invoke the “FindTweets” API. Refer to my previous “Hello World” post if you're not sure how to invoke an API.

Now go back to API Publisher and change the API state to “BLOCKED”. If you immediately come back to the “My Subscriptions” page and refresh it, you will see that the “FindTweets” API is displayed as a “Blocked” API.

Any attempts to invoke this API will result in a HTTP 503 response.

Go back to the API Publisher and change the API state back to “PUBLISHED”. Try to invoke the API again, and you will notice that things are back to normal. Now in the API Publisher, go to the “Overview” tab of the “FindTweets” API and click on the “Copy” button.

Enter a new version number (eg: 2.0.0) to create a new version of the “FindTweets” API. The newly created version will be initially in the “CREATED” state and hence will not show up in API Store. However the API Publisher will list both versions of the API.

Now go to the “Life Cycle” tab of “FindTweets-2.0.0” and select the “PUBLISHED” state from the drop down. This time, since your API has some older versions, you will get some additional options to check.

Make sure all the provided options are checked. Checking the “Deprecate Old Versions” option ensures that all the older versions of the “FindTweets” API will be put into the “DEPRECATED” state as you roll out the new version into API Store. Checking the “Require Re-Subscription” option makes sure that existing subscribers will have to re-subscribe to the new version of the API in order to consume it. That is the keys generated against the old version of the API will not be forward compatible with the new version of the API. Having checked all the options, click the “Update” button to apply the changes. If you go back and check your “My Subscriptions” page you will notice that the API has been marked as “Deprecated”.

However, if you go to the API Store home page you will notice that the old version of “FindTweets” is no longer listed there. Only the latest published version is listed on this page.

Even if you somehow manage to access the “FindTweets-1.0.0” API details page on API Store (may be via the link available in “My Subscriptions” page), you will notice that you are no longer given any options for subscribing to the API. In other words, no new subscriptions are allowed on the deprecated APIs.

However if you try to invoke the old version of the API, using the previously obtained key, you will notice that it continues to function as usual.

And now finally, go back to the API Publisher and put the “FindTweets 1.0.0” API to the “RETIRED” state. Check the “My Subscriptions” page after the change.

Also try invoking the API with the old key. You will get a HTTP 404 back as the API has been undeployed from the system.

I hope this post gave you a brief introduction to how API governance and API life cycle management works in WSO2 API Manager. You can try various other options available in the product such as leaving the “Require Re-Subscription” option unchecked when publishing a new version of the API. See how that will automatically subscribe old subscribers to the new version of the API. See the extent to which API meta data is copied when creating a new version of the API using the “Copy” option. Send us your feedback to dev@wso2.org or architecture@wso2.org. 

WSO2 API Manager Permission Model: Controlling What Users Can Do

Most API management solutions segregate their user base into two main categories – API providers (creators) and API consumers (subscribers). Depending on the group to which a user belongs, the system determines what that person can and cannot do once he logs in. At WSO2, we were also thinking about supporting a similar notion of user groups in WSO2 API Manager from day one. However one thing we weren't very sure was whether a model with two user groups sufficient to address the real world business and technical needs in the field of API management. After a number of brain storming sessions, heated discussions and customer meetings we identified three main roles associated with API management.

1. API Creator – Designs, develops and creates APIs. Typically a techee, such as a developer.
2. API Publisher – Reviews the work done by the API creators and approves them to be published in the API Store. Doesn't necessarily have to be a techee. 
3. API Subscriber – Browses the API Store to discover APIs and subscribe to them. Typically a technical person, who's looking to develop an application using one or more APIs.

Once we had identified the user groups, the next challenge was incorporating them into our design in a flexible manner. We didn't want to tie our implementation to a specific set of user groups (roles) as that will make the end product quite rigid. Most users would want to simply plug their existing user store (a corporate LDAP or an Active Directory instance) into WSO2 API Manager, and they wouldn't want to introduce WSO2 API Manager specific user groups into their user store. Ideally they would like to designate some of their existing user groups as API creators, publishers and subscribers. And then we needed our solution to handle various edge cases without much of an effort. For an example, some companies would not want API creators and publishers to be different user groups. Some companies would prefer, API creators or publishers to also act as API subscribers so they can test the production APIs by directly consuming them. As you can see these edge cases can easily blur the distinction we have between different user groups.

So how do we introduce the notion of 3 user groups, while having the flexibility to reuse the roles defined elsewhere and combine multiple roles together when required? Thankfully, our WSO2 Carbon architecture has the perfect answer. Carbon gives us a powerful hierarchical permission model. So we introduced 3 Carbon permissions instead of 3 predefined user groups. 

1. Manage > API > Create
2. Manage > API > Publish
3. Manage > API > Subscribe

These permissions can be assigned to any user group as you see fit. If required, multiple permissions can be assigned to the same group too. For an example take an LDAP server which defines 3 different user groups – A,B and C. By properly assigning the above permissions to the respective groups, we can designate users in group-A as API creators, users in group-B as API publishers and users in group-C as subscribers. If we grant both Create and Publish permissions to group-A, users in that group will be able to both create and publish APIs.

Lets take an API Manager 1.0.0-Beta1 pack and try some of these stuff out. Download the binary distribution and extract it to install the product. Go to the "bin" directory of the installation and launch the wso2server.sh file (or wso2server.bat if you are on Windows) to start the server. Once the server has fully started up, start your web browser and go to the URL https://localhost:9443/carbon. You should get to the login page of WSO2 API Manager management console.

Login to the console using default admin credentials (user: admin, pass: admin). Select the 'Configure' tab and click on 'Users and Roles' from the menu to open the "User Management" page.

Lets start by adding a couple of new roles. Click on “Roles” which will list all the currently existing roles. WSO2 API Manager uses an embedded H2 database engine as the default user store. So if you were wondering where the user 'admin' and the default roles are stored, there's your answer. To use a different user store such as an external LDAP server, you need to configure the repository/conf/user-mgt.xml file accordingly.

You will notice 3 roles in the listing – admin, everyone and subscriber. The default "subscriber" role has the "Manage > API > Subscribe" permission. But don't get the wrong idea that the system is somehow tied into this role. The default "subscriber" role is created by the self sign up component of API Manager and it is fully configurable. I will get back to that later. For now click on the “Add New Role” option and start the role creation wizard. 

Enter the name “creator” as the role name and click “Next”.

From the permission tree select the “Configure > Login” and “Manage > API > Create” permissions and click “Next”. 

You will be asked to add some users to the newly created role. But for now simply click on “Finish” to save the changes and exit the wizard. You will now see the new “creator” permission also listed on the “Roles” page. Start the role creation wizard once more to add another role named “publisher”. Assign the “Configure > Login” and “Manage > API > Publish” permission to this role.

Now lets go ahead and create a couple of user accounts and add them to the newly created roles. Select “Users and Roles” option from the breadcrumbs to go back to the “User Management” page. Click on “Users”. 

This page should list all the user accounts that exist in the embedded user store. By default  the system has only one user account (admin) and so only that item will be listed. Click on “Add New User” to initiate the user creation wizard. 

Specify “alice” as the username and “alice123” as the password and click “Next”. 

Check the “creator” role from the list of available roles. The “everyone” role will be selected by default and you are not allowed by the system to uncheck that. Click on “Finish” to complete the wizard. Both “admin” and “alice” should now be listed on the “Users” page. Launch the user creation wizard once more to create another user named “bob”. Specify “bob123” as the password and make sure to add him to the “publisher” role. When finished, sign out from the management console.

So far we have created a couple of new roles and added some users to those roles. It's time to see how the API Manager utilizes the Carbon permission system. Point your web browser to https://localhost:9443/publisher to get to the API Publisher portal. Sign in using the username “alice” and password “alice123”. Since you are logging in with a user who has the “Manage > API > Create” permission, API creation options will be displayed to you.

Click on the “New API” button and create a new API (see my previous post on creating APIs). When you're finished click on the newly created API on the API Publisher home page. 

You will be shown options to edit the API and add documentation, but you will not see any options related to publishing the API. To be able to publish an API, we must login as a user who has the “Manage > API > Publish” permission. So sign out from the portal and login again as the user “bob”.

First thing to note is you don't get the “Create API” option on the main menu. So go ahead and just select the API previously created by the user “alice” from the home page.

While on this page, note that there are no options available for “bob” to edit the API. However you will see a “Life Cycle” tab which wasn't visible earlier when you were logged in as “alice”. Select this tab, and you should see the options to publish the API. Select the entry “PUBLISHED” from the “State” drop down and click “Update” to publish the API. The “Life Cycle History” section at the bottom of the page will get updated saying that “bob” has modified the API status. 

We are done with the API Publisher portal for the moment. Lets head out to API Store. Sign out from the API Publisher and point your browser to http://localhost:9763/store.  First of all try signing in as “alice” or “bob”. You will end up with an error message similar to the following.

Only those users with the “Subscribe” permission are allowed to login to the API Store. So we need to create a new user account with the “Manage > API > Subscribe” permission. We can use the self sign up feature of API Store to do that. Click on the “Sign-up” option at the top of the API Store portal. 

Create an account with the username “chris” and password “chris123”. When completed you should see a dialog box similar to the following.

Now go ahead and login as “chris”. This time the login attempt will be successful. The self sign up component, not only creates the user account, but also adds him to the “subscriber” role that we saw earlier in the management console. Since this role already has the “Manage > API > Subscribe” permission, self signed up users can always login to the API Store without any issues. Also try signing into the API Publisher portal as the user “chris”. You will get an error message similar to the following.

A word about the default “subscriber” role. This is not a role managed by the system, but rather something created by the self sign up component of the API Manager. In other words, this is the role to which all self signed up users will be assigned. You can change the name of this role by modifying the “SubscriberRoleName” parameter in the repository/conf/api-manager.xml file. If you are using an external user store, you can specify an existing role for this purpose too. If you don't want the API Manager to attempt to create this role in the system, set the “CreateSubscriberRole” parameter in api-manager.xml to “false”. If you're going to use an existing role as the subscriber role, make sure that role has the “Configure > Login” and “Manage > API > Subscribe” permissions so that the self signed up users can login to API Store and subscribe to APIs.

In order to see the effect of granting multiple API management permissions to the same role, you can use the default “admin” account. This user is in a role named “admin” which has all the permissions in the Carbon permission tree. Therefore this user can login to both API Store and API Publisher portals. On the API Publisher, he can see all the options related API creation, editing as well as publishing.

I believe you found our work on API Manager permission model interesting. Feel free to try different combinations of permissions and different types of user stores. Send us your feedback to dev@wso2.org or architecture@wso2.org. 

WSO2 API Manager Beta is Out

WSO2 API Manager 1.0.0-Beta1 is now up for grabs. You can download the binary distribution from here. This release comes with many bug fixes, functional enhancements and UI improvements. A full list of issues resolved for this beta release is available here.

One of the most significant changes done in this release is changing the URL contexts of the default API Publisher and API Store portals. API Publisher which was previously accessible on https://host:port/apiprovider has been moved to https://host:port/publisher. API Store which was previously available on http://host:port/apistore is now available on http://host:port/store.

We intend to release another beta version (1.0.0-Beta2) of API Manager before we make the final 1.0.0 release candidate available. In the meantime please try out our latest release and give us your valuable feedback on dev@wso2.org or architecture@wso2.org.

Hello World with WSO2 API Manager

A couple of months back, I took a little break from WSO2 ESB related work and focused my attention on WSO2 API Manager, one of our new and upcoming products. WSO2 API Manager is an enterprise grade, open source API management solution which solves a number of technical and business challenges faced by API providers as well as API consumers. It provides a simple, yet feature-rich platform for API providers to create and manage APIs. WSO2 API Manager is designed with software development and distribution best practices in mind which makes API governance, SLA management and monitoring absolutely trivial. For API consumers it provides a sleek, user friendly, App Store like experience where the consumers can browse through published APIs, subscribe to them, obtain API keys and provide feedback on APIs. 

In this blog post, I intend to walk you through some of the basic features of WSO2 API Manager. It’s written in the form of a short tutorial so you can actually try this out on your own. I’m going to start at the absolute beginning (downloading and installing the product), so hopefully you will be able to follow the material without any hiccups. I will provide more in-depth architectural details regarding WSO2 API Manager in a future post. 

Setting Up

Last week we announced the 1.0.0-alpha release of WSO2 API Manager. You can now download this version from our website. Start by downloading the binary distribution (wso2am-1.0.0-alpha.zip).The only prerequisite needed to run WSO2 API Manager is an installation of JDK 1.6 or higher. We generally recommend Oracle (Sun) JDK 1.6.0_23 but it should work fine on later versions of the JDK too including JDK 1.7. So far we have tested this release on Windows, Mac OS and several flavors of Linux. So if you have anyone of these operating systems with a JDK conforming to the above specs, you should be in a position to install and run WSO2 API Manager.

To install, simply extract the downloaded archive. Then open up a shell (or a command prompt if you’re on Windows) and go to the bin directory of the installation. You will find several shell scripts and batch files in this directory. To start the server up, just execute the wso2server.sh script (or wso2server.bat if you’re on Windows). You will start seeing a bunch of log statements on the console. Server start up may take several seconds. Wait until the server has fully booted up. When the server start up process is complete you will see the following log entry on the console.

[2012-07-07 01:30:31,154]  INFO - StartupFinalizerServiceComponent Server           :  WSO2 API Manager-1.0.0-ALPHA

[2012-07-07 01:30:31,155]  INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 20 sec

[2012-07-07 01:30:31,415]  INFO - CarbonUIServiceComponent Mgt Console URL  : https://10.114.95.181:9443/carbon/

Creating APIs

Now we are all set to put WSO2 API Manager into action. Launch your web browser and navigate to https://localhost:9443/apiprovider. You should see the API Provider login screen as shown below.

Specify username and password as “admin” to sign in. You will be directed to the API Provider home page. This page usually lists all the APIs created. Since we are logging in for the first time and haven’t created any APIs so far, you won’t see anything here except for a welcome message and a “New API” button.

Click on the “New API” button which will take you to the “Add API” form. It’s time to create our first API. As our first example we are going to create an API named FindTweets which is based on the on-line search functionality provided by Twitter (See http://search.twitter.com). Consumers of our FindTweets API will be able to search for Tweets by providing a keyword as the input. Our API will accept both GET and POST requests and the output will be an Atom feed.

Enter following values into the respective fields of the Add API form. Optionally you can also download a nice looking Twitter icon and upload it as the API image. Leave other fields as they are.

• Name: FindTweets
• Context: /twitter
• Version: 1.0.0
• Endpoint: http://search.twitter.com
• Tiers: Gold, Silver, Bronze (select all 3 – this field supports multiple values)
• Business owner: Bruce Wayne
• Business owner e-mail: dark.knight@wayne-enterprises.com
• Technical owner: Peter Parker
• Technical owner e-mail: spidey@dailybugle.com

Once you have entered all the above values correctly, click the “Create” button to save the API configuration. You will be taken back to the API Provider home page where the new FindTweets API will be listed.

Now click on the FindTweet API. The API overview tab should get opened for you where you will be able to see a summary of the information you entered when creating the API.

Publishing APIs

When you create a new API, it's initially in the “CREATED” state. When an API is in this state, the consumers cannot see it in the API Store.  You can verify this by pointing your web browser to http://localhost:9763/apistore.  In order for the API to appear in the API Store, we should promote the API to the “PUBLISHED” state. To do this, go back to the API Provider home page and click on the FindTweets API. Select the “Life Cycle” tab, where you will see a drop down with different states the API can be in. 

Select the “PUBLISHED” state in the drop down and click the “Update” button to apply the state change. The API Provider will immediately start showing the API state as “PUBLISHED”. Now if you go back to the API Store at http://localhost:9763/apistore and refresh the page, you should be able to see the FindTweets API listed on the home page itself, before even signing into the portal.

Subscribing to APIs

Up to this point we have created a new API and successfully published it into the API Store. Now it's time to subscribe to our FindTweets API and start consuming it. Login to the API Store using the default administrator credentials (user: admin, pass: admin).  Click on the FindTweets API to go to the API overview page. There you should be able to see the “Subscribe” option. 

In order to subscribe to an API, you should first have an application in place. An application is a logical collection of APIs. Idea is that an API consumer (typically a developer) develops applications using one or more APIs. So the consumer can create a logical application in WSO2 API Manager, and subscribe to all the relevant APIs using that application. Afterwards the consumer can obtain a key against his/her application using which any API in that application can be invoked.

When you login to API Store for the first time, you are given an application named “DefaultApplication”. Select this entry in the “Applications” drop down. The “Tiers” drop down will list all the tiers you specified when creating the API. If you followed the instructions in this post to the letter, you should see Gold, Silver and Bronze listed here. Select the “Gold” tier and click on the “Subscribe” button. A pop up will appear asking whether you want to proceed to the “My Subscriptions” page. 

Select this option and go to the “My Subscriptions” page. There you should be able to see the “DefaultApplication” entry along with its subscription to the FindTweets API. 

Click the “Generate Production Key” option and then click the “Show Key” option to obtain a key for your application. Congratulations! You are now all set to consume the FindTweets API.

Testing the API

You can use any REST client to invoke the FindTweets API. For command line geeks like me, I can recommend Curl. If you're looking for some kind of a GUI tool, I can recommend Jmeter or SOAP UI. I recently came across a neat little Chrome extension named “Advanced REST Client” which is also a pretty awesome way to interact with the WSO2 API Manager.

If you're using Curl, the following command should invoke the FindTweets API and get you a result.

curl -v -H “Authorization: Bearer xxxxx” http://localhost:8280/twitter/1.0.0/search.atom?q=wso2

Replace the 'xxxx' with the application key you obtained earlier. If you pass the right key, the API Manager will route the call to search.twitter.com and get you an Atom feed consisting of all the recent Tweets tagged with the keyword 'wso2'.

Here's how you can try the same using the Advanced REST Client for Chrome.

Note how we are passing the application key in the Authorization header of the message, prefixed by the string “Bearer”. This is because WSO2 API Manager enforces OAuth security on all the published APIs. Any consumer that talks to the API Manager should send their credential (application key) as per the OAuth bearer token profile. If you don't send an application key or send a wrong key, you will receive a 401 Unauthorized response in return.

Also remember that we subscribed to the FindTweets API using the “Gold” tier. This tier only allows 20 requests per minute. So if you keep sending requests fast, after 20 requests the throttling policy will kick in and you will start receiving 503 responses.

So there you have it; a simple 'Hello World' scenario with WSO2 API Manager. I'm sure at this point you're itching to try out more use cases with this product. Here are a few things you can try out:

1. Add more users with different permissions and see how the user experience changes according to the available permissions
2. BAM integration for monitoring APIs
3. Adding API documents
4. Rating and commenting on APIs
5. Deprecating and blocking APIs

Expect a few more blog posts from me covering the above topics. If you want to see a recording of the scenario I have described above, take a look at this webinar recording. Last but not least, go through this blog post by Kin Lane, the famous API evangelist.