Archive for the ‘REST’ Category

Twitter User Similarity and Collective Intelligence

May 13, 2010

This is the second part of a blog regarding a recent mini-project where I implemented a Twitter user similarity service.  The first part described my experience with the mechanics of the Twitter REST API – this part focuses more on the “collective intelligence” aspects.

The requirements were simple: define a concept of “similarity for two users” and implement a Twitter solution for it.

Resources used:

Being a bit rusty on basic CI concepts (my chagrin but in my defense there is so much computer stuff to know out there), I did a quick search for high quality links on the topic, drilled down a bit and read the high-value articles. I downloaded all free PDF chapters of the books and read relevant sections.  I went to my local Borders bookstore which was fortunately stocked with all the above books. I had already purchased Segaran’s book, so I used chapter 2 “Making Recommendations” which discusses the Euclidean distance Pearson correllation formula and this seemed to fit the bill. AIW also had an even more detailed discussion on the subject – too much to implement in the short time frame, but definitely a candidate for version two. I reviewed my statistics books, and lo and behold it turned out these were not exotic algorithms, but rather standard statistics data  comparison techniques. Too paraphrase an old sailing jingle: so much knowledge, so little time (so many boats, so little time).

I settled upon a defining the concept of similarity based on comparing word counts between two users for a set of Twitter status message for a given timeline. As usual I leveraged Spring for effortless configuration and bean wiring (thanks again Rod!). The basic logic was to issue calls to the Twitter API “method” user_timeline for each user. This returned a list of tweets for each user which I would iterate over and concatenate the Status text elements. I then computed a map of words and a count of all their occurences. This map was then  fed to the similarity scorer which would return a value between 0 and 1.

Last but not least was the WordCounter class. This object accepts raw text and returns a map of words and their counts. Of special interest is the lexical analyzer. For the first pass I used a simple String.split() and a list of stop words. But minimal analysis revealed a submerged world of complexity involving punctuations, stemming, etc. Whew! Ideally it too should be in interface.

Here’s a UML class diagram of the overall system:

The entry point is a service which returns a double value between 0 and 1 indicating user similarity.

    public interface SimilarityService {
        public double getSimilarityScore(String user1, String user2)
    }

This service interface has four implementations: two real providers (Twitter4j and JTwitter) that issue actual calls to the Twitter API for two user timelines. The mock implementation operated on files containing the concatenated raw text. As an inspirational freebie, I threw in the RssSimilarity provider which performed the similarity scoring on RSS feeds. Its quite cool at how much can be done so easily and quickly when you’ve got the right abstractions and layering in place. Nothing excessively fancy here except solid engineering practices all wrapped in rocking Spring. The other extension point was the similary scorer which computed a similarity score for two word count maps.

    public interface SimilarityScorer {
         public double calculateSimilarityScore(Map wordCount1,
              Map wordCount2);
    }

The two provided implementations are:

  • Euclidean Distance
  • Pearson Corellation

Other possible candidate solutions to be investigated are:

  • Manhattan (taxicab) distance
  • Jaccard distance

Overall, this was one of the more intellectually challenging projects in a while. On the “interest” scale it certainly compares with the NoSQL and eventual consistency stuff I’ve been recently doing. I certainly aim to pursue this topic more – hopefully in a remunerated capacity!

Advertisements

Twitter REST API

May 13, 2010

I recently finished a mini-project to calculate the similarity of two Twitter users. Being a REST fan and having implemented a substantial REST service, I’m always eager for an excuse to get my hands dirty with a bit o’ REST. I find it fascinating that no two REST APIs (web APIs) are the same, especially in terms of documentation.

As we all know the term “REST” refers to a style and not a standard, so it is not surprising that actual implementation vary quite a bit. This lack of  specificity is most pronounced on the client side. With SOAP, client access is almost always mediated by platform-specific client stubs automatically generated from the WSDL contract. With REST there is no such standard contract (despite WADL which has not become a dominant force), and therefore no fully automated way to build client stubs. You can partially automate the process if you define your payload as an XML schema, but this still leaves the other vital part of specifying the resource model. Section 1.3 of the JAX-RS specification explicitly states: The specification will not define client-side APIsSee my comments on the current non-standard situation of client stubs in some JAX-RS providers.

API Data Formats

In the absence of standard client stub generation mechanisms, documentation plays an increasingly important role. The fidelity to genuine REST precepts and the terminology used to describe resources and their HTTP methods becomes of prime importance to effective client usage.

How do we unambiguously describe  the different resources and methods? The number and types of payload formats influence the decision. Do we support only one format, JSON or XML? If XML, do we have a schema? If so, what schema do we use? XSD or RelaxNG? Multiple XML formats such as Atom, RSS and/or proprietary XML? By the way, the former two do not have a defined schema. Do we support multiple formats? If so, do we use prescribed REST content negotiation?

Considering the strong presence of the Twitter REST API and my short albeit intense usage of it, I am a bit reluctant to “criticize”. So upfront I issue a disclaimer that my knowledge is partial and subject to change. One very interesting fact I recently read in the book Building Social Web Applications is that over 80% of Twitter’s usage come from its API and not from its web site! Caramba, that’s quite an ecosystem that has evolved around Twitter! All the more reason to invest in API contract specification and  documentation.

General API Documentation

Professional  high quality API documentation is obviously a vital need especially as API usage increases. With an internet consumer-facing API, clients can access resources using any language of choice, so it is important to be as precise as possible. Having worked with many different APIs and services, I have come to appreciate the importance of good documentation. I regard documentation not as separate add-on to the executable code, but rather as an integral part of the experience. It is a first-order concern.

The metaphor I would suggest is DDD – Documentation Driven Development. In fact, on my last big REST project where I took on the responsibility of documenting the API, I soon found it more efficient to update the documentation as soon as any API change was made. This was especially true when data formats were modified! The document format was Atlassian Wiki which unfortunately didn’t allow for global cross-page changes, so I had to keep code and its corresponding documentation closely synchronized; otherwise the documentation would’ve quickly diverged and become unmanageable.

Deductive and Inductive Documentation

In general, documentation can be divided into deductive and inductive. Deductive documentation is based on the top-down approach. If you have an XML schema all the better – you can use this as your basic axiom, and derive all further documentation fragments in progressive refinement steps. Even in the absence of a schema, you can still leverage this principle.

Inductive documentation is solely based on examples – there is no general definition, and it is up to the client to infer commonalities. You practically have to do parallel diffs on many different XML examples to separate the common from the specific.

Theorem: all good documentation must have examples but it cannot rely only on examples! In other words, examples are necessary but not sufficient.

Google API as a Model

Google has done a great job in extracting a common subset of its many public APIs into Google Data Protocol. All Google APIs share this definition: common data formats, common errors mechanisms, header specification, collection counts, etc. Google has standardized on AtomPub and JSON as its two primary data formats (with some RSS too). It does an excellent job on having an unambiguous and clear specification of its entire protocol across all its API instances.

Take the YouTube API for example. Although neither Google nor Atom use an XML XSD schema, the precise details of the format are clearly described. Atom leverages the concept of extensions where you can insert external namespaces (vocabularies) into the base Atom XML. Google Atom does not have to reinvent the wheel for cross-cutting extensions, and can reuse common XML vocabularies in a standard way. See the Data API Protocol Page – XML element definitions page for details. Some namespaces are openSearch (Open Search Schema) for collection counts and paging, media for MRSS (yes, you can insert RSS into Atom – cool!).

Twitter Data Format Documentation

The Twitter General API documentation page and the FAQ do not do a good job in giving a high-level description of the Twitter data formats for requests and responses. There is only a one basic mention of this on the Things Every Developer Should Know page:

The API presently supports the following data formats: XML, JSON, and the RSS and Atom syndication formats, with some methods only accepting a subset of these formats.

No clear indication is given as to which kinds of resources accept which formats. Common sense would lead us to believe that JSON and proprietary XML are  isomorphic and supported for both request and responses. Being feed formats, RSS and Atom would be supported only for responses.  It is unfortunate that this is not explicitly stated anywhere.

More disturbing is the lack of an XML schema or any attempt to formally define the XML vocabulary! It seems that only XML examples are provided for each resource. Googling for “Twitter API XSD” confirms my suspicion in that it returns many mentions of “inferring XML schemas” from instances – a scary proposition indeed! What is the cardinality of XML elements? Which ones are repeatable or not? What about the data types? The DRY (don’t repeat yourself) principle is violated since you have the same XML example redundantly repeated on many pages. You can maybe get away with this for a small API, but for a widely used API such as Twitter I would have thought Twitter would have invested more resources in API contract specification.

Twitter Content Negotiation

Another concern is the way Twitter handles content negotation. Instead of using the REST convention/standard of  the ACCEPT header or a content query parameter, Twitter appends the format type to the URL (.json, .xml) which in effect creates a new resource. For example Google GData uses a query parameter such as alt=json or alt=rss to indicate data format.

TWitter API Versioning

This lack of explicit contract specification leads to problems regarding versioning. Versioning is one of those very difficult API problems that has no ideal satisfactory answer. Instead, there are partial solutions depending on the use case. Without a contract, it is difficult to even know what new changes have been implemented.

Let’s say a change is made to an XML snippet that is shared across many resource representations. Twitter would have to make changes to each resource documentation page. Even worse, it has to then have some mechanism to inform clients as to the contract change. Having some schema or at least some common way of describing the format would be a much better idea.The Twitter FAQ weakly states that clients have to proactively monitor the following:

Google uses HTTP headers to indicate API versions as well as standard namespace naming conventions.

API Client Packages

Typically REST APIs will leave it to third parties to provide language-specific client stubs. This is understandable because of the large number of languages out there – it would be prohibitive for a small(er) company  to implement and test all these packages! However the downside is that these packages are by definition non-standard (caveat emptor), and it is an open question as to how reliably they implement the current service definition.

Documentation wildly varies.   Firstly, it is not always clear which API to use if several choices exists. Being mostly a Java guy, I focus here on Java clients. For example, Twitter has four Java clients. You most often find minimal Javadoc with no further explanation. API coverage is incomplete – features are missing . For example, for the user-timeline “method” (sidebar: misuse of REST term method!), Twitter4j supports the count query parameter whereas JTwitter apparently does not. The problem here is of client fidelity to the API. When the Twitter API changes, what is the guarantee that the “mom and pop” client will sync up?

Speaking of the devil, a rather interesting development happened just the other day with Amazon’s AWS Java client. On March 22, 2010 Amazon announced  rollout of a brand new AWS SDK for Java! Until then, they too had depended on third-parties – the venerable Jets3t (only supports S3 and CloudFront) and typica. Undoubtedly this was due to client pressure for precisely those reasons enumerated above! See Mr. Jets3t’s comment on the news.

Conclusion

One of the wonders of the human condition is how people manage to work around major obstacles when there is an overriding need. The plethora of Twitter clients in the real world is truly a testimony to the ubiquity of the Twitter API.

However, there is still considerable room for improvement to remove incidental accidental complexity and maximize client productivity. Twitter is still a young company and there is still an obvious maturation process ahead.

After all, Google has many more resources to fine tune the API experience. But if I was the chief Twitter API architect, I would certainly take a long and hard look at the strategic direction of the API. Obviously there is major momentum in this direction especially with the June deprecation of Basic Auth in favor of OAuth and the realignment of the REST and Search APIs. There is no reason to blindly mimic someone else’s API documentation style (think branding), but even less reason not to learn from others (prior knowledge) and to minimize client cognitive overhead.

Comparison of JAX-RS Client Proxies

November 18, 2009

Though JAX-RS has done wonders for standardizing server-side implementation of Java REST services, there is definitely a need for some client-side standardization. Considering that for every REST implementation there are order-of-magnitude more clients, it is rather puzzling that more movement hasn’t occurred in this space.

Section 1.3 Non Goals of the JAX-RS 1.0 (Sep. 2008) spec states:

Client APIs The specification will not define client-side APIs. Other specifications are expected to provide such functionality.

Nevertheless, vendors have not been idle and have used their particular client-side frameworks as a value-add selling point. In this blog, I report some first impressions on CXF and RESTEasy‘s client APIs. Next report will be on the Jersey and Restlet versions.

Perhaps the reluctance to tackle a standard client API has something to do with the complexity associated with SOAP client proxies, but in the absence of REST client proxies, every single customer has to recreate the wheel and implement rather mundane low-level plumbing with httpclient. A chore best avoided.

Both CXF and RESTEasy support a nearly equivalent client proxy API that mirrors the server-side annotated resource implementation. They differ in two ways:

  • Bootstrap proxy creation
  • Exceptions thrown for errors

“Naturally”, each provider has a different way to create a proxy. Both are rather simple, and since they are a one-time action, their impact on the rest of the client code is minimal.

The “happy path” behavior for both implementations is the same – differences arise when exceptions are encountered. RESTEasy uses its own proprietary org.jboss.resteasy.client.ClientResponseFailure exception while CXF manages to use the standard JAX-RS exception javax.ws.rs.WebApplicationException. Therefore, round one goes to CXF since we can write all our client tests using standard JAX-RS packages. In addition, this enables us to leverage these same tests for testing the server implementation – an absolute win-win.

Note that the test examples below use the outstanding testng.

Proxy Interface

Here’s the client proxy VideoServiceProxy that is the same for both CXF and RESTeasy. Very nice guys!

public interface VideoServiceProxy {
@GET
@Path("genre")
@Produces("application/xml")
public GenreList getGenres(); 

@GET
@Path("genre/{id}/")
@Produces("application/xml")
public Genre getGenre(@PathParam("id") String id) ; 

@POST
@Path("genre")
@Consumes("application/xml")
public Response createGenre(Genre genre) ; 

@PUT
@Path("genre/{id}/")
@Consumes("application/xml")
public void updateGenre(@PathParam("id") String id, Genre genre) ; 

@DELETE
@Path("genre/{id}/")
public void deleteGenre(@PathParam("id") String id) ; 

}

Proxy Bootstrapping

The client side bootstrapping for the proxy is shown below.

CXF Bootstrapping

import org.apache.cxf.jaxrs.client.JAXRSClientFactory;
import org.apache.cxf.jaxrs.client.JAXRSClientFactoryBean;

public class GenreTest {
    static String url = "http://localhost/vapp/vservice/genre";
    static VideoServiceProxy rservice ;

    @BeforeSuite
    public void initSuite() {
        rservice = JAXRSClientFactory.create(url, VideoServiceProxy.class);
    }
}

RESTEasy Bootstrapping

import org.apache.commons.httpclient.HttpClient;
import org.jboss.resteasy.client.ProxyFactory;
import org.jboss.resteasy.plugins.providers.RegisterBuiltin;
import org.jboss.resteasy.spi.ResteasyProviderFactory;

public GenreTest {
    private static VideoServiceProxy rservice ;
    private static String url = "http://localhost/vapp/vservice/genre" ;   

    @BeforeClass
    static public void beforeClass() {
        RegisterBuiltin.register(ResteasyProviderFactory.getInstance());
        rservice = ProxyFactory.create(VideoServiceProxy.class, url, new HttpClient());
    }
}

As you can see, the CXF version is slightly less verbose and simpler in that it has less imports.

Exception Handling

Happy Path – No Errors

For a happy path the test code is the same for both CXF and RESTEasy.

@Test
public void createGenre()  {
    Genre genre = new Genre();
    genre.setName("Animals");
    genre.setDescription("All animals");
    Response response = rservice.createGenre(obj);
    int status = response.getStatus();
    Assert.assertEquals(status,Response.Status.CREATED.getStatusCode());
    String createdId = ResponseUtils.getCreatedId(response); // get ID from standard Location header
    Assert.assertNotNull(createdId);
}

CXF Unhappy Path – Error

    @Test
    public void getGenreNonExistent() {
        try {
            Genre genre = rservice.getGenre(nonExistentId);
            Assert.fail();
        }
        catch (WebApplicationException e) {
            Response response = e.getResponse();
            int status = response.getStatus();
            Assert.assertEquals(status, Response.Status.INTERNAL_SERVER_ERROR.getStatusCode()); // TODO: fix 404 not being thrown?
            //Assert.assertEquals(status, Response.Status.NOT_FOUND.getStatusCode());
        }
    }

RESTEasy Unhappy Path – Error

    @Test
    public void getGenreNonExistent() {
        try {
            Genre obj = rservice.getGenre(nonExistentId);
        }
        catch (ClientResponseFailure e) {
            ClientResponse response = e.getResponse();
            Response.Status status = response.getResponseStatus();
            Assert.assertEquals(status, Response.Status.INTERNAL_SERVER_ERROR); // TODO: fix 404 not being thrown
            //Assert.assertEquals(status, Response.Status.NOT_FOUND);
        }
    }

Note, that there is a problem in correctly throwing a 404 with WebApplicationException on the server-side. Thought my current server implementation is CXF, I have also verified that this problem exists for RESTEasy and Jersey. The framework always returns a 500 even though I specifiy a 404. This is definitely not OK. Its a TBD for me to further investigate.

    throw new WebApplicationException(Response.Status.NOT_FOUND);

I definitely plan to check out Jersey and Restlet in more detail, so stay tuned!

JAX-RS: Java™ API for RESTful
Web Services
Version 1.0
September 8, 2008

Partial Update and Immutable Fields Update Dilemma

May 17, 2009

One of the recurring problems I have encountered in creating services – be they REST, SOAP or Hibernate – is how to handle the partial update problem. A partial update refers to the ability to update a subset of fields (subelements or attributes) exposed by an object. If the representation exposes resource/object references, then the problem gets even more complex – this differential object-graph update is the topic of another blog.

Some discussions of the problem – such as Joe Gregorio in How To Do RESTful Partial Updates advocate breaking up the resource into multiple subresources that can be manipulated individually. See Subbu’s PATCH: It is the Diff for another RESTful point of view. Although my discussion will be REST-oriented, the intent is more general in scope and also aims to encompass SOAP services. The goal is to capture the commonality of the partial update problem in different contexts.

Here’s a simplified object domain model to illustrate the issue. All objects extend BaseObject – to avoid visual clutter I have not drawn the derivation connector. Note that BaseObject’s id and version fields and VideoAsset’s workflowState field are read-only fields.

Below is a full representation of the resource that you would obtain from a GET.

<videoAsset xmlns=”http://www.myvideo.com/services/rest/v1″&gt;
<id>1066</id>
<version>1</version>
<title>Battle of Hastings</title>
<originalFileKey>Battle-of-Hastings</originalFileKey>
<longDescription>Battle of Hastings</longDescription>a
<shortDescription>Battle of Hastings</shortDescription>
<videoSeconds>1000</videoSeconds>
<workflowState>NOT_DISTRIBUTED</workflowState>
<genres>
<link href=”History” kind=”name”/>
<link href=”Military” kind=”name”/>
</genres>
<medias>
<media>
<version>2</
version>
<uri>http://www.myvideo.com/media/hastings.flv</uri&gt;
<profile href=”defaultFlv” kind=”name” type=”VIDEO”/>
</media>
<media>
<
version>2</version>
<uri>http://www.myvideo.com/media/hastings.jpg</uri&gt;
<profile href=”defaultThumbnail” kind=”name” type=”IMAGE”/>
</media>
</medias>
</videoAsset>

Now let’s say you want to modify (PUT) just one field – title for example. Are you required to include all the fields of the videoAsset element or can you just specify the new value?

<videoAsset xmlns=”http://www.myvideo.com/services/rest/v1″&gt;
<title>Battle of Hastings in England</title>
</videoAsset>

The problem lies in the semantics your service associates with absent fields. If your parser finds a value for title – all is fine. But what about missing fields such as longDescription? We potentially have three cases:

  • Do not set the value of longDescription
  • Set longDescription to empty string
  • Set longDescription to null

In the above example, how do we interpret the missing fields of videoAsset? Does the fact that longDescription is missing mean that we wish to set its database value to null or not do anything at all?

In the example below, we explicitly include an empty longDescription element which the parser will pick up as an empty string. We can therefore unambiguously set its database value to an empty string.

<videoAsset xmlns=”http://www.myvideo.com/services/rest/v1″&gt;
<title>Battle of Hastings in England</title>
<longDescription></longDescription>
</videoAsset>

You can resolve this problem by using the XSD nillable constraint. But it is not always apparent what is nillable or not, and this can lead to unnecessary use of nullable columns in the database. Of course, this will work if you use XSD. If your XML payloads have no schema definition such as Atom, this wouldn’t work as neatly. Offhand, I’m not sure how RelaxNG or Schematron address this issue.

<videoAsset xmlns=”http://www.myvideo.com/services/rest/v1″&gt;
<title>New Value</title>
<longDescription></longDescription>
</videoAsset>

A major complicating factor is the difficulty that ORM frameworks have with partial updates in general. Are the semantics for missing fields in an update merge or overwrite? The irony is that both SQL and XML can easily handle granular updates – in SQL you just update the required columns and in XML you just parse the elements. It is the Hibernate/Java middle tier that introduces “accidental” complexity for this use case. Since Hibernate is “object-oriented” it updates by objects not fields. The typical solution is to “merge” the new value with the existing object, but this requires the system to read in the entire object and then persist it with the new values – hardly an ideal case. Under any load this scenario would be found wanting – both for data overhead and data consistency.

Imagine if we could just extract the title from the XML payload and just directly create a SQL update statement:

update videoAsset va set va.title=’New Value’ where va.id=1066

Of course, the ORM framework provides a host of other benefits, but the point here is to understand the limitations and new accidental complexities introduced by the framework. Alas, as always it ends up as an issue of balancing competing concerns.

Mutable Operations with Immutable Fields

Another related problem is the presence of immutable fields in the inbound payloads of mutable operations such as POST and PUT.

If you are using XSD schema (as in SOAP), much depends on how you define the optionality constraints for your elements (minOccurs and maxOccurs). Some fields can naturally be optional (description) but others such as title and originalFilekey are always required for a videoAsset.

The principal problem here is if you are using the same XML representation for multiple use cases (GET, PUT and POST). For simplicity’s sake this is often the case, but to accurately represent each particular use case you would ideally want to use a different representation. A GET-oriented XSD type (VideoAssetGet) would represent a view of the object where all optionality constraints were accurately modeled for retrieval. A POST (VideoAssetCreate) type would only expose mutable fields and would not contain read-only fields such as id, version (derived from BaseObject) or workflowState. It makes no sense to specify these fields for either a POST or PUT since they are immutable. This can get quite messy for large objects that have a complex mix of write-able and read-only fields.

So your choice boils down to using similar but different objects for each verb or you use one “fat object” to represent a union of all fields across GET, PUT and POST and make sure you have good documentation. The downside is that the service then becomes less “self-documenting”.

Typically what is done is to use one representation and ignore read-only values for POST and PUT calls – AtomPub does this with the id element the feed/id element. GoogleData also insists that you have to always GET the state of a resource and use this data to PUT it. I guess you can’t go wrong with modeling your service on Google, right?

REST Resource Modeling

December 17, 2008

If I had to pick two top REST features, they would be resource modeling and resource representation. In this blog entry I explore the former with specific reference to a recent REST project I worked on.

Resource modeling refers to defining a resource name space and constructing the corresponding identifying URLs.

Some key issues in resource modeling are:

  • Define basic resources based on domain model – business objects become resources
  • Transform those non-CRUD business operations into resources and pick appropriate HTTP methods
  • Define URL namespace
    • Opaque URLs or not?
    • Express resource relationships as URL hierarchies or embed them in the representation?
  • Should we make user account explicit or implicit in the URL?
  • Other issues: matrix parameters, query parameters, queries

Basic Resources

Core business objects can be easily transformed into corresponding resources. Assume we have the following domain objects:

  • Video
  • Genre
  • Playlist – three concrete types: static playlist, dynamic playlist or playlist group
  • Player

The resource URLs could be:

  • /v1/ACCOUNT/videos
  • /v1/ACCOUNT/genres
  • /v1/ACCOUNT/playlists
  • /v1/ACCOUNT/players

There are typically five basic CRUD operations that we may want to perform on these resources:

  • Get all resources – GET /v1/ACCOUNT/videos
  • Get one resource – GET /v1/ACCOUNT/videos/1776
  • Modify one resource – PUT /v1/ACCOUNT/videos/1776
  • Add one resource – POST /v1/ACCOUNT/videos
  • Delete one resource – DELETE /v1/ACCOUNT/videos/1776

Note that Get all and the POST operations do not have a resource ID.

One seemingly minor but important note: you have to decide whether to use the singular or plural name for the resource – video or videos. Do not use both! I recently encountered a proposal for having /videos to return all videos and /video/1812 when you want to return one video. This makes little sense to me – there is one resource called video (or videos) not two resources – video and videos.

Non-Basic Resources

Unfortunately, not all business requirements can be mapped into the CRUD-like resource paradigm. What needs to be done is to noun-ify the “verb-like” business functionality by creating an “artificial” resource noun and then apply one of the verbs. Often it is not apparent if a POST or PUT is more appropriate.

The problem is how do we transform verbs into nouns? If we need to perform an operation on a resource that cannot fit into one of the four HTTP verbs, what to do? How do we come up with an intuitive resource name and then figure out which verb is most applicable. There are no real guidelines, and an inordinate amount of time can be spent on this exercise.

With SOAP you would simply create a new operation and be done with it. Obviously, the freedom to pick any operation name is a definite advantage of SOAP. I have seen meetings get bogged down in this morass. Typically the business analysts couldn’t care less how you implement their requirement while the architect has to worry about maintaining the semantic and logic coherence of the service.

An example that I faced in a recent project, where we received a new business requirement to recall and distribute videos. We already had our standard methods for a video, and the question was how to represent this in the best REST-ian fashion? Should we invent a new synthetic resources

  • /service/v1/videorecall
  • /service/v1/videodistribute

and then execute a POST. Somehow this “smells bad” – frankly there really is no concept such as video recall object. It seems that the noun-ify the verb pattern doesn’t fit the bill. Ultimately, the solution we opted for was POST with:

  • /service/v1/videos/recall
  • /service/v1/videos/distribute

Opaque URLs

Surprisingly even such an important topic such as the nature of a resource URL is not an agreed upon topic in the REST community. One school contends that human-understandable URLs are crucial, while the other side argues that clients should not assume any meaning to the internal structure of the URL. Suffice to say, that the understandable URL makes a more compelling case in my opinion.

Resource relationships

Resources have relationships to other resources. These relationships can be expressed in two basic ways:

  • Relationships specified in the URL as a hierarchical name compound
  • Relationships embedded in the output representation
    • Target resource as link
    • Target resource with details

An example best illustrates the issue. Our video resource has a many-to-many relationship with genres (categories).

In the URL approach, the following URL would retrieve the genres of a given video: /videos/1776/genres. The core resource /videos/1776 would retrieve only basic data for the video. If you wanted to obtain any relationships you would have to issue another call. To get permissions for a video: /videos/1776/permissions.

<genres>
<genre>
<id>11</id>
<name>Animals</name>
</genre>
<genre>
<id>22</id>
<name>Sports</name>
</genre>
</genres>

The embedded approach returns basic data for the video as well as relationship information. You still have to make a decision if you want link or detailed information on the references.

Relationships as links:

<video>
<id>1776</id>
<name>Touching the Void</name>
<genres>
<link href="http://www.k2.com/service/v1/acc/genres/11" />
<link href="http://www.k2.com/service/v1/acc/genres/22" />
</genres>

Relationships as details:

<video>
<id>1776</id>
<name>Touching the Void</name>
<genres>
<genre>
<id>11</id>
<name>Animals</name>
</genre>
<genre>
<id>22</id>
<name>Sports</name>
</genre>
</genres>
</video>

There are advantages and disadvantages with all approaches and ultimately the choice depends on your intended uses cases. The situation is also complicated by the PUT and POST methods. Typically you would want to use the same representation (data format) for GET and PUT/POST.

The URL Relationship patterns can lead to a chatty protocol since if your use case typically wants to grab all information related to a video, you would have to make 1+n calls where n is the number of video associations. It can be equally expensive to create or modify videos. Furthermore, if your object model requires mandatory relationships, the URL Relationship pattern cannot satisfy this requirement.

Google YouTube API uses the URL-based relationships. For example a video has an optional one-to-many association with comments. If there are comments for the video, the resource representation for a GET will contain a URL to a collection representing the comments.

<gd:comments>
<gd:feedLink href="http://gdata.youtube.com/feeds/api/videos/Vx4_OjyPqag/comments"   countHint="993"/>
</gd:comments>

Examples:

Make user explicit or implicit in the URL?

Most services have a concept of a current user (or account). The question then is, should we expose this user name in the URL or not?

  • /service/v1/USER/videos/1519
  • /service/v1/videos/1519

Both styles would require standard authentication. The case can be made that having the user name in the URL is redundant with authentication. On the other hand, if we want a richer URL hierarchy which would allow a user to access another user’s resources (with permissions, of course), then having the user name in the URL is important.


Resource Query Language

Any service will inevitably need some way to filter the amount of data returned. This is a complex topic, but it is worth to briefly examine some salient points. When designing a REST service, this can be a very important and time consuming task, so be prepared for significant effort.

Bare-bones REST offers us two basic ways to retrieve data:
Conceptually, we want to issue three kinds of GET requests:

  • Get all resources – /videos
  • Get one resources – /videos/1492
  • Get some resources – /videos?some_query_syntax

Bare-bones REST inherently supports the first two – the constraint-qualified GET presents us with a conundrum.

There is unfortunately no standard way to express a query and each vendor provides their own syntax. So when you are designing a REST service and you need querying (you will eventually need it), you will have to go through a non-trivial process to design a query language. This requires specifying the grammar, documenting it with examples, making it consistent across all resources, and writing lots of tests (hint: think combinatorial explosion of query options).

In my recent project we were fortunate to already have an in-house XPath dialect for our key domain objects. We were able to use (without modifications) this dialect for our internal domain service, internal and external SOAP services, as well as adapting it to the REST service with trivial effort. This was truly an example where proper foresight handsomely paid off. An example:

  • /videos?q=@id=’1776′
  • /videos?q=@name=’Aconcagua Adventures’
  • /videos?q=@id=’152′ or @id=’153′
  • /videos?q=@creationDate = xs:dateTime(‘2008-07-29T18:10:36.900Z’)

Some links on public web API query syntaxes:

Examples

Google:

ADO.NET Data Services:


Adventures in REST-alandia

December 11, 2008

Overview

I recently managed a REST project from inception to deployment. As in any endeavor, there were quite a few interesting lessons to be learned.

I had long evangelized REST at this company, and finally got a chance to implement my “dream”. Although loath to use the word “fun”, I admit it was so albeit a lot of work. REST, as we all know, is a “style” and not a “spec”, so much is left to non-standard interpretation. I worked on this project right after a SOAP/WSDL project so it was interesting to compare the pluses and minuses (cost/benefit analysis) of both.

Our REST service exposes two kinds of features for video playlists and playlist groups – basic CRUD operations and more complex business-specific operations. The former are a good fit for REST – the latter are not such a good fit.

  • /service/v1/ACCOUNT/playlists/1776
  • /service/v1/ACCOUNT/videos/1066
  • /service/v1/ACCOUNT/players/1054
  • /service/v1/ACCOUNT/genres
  • /service/v1/ACCOUNT/permissions
  • /service/v1/ACCOUNT/release

One of more interesting outcomes was the active use of the live REST system by our business analyst. Granted he’s a sharp cookie, but as any one of us, he’s a very busy person. The fact that he managed to quickly teach himself the basics of HTTP GET, PUT and POST using Google’s restclient tool, underscores the lower barrier-of-entry of REST over SOAP and the former’s ultimate advantage. This is a good example of the empowering nature of an “on-site customer” which is so important to agile projects. Once the basic REST model is comprehended, its “intuitive” nature is easily grasped.

Resource Representation

We ended up using our own custom XML vocabulary defined by an XSD. I had seriously considered using Atom Pub as our format since I had been on an intense Atom-learning binge, but after careful cost-benefit analysis, I opted for custom XSD. XSD was a known and Java has better tools for Java-binding with XMLBeans. I was initially “surprised” to learn that Atom doesn’t have a normative XSD or RelaxNG schema, and even though there are rocking Abdera and Rome toolkits out there, at the end of the day you are still obligated to manually map XML to Java (and vice versa). The automated and faithful binding of XMLBeans is a compelling advantage.

Our data model would obviously require significant Atom namespace extensions, and there was obviously lots of extra work here in the too few months we had to roll out our application. We already had a doc/lit SOAP service deployed with its own XSD, so reusing this grammar was another reason to go the custom XML route. Finally, there was no compelling business reason to use Atom, and no-one else except me was familiar (or interested) with Atom. All in all the risk outweighed the benefit. Shameless plug: this is what you get when you hire me – objective advice not tainted by latest sexy trends even though it might be better for my resume to have that latest trend on it!

Nevertheless, the framework did have built-in support for content negotiation, so besides the standard XML format, Atom, MRSS and JSON were returned for selected resources as an experimental next-generation feature. All you have to do is specify a query parameter “format=atom” for GET and lo and behold the format would be different. Conneg is truly one of the key distinguishing features of REST from other paradigms that can take a while to get used to. Since it is such a paradigm shift, the consequent plethora of rich opportunities are not immediately apparent. From an implementation perspective, the pluggable nature of Spring dependency injection readily leads itself to variant formats.

Errors

REST proponents make much of the simplicity of returning standard HTTP error codes, but upon further examination some serious limitations of this approach are apparent:

  • HTTP error codes are split into 400 client and 500 server errors. This is definitely unlike most standard programming use cases.
  • HTTP error codes are not rich enough to capture business errors.

As we all know, happy path programming is never adequate, and true enterprise-style applications have to account for a wide variety of errors. Paying business clients do NOT want to see Java exceptions or stack traces – but the cost of mapping these to intelligent error messages can carry a non-trivial cost. Much depends on the imagination of the product owners – they need to envision how negative the down-stream cost of confusing and incoherent error messages will be on the client and insist that development put in place an exception handling mechanism that will deliver accurate and understandable messages.

It just seems that a simple concept such as an error reporting mechanism and format could be standardized upon. SOAP has the SOAP Fault element, and all SOAP applications can interoperably process an application’s errors. We need to build upon HTTP errors and have a richer mechanism.

Client and Server Error Codes

The distinction between errors caused by the client or server is not typically encountered in non-REST programming scenarios. This is a prime example of one of those pesky “devil in the details” (Tasmanian devil?) that can cause inordinate grief. For every error encountered, we have to decide whether is a client or server error. The need to pass down the client/server context to a the culprit exception is a non-trivial exercise.

For example, assume we have an XML syntax error on a playlist representation. If this error generated by the client or the server? If it is an inbound request, a 4xx client error code should be generated. If the error is encountered when returning a response, a 5xx code should be returned since it is the server that messed up. The upshot is this: unlike “normal cases” the context (client or server) has to be passed down to every place where an XML syntax error could be encountered. Ugh! This is definitely a lot of work. Much simpler to return the ubiquitous 500 for most everything even though it ain’t pretty.

The inevitable choice:

  • Dispense with this expensive error cause differentiation and return less helpful error messages.
  • Be faithful to the spirit of REST/HTTP and give clients informative and intelligent messages but pay a non-significant cost.

Adequacy of HTTP Error Codes

The HTTP model supports a limited range of errors codes – some for the server and some for the client. Surprisingly, there is no extension mechanism available to supply application-specific error codes. Of course, the HTTP response Reason Phrase can be used, but what is missing is a code – a well-known extensible value that a program can switch on.
Only two server error messages – 500 (Internal Server Error) or 501 (Not Implemented) are available. Obviously, this is hardly adequate to account for real-life error cases.

Client error codes are similar – 400 (Bad Request. The request contains bad syntax or cannot be fulfilled) or 409 (Conflict). For example, if the client specified a non-existent video for a playlist, how can we convey the precise message to the client? It is not a syntax error – its a semantic error. HTTP has no provision for the latter. A 409 typically represents a concurrent update clash or a unique constraint.

Two non-standard solutions for returning application-specific error codes:

  • Return the application error code in the response body
  • Return the application error code in a custom response header, e.g. X-MyApp-Error-Code

Response body error messages require the client to switch on the returned content. This implies that errors that must be trapped by standard Servlet web.xml features are for the whole web application. If the REST service owns the WAR that is OK – if it shares it with other services (our case), then it has to account for error use cases of these other services – not a necessarily a non-trivial task if you are adding a new service to an existing multi-service WAR.

Response headers have the advantage of consistency in that the client looks for errors only in one place. The client checks for “major” errors in the standard HTTP Status Code – and “minor” errors in a custom HTTP header.

Resource Modeling Limitations

Classical CRUD semantics for resources are a good fit for REST services. Inevitably however, more complex business requirements that enlist several resources are not as easily supported by REST.

The unlimited operation name space of SOAP operations allows you to easily add any new functionality that the business mandates. The constraining nature of four HTTP verbs can often make it difficult to come up with an appropriate resource name. Noun-ifying a new operation doesn’t seem to be always the best course. As a conscientious designer, I want to keep my resource model as coherent as possible and REST compliant. Pragmatically, business people are less concerned about this, and often time pressures can lead us to add non-REST-ian resource names. With SOAP you just add a new operation name and some arguments – with REST there can be a lot of hand-wringing and “religious discussions” on what is the best approach. Again, this is one of the consequences of implementing a “style” as opposed to a “specification”.

Moving a Playlist from one Playlist Group to Another

For example, we encountered the problem of moving a child playlist from one playlist group to another. Assume we want to move “Beyonce 30-th Birthday” playlist from “My Playlist Group” to “Your Playlist Group”. There is no natural REST mapping of this use case – a lot of extra cognitive overhead has to be expended in creating a non-standard solution. In the procedural (e.g. SOAP) mode, this could be easily implemented as:

move(“Beyonce 30-the Birthday”,”My Playlist Group”,”Your Playlist Group”)

or if you use IDs:

move(1776, 1812, 1861)

In a REST-ian model, this action enlists three resources: playlist, source playlist group and target playlist group. Which resource is this operation to be based upon? The playlist or one of the playlist groups? Which HTTP method should be used: POST or PUT? One example:

PUT /playlist/1776

<source id=”1812″/>
<target id=”1865″/>

Subbu Allamaraju – a Yahoo REST thought leader – has proposed the “Account Transfer Pattern” where you create a new resource that atomically encompasses the participant resources. He also specified the operation as a POST – we are creating a new “transfer” resource – in this case a transfer of a playlist from one group to another.

To summarize, there is no standard REST “spec” or common usage for such use cases – and it is up to each application to create its own composite resources with non-trivial cognitive overhead.

Distributing and Recalling a Video

Another example was a new requirement that we add the capability of distributing and recalling a video. Conceptually (in an OO non-RESTian manner) this tells me to add a new method on the video object. But I don’t have this ability in REST for I am limited to the four holy verbs. So do I create new resources such as:

  • /service/v1/ACCOUNT/videodistribute
  • /service/v1/ACCOUNT/videorecall

To distribute a video, do I use a POST or PUT? But this just doesn’t seem intuitively correct to me. After some pondering I decided upon:

  • /service/v1/ACCOUNT/videos/distribute
  • /service/v1/ACCOUNT/videos/recall

using a POST where the request body contained a list of video IDs to distribute or recall plus other information such as a message.

Batching

REST is a good fit for manipulating individual resources, but when it comes to manipulating several resources at once things get a bit more difficult. The main problem with batching is how do we successfully report errors for those batch items that failed and those that did not? The whole HTTP error mechanism proves inadequate here. If we submit ten items in a batch job and three fail, we do not want to return a 400 since this would not correctly convey that seven had succeeded. We are left with the task of inventing our own mechanism – again a non-standard way to solve a standard problem.

There has been recognition of this problem in the REST community, and each application is providing its own conventions – for one example, see Youtube batching .

Using our video example, a regular singular get is:

  • /service/v1/ACCOUNT/videos/1492

Now we want to “batch-ify” the resource:

  • /service/v1/ACCOUNT/videos/1492,1519

Simple enough you say – just return a list of video representations. But what if 1492 doesn’t exist and 1519 does exist. In the singular version we can return the conventional 404. In the batch version, we don’t have this option and ultimately we have to somehow convey those batch items that succeeded and those that didn’t. We have to then enhance our response data format to contain this information. Things get messy pretty quickly.

In the first SOAP-based incarnation of the service, we had actually designed all our SOAP operations as batch. Every response object was derived from a base class that had a list of result statuses (success or error) and the actual result data if successful. Although this was a clean solution, for those clients that were only manipulating one object this proved irksome. Furthermore, implementing batch on the back end is a non-trivial exercise. Nevertheless, for those business use cases and for performance sake (i.e. avoiding multiple remote calls) batch is needed.

Conclusion

Alas, as Fred Brooks pointed out a while back, there is no silver bullet in our business! REST excels for “simple” cases – but the case can be made that SOAP has better support for arbitrary non-CRUD complex usage scenarios.

Web Linking

We all know that web linking is a “good” thing, but again, when it comes time implement it things get get a bit sticky. But that “we” might not extend to the business people involved in the resource modeling. Typically business folks aren’t that well versed in REST-ian convention and don’t really care that much. They just want the job done with minimal cost. Furthermore, IDs can be simpler to understand and manipulate. Amazon Web services deal in IDs.

<video>
<id>1453</id>
<genres>
<link href=”1618″ />
</genres>
</video>

or

<video>
<id>1453</id>
<genres>
<link href=”http://www.foobar.com/service/v1/ACCOUNT/genres/1618&#8243; />
</genres>
</video>

Our particular business requirements made links especially problematic. Firstly, the business wanted to use genre names and not IDs. IDs were too hard to remember and to keep track of. As long as both names and IDs are unique, this could have been done. However, our genres were hierarchical and names were not unique but full genre paths were. For example, “animals/canines/dogs” instead of “dogs” since there was nothing to prohibit you from having “pets/dogs”. As you can see, putting a slash delimited path in the URI can be confusing.

Furthermore, our requirements were to only expose genres and permissions as collections and there was no requirement to expose singular versions. GET service/v1/ACCOUNT/genres would return a collection of genre representations but service/v1/ACCOUNT/genres/1618 would return a 404. Granted it seems inconsistent and “not pretty”, but the alternative is to deploy and test a feature the business doesn’t explicitly want. So web links to individual genres could not be dereferenced – you would get a 404!

The outcome was to add a “kind” attribute to the link element with three possible values: id, name or uri. We rolled out the application using id or name links but retained the ability to support uri in the future.

Another problem with the uri version is that it required quite a bit of extra work on the implementation side since the request context had to be passed down the stack to every place a reference was being constructed. It wasn’t sufficient just to have the object’s ID or name – we also had to have the current URL of the application. Again, the devil is in the details.

WADL

There has been apparent movement to using WADL as a service definition language a la WSDL.
Some preliminary thoughts on WADL:

  • A good idea in theory – in practice questions remain.
  • Most surprising and interesting is that WADL hasn’t been improved since Nov. 2006. Que pasa? Does anyone own this still? Progress is needed, is anyone responsible for this? Hmmm. Red-ish flags.
  • Current WADL doesn’t really support the DRY principal – resource-wide features (query parameters, request/response headers, XML representatons) are not specified in one place. Instead, they are repeated in-line for every resource URI. For any REST API with a significant number of resources, this quickly leads to repetitive, confusing and overwhelming feature descriptions.

Implementation

The REST service was implemented as a Java web application with a custom in-house REST framework based upon Spring. The reason for this was partially historical – the project had started as a skunks work and was substantially developed by the time it was officially blessed. So considering the compressed delivery schedule and limited resources, we went ahead with what we had. In addition, I had reservations about the maturity of two prime candidate frameworks – restlet and Jersey.

Neither had self-evident easy support of Spring and that was a deal breaker in itself. Jersey at that time didn’t seem mature enough plus I have reservations about its exclusive use of annotations to define the REST contract. Mixing service-level declarative information with implementation Java code is not a good practice in my opinion even though it is popular in many circles today – the fad du jour syndrome. But that’s another topic. Furthermore our business requirements were relatively simple from the REST-ian perspective (no fancy matrix parameters or conneg) so the home-grown system was adequate.

Sidebar: I am pleasantly surprised with JPA’s well-thought out mechanism for over-riding annotations with XML descriptors. Too bad JAX-RS doesn’t provide a similar mechanism.

One thing I wished I had time to finish was a Spring namespace extension that defined a REST-ian DSL that could more succinctly capture REST declarative semantics. But time was short and darn – that Spring feature is quite complex. So many cool software toys – so little time.