Topic - RESTFul HTTP Fulcrum
Topic - RESTFul HTTP Fulcrum Topic - RESTFul HTTP Fulcrum
from forum Feedback and Suggestions
 forum index   my profile   search 
 new topic  post reply 
moderators: pjr tab
RESTFul HTTP Fulcrum
Joined: 28-July-2007
Posts: 83
Location: Boston, MA USA
Posted: 1-April-2008 14:27
I recommend extending the Front-End HTTP Fulcrum to create a RESTFul HTTP Fulcrum.

I. Update HTTP Method to NetKernel Verb Translation

Rather than have every HTTP Request translate into a SOURCE sub-request, map the following HTTP Methods to NetKernel Verbs

GET --> SOURCE
POST --> SINK
PUT -->  NEW
DELETE --> DELETE

III. Update HTTP Parameter Mapping Approach

Simplify how parameters are passed from the HTTP Request to the NetKernel sub-request. The current approach where some parameters are passed as param or param1 is a bit confusing and not very intuitive (sourced as param:param or param:param1).

URL parameters would be passed as individual parameters rather than as NVP XML document. For instance, ?foo=bar would translate to param:foo and would be sourced as a StringAspect. And if an xml document (student-data.xml) is uploaded via a PUT/POST, then it would be passed as param:student-data and would be sourced as either a StringAspect or IXAspect.[/i][/b]
Joined: 3-August-2006
Posts: 48
Location: Tucson, AZ
Posted: 1-April-2008 18:20
JDeane wrote:

III. Update HTTP Parameter Mapping Approach

Simplify how parameters are passed from the HTTP Request to the NetKernel sub-request. The current approach where some parameters are passed as param or param1 is a bit confusing and not very intuitive (sourced as param:param or param:param1).

URL parameters would be passed as individual parameters rather than as NVP XML document. For instance, ?foo=bar would translate to param:foo and would be sourced as a StringAspect. And if an xml document (student-data.xml) is uploaded via a PUT/POST, then it would be passed as param:student-data and would be sourced as either a StringAspect or IXAspect.[/i][/b]


I think this suggestion is yet another voice asking that the HTTP transport be rewritten, not only to improve the old code, but also to fix the verb handling and the argument handling. JDeane hints at what several of us have been saying for awhile now....that arguments should be accessible from Accessors in a TRANSPORT INDEPENDENT manner; the accessor should not have to know where to look for its arguments. So, in my opinion, the current HTTP scheme is exactly backwards; the Query parameters should be put into the main argument namespace and the HTTP-specific parameters should be accessible only from a sub-namespace (such as param:http:XXX). This should be true for all transports but the HTTP transport is the canonical model for the others.
Joined: 7-February-2006
Posts: 55
Posted: 2-April-2008 03:30
+1

:)
Re: RESTful HTTP Fulcrum
Joined: 7-February-2005
Posts: 591
Location: UK
Posted: 3-April-2008 09:34
Hi Jeremy/Tom,

You're right the current HTTP fulcrum is showing the effects of maintaining backwards legacy compatibility.  As you know the front-end-fulcrum is just the module that is hosting the HTTP transport. The HTTP transport is based on Jetty 5.1 which has been remarkably stable and dependable.

Our philosophy has been to use the Unix principle of creating tools that can be composed together to form higher order solutions.

With this in mind - the HTTP transport is application model neutral - when it receives an external HTTP request it just wraps the low-level HttpRequest/HttpResponse objects into a single aspect.  It creates and issues a sub-request into its host module (the Fulcrum) with the aspect as a pass-by-value argument.  The URI of the request is the same as the external URL.  However this is an internal request to the NK address space - *not* an application protocol network request.  The verb of this request is SOURCE.   The reason for this is fairly simple - at this level the transport is not attempting to apply any semantic or logical interpretation of the HTTP request.  It is a transport and is neutral to the application model.  Therefore the transport is saying - "hey I need a resource to send back as my HTTP response - here's the resource I've been given".  So the transport really is SOURCEing a resource from the address space.  It just happens to have exactly the same name as the external resource but we've moved inside.

If you want to work with the raw HTTP request, then this initial transport request can be mapped to your application. So you have full opportunity to interact with the HttpRequest and HttpResponse - in this respect, if you route these requests to an accessor, then that accessor would be almost exactly equivalent to a servlet.

However as we started developing solutions with NK we realised that it was convenient to introduce a companion tool.   A mapper service to which the initial internal request can be routed.  The job of this service is to take the low-level request and to pre-process based upon a declarative ruleset provided by an external configuration resource.  This is called the HTTPBridge - it does things like convert query parameters to a NameValuePair map etc.

As time has gone on the number of jobs that the bridge can do has grown.  It even has a SOAP mode in which it can process the HttpRequest for WS-* message processing and deal with the protocol level HEADERs etc etc (strong-RESTful advocates spit now!).

True to our design philosophy, this tool still tries to stay application neutral.  Therefore when it has pre-processed the request it issues an active URI - with the same base URI as the original request (apart from the scheme is changed from http: to jetty: - this is to prevent the possibility of accidental loop-back requests over http: back through the layer1 http client out on the network and back to the transport).  Any pre-processed state from the original request is added as active URI arguments - +param@[...] +cookies@[,...] - depending on the rules provided in the bridge config.

Yet again this request is issued with a SOURCE verb. Why? Because the bridge is a mapper, it is requesting a resource in order to provide it as the response to the initial transport request.  This tool really does want to SOURCE a resource.  The external HTTP verb can be pre-processed and added as an argument to the bridge's inner request.

So, the HTTP bridge is designed to remain neutral since it can be used by many internal applications.  I agree it could use some love - its actually a very simple piece of code that looks messy because its got the messy job of normalizing the distributed information that is coming in at the HTTP protocol level. (I don't think anyone would say HTTP is beautiful).

I understand your perspective and on first glance it seems like - "hey this NetKernel thing claims to use REST and yet its not mapping the HTTP verbs into NK's equivalents".  Ironically it is actually because we *are* taking great care to observe a RESTful style for this tool set that we are not mapping the verbs through ;-)

Now pragmatically speaking I can see that what you propose would be really helpful for people wanting to develop HTTP REST services.  As I see it there are two choices.

1)  Write another mapper tool - a verb mapper - it would be placed after the HTTPBridge and would map the HTTP request verb coming from the bridge as an active argument to the request level NK verb on the mapped sub-request.

2) Write a new HTTPBridge with a mode that supports this verb translation.  I don't propose to add this to the current bridge - there've been too many retro-fits in the last 5-years.  It needs a clean slate.

I'd really welcome your thoughts and would love to turn this sentiment for change into a community process.

Now the really ironic part of this for me is that several years ago in building up this generation of NK tools we learned that formally REST is incomplete.  Tony and I are currently heads down on the next generation NK - we're hoping to have it available for community participation in time for the Architects Weekend NetKernel conference in September.  In fact I am working on the next generation HTTP processing toolset right now and the refined abstraction makes a whole load of new patterns possible.  Sorry I know its crappy to drop hints and not reveal it but at the moment the details that are occupying Tony and I are just too abstract to get into yet.

So for the time being lets please work together to define the solution that we need for NK3.  And please be patient with us on next gen NK and come along in September to find out everything!

Cheers,

Peter
RESTful HTTP Fulcrum
Joined: 28-July-2007
Posts: 83
Location: Boston, MA USA
Posted: 3-April-2008 14:58
On our project, using NetKernel as an ESB, our goal was to make our primary services RESTful and transport impendent. In order to do this we had to add a post-processor to the current HTTP Bridge that mapped an HTTP method to an NK verb. After mapping the verb, the post-processor issues a sub-request to primary service using that NK verb (its response is returned to the HTTP Bridge and so on…) and the URI. Note that the post-processor had to copy all the arguments from the original request to the new sub-request (hint…hint…that would be great utility).

For JMS, we followed a similar approach where a JMS property (create, retrieve, update, delete) was mapped to an NK verb. And the sub-request was issued using that NK verb and a URI also included as a JMS parameter.

Thus, I could imagine a generic mapper dynamically configured via an XML file. This mapper could then be associated to a transport specific bridge (e.g. HTTP, JMS). Of course the existing Bridge code would have to be refactored so that a mapper could be plugged in.

But the verb mapping is only a minor concern compared to how the HTTP parameters are passed along (see my previous post). And that would certainly require a complete rewrite of the HTTP Bridge (definitely in the category of red-headed step-child code).

Finally, at lest you are not as bad as Steve Jobs who, every six months, claims he has
boil the ocean
software ;)[/quote]
On Parts and Wholes
Joined: 7-February-2005
Posts: 591
Location: UK
Posted: 4-April-2008 05:55
Jeremy writes...

But the verb mapping is only a minor concern compared to how the HTTP parameters are passed along (see my previous post). And that would certainly require a complete rewrite of the HTTP Bridge (definitely in the category of red-headed step-child code).


I can see your point of view.  Having the parameters as URI addressable resources would be convenient for certain classes of application.  I agree that this is a mode that the updated NK3 bridge should support.   [continuing the crappy teasers - this is already supported in NK4's bridge].

But stepping back.  You've actually inadvertently stood on an intellectual landmine.  I'll explain why in a bit.  First let me explain why the bridge works the way it does at the moment.

The parameters can be seen as a set of individual resources.  They can also be considered as a single entity 'parameters' - that is, we can consider the parameters as a single set.  In the current HTTP bridge it treats them as a set - this is the map of name value pairs.

The current bridge makes an engineering compromise and passes the set.  As a set the parameters can be transrepted and can be processed collectively - for example I can write a stylesheet to  generate a form by operating on the set of parameters (in XML form) as a whole, or I can create a SQL statement by using the parameters as a set in a single statement-generating transform service.  From a composite solution perspective each set operation on the parameters can be thought of individually.  It turns out that from a system engineering point-of-view this granularity is about right - but it also keeps us out of jail on some deep philosophical intricacies which for pragmatic and sound engineering reasons NK3 hedges on.

To see what I mean lets take  'parameters as individual resources' to the extreme limit.

Imagine you have a form with 10 fields and the bridge processes that into an active URI with 10 named arguments.  No problem, very convenient, no need to write any code to operate on the collection - just SOURCE this:param:fieldx or whatever the field is called.  (As I said, lets make the new NK3 bridge support this as a mode - it would make some applications simpler).

OK so lets keep thinking about this.  Now consider a form with 20 fields or 100 fields or, I've actually spoken to people working on RESTful geospatial mapping applications where they have thousands of parameters and actually run up against the 4K URL browser limit - but that's an extreme case. In principle 20 parameters is still ok for an application.  But what about to an XRL mapper layer sitting behind the bridge?  Its model is to take a base URI and map it one-to-one with an inner URI and declaratively pass on arguments on the sub-request.  Suddenly you've got an explosion of complexity in this declarative code.  The developer at this layer has to think about every argument - miss one and the application below fails - brittleness creeps in.   Or, as would be more likely  the developer decides to use <args>all</args> and passes each one through to be on the safe side.   Suddenly the computation required to implement the mapper pattern goes from almost immeasurable to something finite (small probably but definitely a real system cost).

OK. OK. I know this is being provocatively stupid. But bear with me - I'm coming to the point.

Consider another case.  What if I have a form and it has repeated field names (actually a pretty common scenario in real world applications).   So the bridge splits this up. Does the bridge identify each part with a unique name +field1@...+field2@... etc  or does it give them the same name and put them in order?   (Oh crap turns out we've stood on a cluster of  landmines).   Name uniqueness is one of the deep questions in REST and the wider Resource Oriented world.  Should the developer code for uniqueness or code for order?

If you code for uniqueness your code becomes tightly coupled and brittlenesss creeps in.  If you code for order, who's going to offer the guarantee of order -  OK the bridge can guarantee order at the first layer.  Does a mapper have to guarantee order? What if the developer writing the declarative argument passing doesn't know the order?  Should they? They must if they are going to sub-sample and preserve order? And for sanity you must sub-sample otherwise the layer below's interface has to be as wide as the mappers. But now you've got coupling between layers.  Don't even think about the implications of order preservation for rewriting and coding regexs.

Sorry to say that you've left the domain where the kernel should do anything to help either - for many many practical and philosophical reasons the kernel doesn't care what you call anything, the name you give to any resource is your right and responsibility - order within a name is application specific both syntactically and semantically.   Imagine the computational overhead of the kernel doing order checking on every request?

We can go further.  Should the parameter be addressable more finely?  It is represented as a String which is a set of characters.  Should that set be individually addressable? If field1 had a value of 'hello'. Should we support EXISTs on this:param:field1:hello ?   [reductio ad absurdum: the characters are actually a representation of integer numbers, from set theory numbers are sets derived from the empty set, the empty set is usually represented as {} so 0={}. 1={{}}, 2={{},{}}={1,1}. What about a really fine grained addressing scheme this:param:field1:{{}}{}}{{}}{{}{{{}}{{}}}{}}}}}  or should that be {}{{}{}{}{}{}:{}{}}{}{}{}{}{}:{}{}{{}{{}}}}}}}}}} ?]

Now I'm being deliberately super obtuse to make a deeper point and hopefully to show that we try not to make arbitrary choices.  This discussion is an example of  something which comes up repeatedly, during the development of the kernel, the address space abstraction and even sometimes with higher order tools like the bridge.  We call it 'The Part Whole Dichotomy'.   That's a grand name - to use a name like that you're either a pseudo intellectual idiot or you are trying to be very careful and formally correct.  I'll let you decide which I am!

What this term acknowledges is that there is a fundamental duality in the foundation of resources.  This is nothing new, Plato was the first to recognize this.  Is it a can-of-worms or a can containing worms?  Is the can a can or a cylinder of metal with two circular ends?  Is the metal metal? Or is it an amorphous collection of elemental particles arranged in a representation of the abstract platonic circle form?  Are the atoms sets of protons, neutrons and electrons?  I'll stop there cos I've forgotten too much physics and I don't know enough about worm biology to keep going.

In the introduction to Resource Oriented Computing paper I stated the first axiom as 'A resource is an abstract set of information'.  [Notice 'set'].  What I didn't talk about was granularity.   What ROC applications are doing is progressively selecting and transforming between sets.  As layers of an ROC application go down the granularity with which you want to address resources goes up.   Unfortunately there is no one size fits all model for resources.  This is where judgement and a bit of experimentation helps and eventually experience lets you take short cuts.

I'm sorry this has been such a laborious essay again.  I'll repeat that I understand the pragmatic requirements to add parameter addressing to the NK3 bridge.  Hopefully by getting into some detail I've provided a glimpse of  the sort of careful step-by-step thinking that we're currently engaged in with the next gen NK work.  As another teaser I can say that NK4 addresses and applies itself to these fundamentals of ROC without hedging.
RESTFul HTTP Fulcrum
Joined: 28-July-2007
Posts: 83
Location: Boston, MA USA
Posted: 4-April-2008 13:40
Ok, so my ears are still ringing from the intellectual landmine explosion but here it goes ;)

Great points and after your explanation above I understand. I just have one last question/comment. What about forms that also upload attachments? Would it not be better for the attachment to be available as a param:<name of the file>? This would also make it easier to process RESTful requests that are handling HTTP PUT/POST <some XML or JSON file>.
File addressing / Also point for debate
Joined: 7-February-2005
Posts: 591
Location: UK
Posted: 4-April-2008 14:07
Jeremey writes...

What about forms that also upload attachments? Would it not be better for the attachment to be available as a param:<name of the file>? This would also make it easier to process RESTful requests that are handling HTTP PUT/POST <some XML or JSON file>


+1 - that works for me.

Question for the general discussion.  I'd like to change the in-bound transport originated request URI.  It is currently dangerous that it issues the internal request using the external URL since you can accidentally let it slip out the bottom of the system and callback on the top.  Currently the bridge modifes the scheme to protect from this, but this is really the wrong place.

So the transort needs to change the scheme but preserve the path of the external HTTP request.  I have a proposal.  The boundary between the HTTP world and the NK world is kind of like a mirror.  Would a reasonable convention be to have any transport that recieves external URIs to reverse the letters of the scheme?

So for example.

external URL request: http://1060.org/forum/
internal URI request: ptth://1060.org/form/


What to do about mutliple server wire protocols - say an app is exposed to http:// and https://.   Does the app care since the protocol decision is an administrative deployment. Should it be sptth:// or just uniformaly move http:/ and https:/ to ptth:/ ?

Tired minds need some opinions.  [If the earlier post is long and baffling I apologise - I read your post last thing at night and was dreaming in address spaces all night.  Finally I got up at 3am this morning to get my thoughts down.]

Cheers,

P.
Enternal Request URI Scheme
Joined: 28-July-2007
Posts: 83
Location: Boston, MA USA
Posted: 4-April-2008 14:46
On the ESB we implemented using NK, resources are exposed using what we called "relative URIs". A relative URI does not have a scheme or parameters associated to it.



Cononical: http://localhost:8080/employee/12345678/profile

Relative: /employee/12345678/profile



In fact that is also how we addressed security. Privileges are comprised of an action that can be taken on a resource identified by a relative URI.



Action: SOURCE | SINK | NEW | DELETE

Relative URI: /employee/12345678/profile



Of course we ultimately had to apply a schema to the relative URIs so that they could resolve to accessors. All internal services exposing "resources" use the ffcpl scheme. And the transports (e.g. HTTP, JMS) are responsible for translating the external URI into our standard internal URI (ffcpl). Note the internel services use rewrites if we need to use other schemes such as "active".

My point is that I do not believe the external URI scheme to be important. If you want to prevent the chance of a call back, I support the mirror approach. But perhaps some Architects working on Web Applications may have a different opinion.
 new topic  post reply  To find out about new replies to this post as they occur
please subscribe to one of these feeds:
AtomRSS moderate 
© 2003-2006, 1060 Research Limited. 1060 registered trademark, NetKernel trademark of 1060 Research Limited.