1060.org

Deconstructing the Bugxter Web-Application

TAB 28/3/2006 - This document is now obsolete. A better guide to deconstructing a web application can be found with the 1060 Forum Guide

Introduction

In this article we will deconstruct the Bugxter web-application in order to highlight some typical NetKernel architectural patterns. Bugxter is a flexible database backed bug/issue tracking application. It is available for installation through the NetKernel Install Wizard.

After installation you can explore the details of the Bugxter implementation by unzipping the module jar files - bugxter_web3.0.0jar and bugxter_services3.0.0jar.

Bugxter has been a test-case application during the evolutionary development of the NetKernel system. Version 1 of Bugxter was a very simple XML only test harness for the DPML runtime. Version 2 was a fully functional application but used a page-based design similar to a PHP, ASP or JSP architecture - this was found to be inflexible and did not take advantage of the NetKernel infrastructure. Version 3 employs a layered-service approach which decouples presentation, control, data model and flexibly overlays access control and caching - it is the layered approach to application composition offered by NetKernel which we hope to highlight with this article.

To get the best from this article we recommend that you first read the Introduction to NetKernel article which was first published on theserverside.com.

Bugxter Specifics

In order to describe the general design patterns used in the Bugxter application we first need to briefly describe the specific breakdown of the application.

Figure 1: Bugxter functional units

Figure 1 shows the high-level logical units of the Bugxter application. The primary unit is "Bugs" which handles the creation and update of bug entries in a relational database. "MyBugxter" offers a personal view of the bug database - bugs that are assigned or owned by you. "Admin" and "Products" enable the setting up and maintenance of the application. Other areas cover security including authorisation, membership, search and statistics.

The core components can be categorized into three distinct functional groups - the main "Public Application" are the common areas accessible to all users; this includes bug reporting, viewing and search as well as general statistics. "MyBugxter" is a personal area presenting information unique to a given user. "Admin" is restricted to users with admin privileges.

The Web-application URLs reflect the functional zones.

The zones are depicted in figure 1 as sectors of a circle - the circle represents the entire URL address space of the application.

Lastly, in addition to the core application there is an external site, which provides the login/splash page, membership sign-up and privacy policy. This is located in /bugxter/ For clarity this is not depicted.

Bugxter is an access controlled application. It allows guests to login and view the system, it presents user specific views in MyBugxter, and only allows admin users to access the admin interface.

Bugxter uses a server-side session to hold a session token - the session token is created when the user logs in. The session token is a credential which demonstrates that the issuer of the request has previously established trust (logged in). A session token is also a proxy for the users identity - the token is used in lieu of user-name as the key to look-up a users personal information when it is needed in a given application context. We will show below how the access control and session management is achieved as an overlay above the application.

Page-based Web-Applications

The standard approach for developing web-applications is to treat each page of the application as a self contained unit of the application. PHP, JSP, ASP mix dynamic content with a static HTML template - when a page is accessed, the active procedural code is executed and the static template is populated.

Generally for large-scale Web-applications the content is a representation of a query made on a back-end database. Some simple applications might make the database call directly in the page. More sophisticated applications use a so-called '3-tier architecture' where the front-end active page invokes a middleware layer which may or may not offer transactions, data persistence and other bells n whistles. The middleware layer interacts with the back-end database.

The page-based approach has, with varying degrees of sophistication, characterised the general design-pattern of most web-applications to date.

Numerous frameworks have been created that provide utility services to a page - for example access control, application flow, session management etc. Whilst these are helpful the fundamental property of a page-based approach to web-applications remains - this approach creates a tight binding between the presentational content of a page and the logic which generates the data.

As we proceed we will demonstrate that Web-applications on NetKernel offer a new and flexible decoupling between presentation and logic - we characterize this as Service-based Composition.

NetKernel Service Composition

NetKernel enables functional units of an application to be held in modules. A module exports a public URI interface and internally hosts a virtual URI address space. The internal address space consists of local resources, both static content and active services, and may also include the public address spaces of other imported modules. Therefore a module can be thought of as an encapsulated virtual private web-space.

The development of a NetKernel web-application is principally concerned with defining the services which exist in the internal address space, defining the public URI interface and mapping requests on the public interface to requests on the internal URI address space. Modular application service components are layered to create the overall application.

This probably sounds complicated but in practice some simple patterns make it straightforward and produces applications that are very flexible, robust and extremely adaptive to changes.

Rather than continue with general discussion of the NetKernel abstraction we'll illustrate by discussing the specifics of the Bugxter application.

Bugxter Modules

Bugxter consists of 2 modules - "app_bugxter_services" provides the core functional units of the application, described above. The bugxter services module interacts with the back-end database. "app_bugxter_web" provides a web-front end to the bugxter services - it is logically separated into presentation and control. The controls invoke the bugxter services and then apply styles to any resulting data to produce presentable XHTML.

The combination of the two modules results in a web-application with a clean Model-View-Control pattern. Before we get to the details of how the web-control interface is defined we will introduce some additional pre-application layers.

Service Layers

The upper part of Figure 2 shows the three functional zones of the Bugxter application which we showed earlier are zones in the public URL interface of the Web-application.

Figure 2: Bugxter modules and layers

On the right of the figure we show the two Bugxter modules and the front-end fulcrum. Shown below each module's name is the set of modules which it imports. Starting lowest first we see that the bugxter services module is imported by the bugxter web-front-end which in turn is imported by the fulcrum front-end at the top of the stack.

The Fulcrum is a module hosting the HTTP transport and the app_blogxter_web module. HTTP requests arriving on the HTTP Transport are issued as NetKernel requests into the internal address space of the Fulcrum module. Modules which are imported into the Fulcrum may match these requests and the request will be issued to that module. The Front-end Fulcrum provided with NetKernel Standard Edition rewrites http: requests into the NetKernel ffcpl: scheme.

The app_bugxter_web module exports a public interface ffcpl:/bugxter/.* - which means it captures all requests in the bugxter/ path. So a request on http://host:port/bugxter/ is rewritten to ffcpl:/bugxter/ when it arrives in the Fulcrum. This request is then captured by the app_bugxter_web module for processing.

Returning to the top of the figure - we can think of the public Web URL address space as a plan view onto a set of layered URI address spaces created from the imports and mappings of the modules. A request on the upper URI address space initiates a process as a cascade of requests against lower layers.

In this case the Session layer exchanges an external cookie for an internal session before handing on the request to the GateKeeper. The GateKeeper validates that the requestor has the required privileges for the request URI by calling a validation service provided from the layer below, if the validation succeeds the request is issued against the Bugxter web front-end interface which in turn issues requests against the Bugxter core services.

Looks complicated but in practice the Session and GateKeeper layers are effectively transparent and in general we don't need to think about the whole stack in one as shown here - we think about a single module's public interface and the internal services which service requests on that interface.

The simple way to think of the Bugxter URI stack is that the Fulcrum issues requests against the Bugxter Front-end, which processes a request by issuing requests to the Bugxter core services.

XRL

Figure 3: Bugxter mapper layers

Both the web-front-end and the core services of Bugxter use an XRL mapper layer to decouple requests on the public interface from their internal service implementations. In figure 2 we deliberately omitted these hidden layers from the diagram they are shown in Figure 3.

In this section we will briefly explain the principles behind XRL and illustrate with an example showing how the Bugxter main page is recursively composed.

All requests arriving in the bugxter web-front-end module are passed to the XRL mapper accessor by the following rewrite rule

The XRL mapper is a service which maps requests on an external interface to a request on the internal address space. The mappings are specified in a links document - in the web-front-end the links document is ffcpl:/links.xml.

Here's an example link for the home page of the application.

The elements of a link are as follows

name - the name of the link. Links are always named. The name may be referenced by using the xrl: URI scheme. Named links are powerful since they provide a decoupling between logical (xrl) and public (URL) addresses. In practice we use xrl:xxxxx for internal resource references and let the XRL infrastructure perform an outbound link translation to the public URL. In this example the name is "main_home" - we can create an href anchor link to the home page with href="xrl:main_home".

ext - this is the public or external URI path. The links document may specify a URI basepath for external URIs, in this application the basepath is ffcpl:/bugxter/ so this path '/main/' is for requests on ffcpl:/bugxter/main/ which is the root path for the main bugxter application.

int - this is the internal URI to request. In this case it is a request to the xrl runtime (active:xrl-html-tolerant is the xrl runtime with html output and error tolerance). XRL is an XML pull runtime. Starting from a template document XRL recursively includes other XML resources to build a complete document. In this case the template is referred to by a named xrl: link xrl:template_main, which if you look in the links document is the name for the internal URI ffcpl:/www/view/templates/main.xml. This template will receive content ffcpl:/www.view/main/home.xml. In a moment we will show the details of this request.

args - the arguments to pass through from the external request to the internal request. Arguments are the named URI parts of an active URI - all internal NetKernel requests are active URIs. In this case the links, param and session arguments are passed through on the internal request - ie these arguments are appended to the internal URI request. If in doubt about what arguments are being passed for any given request you can set a breakpoint in the debugger to inspect them either by name or value.

Example: Bugxter Main Page Composition

This might seem like a lot to take in but it should be a lot clearer with an example...

Above we described how a request for ffcpl:/bugxter/main/ is mapped by the mapper to a request for the XRL runtime to process a specific template document and an associated content document. Figure 5 (below left) shows the Bugxter main page which is generated as a result of this mapped request. The right-hand side of the figure shows how the page is composed. The highlighted colours show the components of the page [green - the template document, yellow - the content document, red - service included into the content, blue - a form whose action URL is dynamically substituted by the mapper layer.

Figure 4: Bugxter main page and composition

You can examine the XML of both the template (ffcpl:/www/view/templates/main.xml) and the content (ffcpl:/www.view/main/home.xml). You will see that the template declares an <xrl:include href="xrl:content"/> which the XRL runtime replaces with the content document. The content document contains two further xrl:include declarations for xrl:info_global and xrl:info_latest. These invoke DPML processes to generate the two dynamic tables in the page - take a look at the links document to see the active:dpml... internal requests.

Figure 5 (below) shows the recursive composition of the main page. The template pulls in the content which pulls in two services presenting the global statistics and latest bug reports respectively. Finally the XRL runtime processes all xrl:resolve attributes - these indicate parsable attributes which require link substitution - in this example the content has a form (shown in blue) with an action="[[xrl:edit_bug]]" attribute - this is replaced by the public URL specified in the edit_bug named link. The dynamic link substitution of XRL means that we generally only need to consider the logic name of a link - we don't need to worry about the public URL, unless we want to relocate it which is simply a matter of changing the ext declaration for the link.

Figure 5: Bugxter main page XRL recursion

Let's examine in detail how the latest bugs component is created (Figure 6, below). As the main page is built the XRL recursion process discovers the <xrl:include href="xrl:info_latest"/> include declaration in the content document. Using the links file it locates the named link info_latest and invokes the corresponding internal URI active:dpml+operand@ffcpl:/www/control/main/info/infoLatest.idoc This starts the execution of the DPML process infoLatest.idoc.

Figure 6: Generation of the Latest Bugs Table

The DPML process issues a URI request onto the core-services interface for the latest bug data. The core-service request goes through another XRL mapping layer which we don't need to worry about (the advantage of the layered URI model is that composable services can be treated as black boxes). The service request results in a query of the back-end RDBMS database for the recent bug data which is returned from the core-service layer as an XML response. The DPML process formats the XML response into an XHTML table by applying the XSLT stylesheet ffcpl:/www/view/main/info/style_latest.xsl

Dynamic Services

The example has shown how XRL is used to build composite documents. What's not demonstrated in this example is how arguments (URL queries, POST data ...) may be recursively passed down to any included service - each xrl:include is really a service interface which may process the arguments specified in the args declaration of the links. If you inspect another page (such as xrl:process_search which has public URL /main/process_search ) you will see how the included content is generated using the arguments.

XRL Tips

• Each local step of XRL processing is relatively simple. Complexity is built-up by composing simple components together. All components are loosely coupled and extremely flexible both to create and maintain.
• Don't try to work through the whole recursion in your head! Instead start from both ends and work to the middle - create a master template then create many small components - build up the components layer by layer until you have what you want in the template.
• Don't think of xrl:include as included documents - think of them as service requests which produce XML.
• Static arguments can be issued (as a param URI argument) to an xrl:include service by placing an XML fragment within the xrl:include element.
• You can use xrl:eval to replicate the content of the generated document - this is a good way to dynamically generate page titles.
• Link substitution works just as well for external URLs - for example name a link "1060" which has <ext>http://www.1060.org/</ext> now you can use xrl:1060 anywhere in your site and the link will be replaced for the full URL.
• Don't confuse the xrl engine (active:xrl) with the xrl mapper (active:mapper) accessor -they are two separate tools related by using the same links document structure. Mapper does inbound/outbound address mapping. XRL is the XML pull runtime and external link resolver.
• Take care that a dynamically substituted URL exists for things like form actions and redirects!
• Named links are great for images or any static resource. If you use a link name in an <img src="..." > or an <a href="..." > make sure to tell xrl to resolve them with xrl:resolve="src" and xrl:resolve="href" respectively. You can also use named link resolving in CSS style declarations - for example for background images etc.
• The mapper can be used to manage a modules URI interface in a more dynamic way than rewrite rules - see the Core-Service module for an example of using mapper to manage a large set of public service interfaces.
• 500 and 404 are special named links which will be invoked by the mapper if the internal URI request fails or the URI interface is not found. You can create an application specific process to handle these errors.
• You can easily create a static version of your site by using the active:mapper-export utility accessor. The NetKernel static documentation is generated using this technique.
• The XRL reference guide has details of syntax and use

Caching

In NetKernel all resources are generated from URI requests - the URI that generated the request is the key used by the cache for the resultant resource. Provided the depdendent resources from which a resource is generated remain valid a request for the same URI will come straight from cache. But unfortunately you'll have to come back for the next installment...

PJR 6/4/2004 - Hopefully this incomplete article is interesting as it stands. There's more to come including layering services for cacheability, using the Golden thread pattern for automatic database query invalidation, details of zoned access control... Coming real soon...