NetKernel Musings
NetKernel Musings
Technical, Business and Practical thoughts about NetKernel
Page123
1 - 5 of 13
3Apr
Thu2008
Development process tip

NetKernel 3.3.1 includes a new feature in the Resource Request Trace Tool which, combined with a simple configuration, provides an easy way to build and verify portions of an application.

The Resource Request Tool is available from the Developer tab in the NetKernel management interface. This tool allows you to specify a resource address and to make a request with the source verb delivered to either to the module's external interface or the inner private address space.

You may choose the "Resolve" button, which only performs the address resolution phase of request processing, to confirm that the module address mappings are set up correctly. Or you may choose the "Resolve and Execute" button to also cause the resolved-to Endpoint to receive the request. New in NetKernel 3.3.1 is the Display Representation button which will display in the browser the representation returned for the request.

The tip is to set up a <rewrite> rule in the <mapping> section of your module's module.xml file to map the address ffcpl:/test to the portion of the application or the service you are developing. For example:

<rewrite>
  <match>ffcpl:/test</match>
  <to>active:dpml+operand@ffcpl:/resources/programs/myProg.idoc</to>
</rewrite>

Then you simply request the URI ffcpl:/test and select the "inner" option and your request will be mapped to the program you are working on. You can quickly verify that the mappings are correct, run the program and see the result without setting up a separate path from a browser through a fulcrum, the modules export section and rewrite sections.

In a refinement of this approach you could use a mapper with a links.xml resource to define the mapping between ffcpl:/test and the code you are developing. By using a mapper you avoid the need to hot-restart your module each time you make a change and you could develop a family of test mappings such as ffcpl:/test/program1, ffcpl:/test/program2 if you are developing a set of capabilities.

25Dec
Tue2007
Using Scala with NetKernel

Scala is a relatively new language that incorporates features of object-oriented and functional programming into a strongly typed, fully compiled language that has the "feel" of a scripting language. Scala compiles to Java byte codes and the language is designed to integrate with Java programs and libraries.

Scala incorporates a lot of features that could be useful for developers writing extensions to NetKernel and NetKernel applications. To start my investigation of both the language and its suitability for work with NetKernel I started with a simple accessor. This posting describes how I got Scala to run on OS X and work with NetKernel 3.3.

First, the Scala code. The following code implements a simple accessor in Scala.

import org.ten60.netkernel.layer1.nkf.impl.NKFAccessorImpl
import org.ten60.netkernel.layer1.nkf.INKFConvenienceHelper
import org.ten60.netkernel.layer1.nkf.INKFRequestReadOnly
import org.ten60.netkernel.layer1.representation.StringAspect

class HelloWorld extends NKFAccessorImpl(NKFAccessorImpl.NOT_SAFE_FOR_CONCURRENT_USE, INKFRequestReadOnly.RQT_SOURCE) 
  {

  override def processRequest(context: INKFConvenienceHelper)
  { val message  = "Hello World from Scala"
    val sa       = new StringAspect(message)
    val response = context.createResponseFrom(sa)
    response.setMimeType("text/plain")
    context.setResponse(response)
  }
}

Scala import statements look and operate like Java import statements. Scala classes do not need explicit constructors, the line starting with class defines the class name and indicates it extends the Java class NKFAccessorImpl and calls the superclass constructor with two parameters.

Overrides of methods must start with the keyword override. In Scala the combination of functional and imperative styles can be see when variables are declared. A var indicates a mutable variable for imperative programming while a val indicates an immutable value seen in functional programming. It is natural to use val for NetKernel code as a fundamental tenet of resource oriented computing is that all responses contain immutable representations.

Scala is a fully compiled, type-checked language - however notice that I do not need to state the type of each of the val objects (message, sa and response). This is due to the Scala compiler's ability to infer types. Scala was designed to minimize the amount of code typed and to make it have the "look-and-feel" of a scripting language.

Scala also supports some level automatic type conversion. While I have not yet investigated it, I believe this feature would allow me to reduce the accessor method code to:

override def processRequest(context: INKFConvenienceHelper)
  { val response = context.createResponseFrom("Hello World from Scala")
    response.setMimeType("text/plain")
    context.setResponse(response)
  }

As I explore Scala I'll investigate this feature and report on it and other discoveries I make.

Getting Scala to run and compile on OS X is straight forward. I run OS X Leopard on a PowerBook G4 but I would imagine Intel Macs would work just the same. I downloaded Scala as an IzPack Java installer, ran the installer and placed Scala at ~/scala. Next I wrote the accessor in a file HelloWorld.scala in the root directory of a module and compiled it with the following from the command line.

~/scala/bin/fsc  -classpath .:[install]/modules/ext-layer1-1.3.4.jar:[install]/1060netkernel-2.8.4.jar  HelloWorld.scala

Note: "[install]" should be replaced by the path to the directory on your machine where you installed NetKernel and the JAR file names are for NetKernel 3.3.

I used the fast Scala compiler (fsc) but the regular Scala compiler (scalac) could be used. The fast Scala compiler installs an instance of itself in memory and listens on a port for compile requests. The regular Scala compiler is slower because it starts up a JRE instance each time it is called.

The compiler creates the file HelloWorld.class that contains Java byte codes. At this point integrating the Scala HelloWorld accessor with NetKernel is just like integrating a compiled Java accessor. Map a logical address to the class with an entry such as this one from the mapping section of my module's module.xml file:

<ura>
  <match>ffcpl:/helloworld</match>
  <class>HelloWorld</class>
</ura>
11Dec
Tue2007
It's like a browser ...

One of the best ways to explain how NetKernel works is to compare a NetKernel Accessor with a browser operating in the World Wide Web.

Browser ABCs ...

The service provided by a web browser is the (usually visual) presentation of a resource located in the World Wide Web. We use web browsers so often we usually don't take time to ponder how they perform their service. It is instructive however, as a browser functions in basically the same way as an accessor does in NetKernel.

Fundamentally, a browser accepts a request for a URL, retrieves that resource, performs a page-flow layout and presents the visual representation to a user. As we examine these steps we'll also see reasons why the World Wide Web is flexible and scalable.

First, consider the nature of a resource. When we request a resource identified by a URL such as "http://www.google.com" we are specifying the address for a resource within the World Wide Web address space. Resources are used to model abstract information. For example, the resource "http://www.1060research.com" is an abstract notion - the "home page" for the 1060 Research web site. It may seem strange to think about this, but the home page doesn't actually exist until it is requested. The act of requesting "http://www.1060research.com" causes the endpoint responsible for it to create and return a physical representation. All representations are immutable - they cannot be changed, they can only expire and become invalid. Architecturally, immutability is wonderful for scaling. In the World Wide Web immutable representations can be stored in various locations to speed up processing such as in a browser or a proxy server or cache. As long as the representation has not expired, it can be provided by any of these caches, saving a trip to and work by the endpoint.

Second, consider what a browser does when it receives an HTML encoded resource representation. The browser interprets the HTML codes and if it finds resource references (such as for CSS, image, sound or other resources), it must issue sub-requests for them before it can perform its service. Once it has gathered all the referenced resource representations it can perform the page-flow layout work and build a visual representation. (Notice that a browser is free to issue the sub-requests asynchronously thereby creating parallel processing requests)

Third, consider the mapping of the logical address to an endpoint. For each request, the browser relies on the Domain Name Service (DNS) to map a logical address to an endpoint. While DNS is heavily cached and uses reasonably long expiry times there is no fundamental reason why an IP address couldn't be different for each request. In fact, IP round-robin assignment is a technique used to distributed requests across several endpoint IP addresses. For very large sites an IP load balancer is used to map a single IP address to one or potentially thousands of servers. In these horizontally scaled designs there is no notion of threads - all that a user and browser sees are requests.

A browser working with the World Wide Web is an amazingly flexible system. All addresses are logical and are bound to an endpoint for each request. Because of this indirection, web sites can appear and disappear, expand or contract at any time. Web servers can be upgraded or changed to different technologies and there is no impact on browsers - there is no physical coupling. Processing can occur in parallel without requiring knowledge of threads and immutable representations can be cached increasing the responsiveness of the whole system.

These are great properties! What if we could use this approach to develop software itself? Would we see the same properties from the web evident in software?

Turns out, we can ... and we do ...

NetKernel ABCs ...

In NetKernel all information is modeled as resources and resources are identified by URI addresses.

When a resource is requested, the address is resolved to an endpoint known as an Accessor. Like the browser, an accessor returns a representation of a resource. Also like a browser, an accessor may interpret codes and request additional resources. Unlike a browser, an accessor can return any form of representation. You can think of an accessor in NetKernel as a browser without the visual presentation surface.

Again, let's consider resources. In NetKernel resources represent abstract information such as "the list of customers" or "currently enrolled students". These resources are identified by URI addresses such as "ffcpl:/resources/customers" and "ffcpl:/enrolled_students". When a resource is requested, an endpoint returns the current state of that resource as a physical, immutable resource representation. A resource may be fundamental and atomic or it may be composed of other resources. For example, an application that assigns students to courses would need to compose a presentation page from the "list of courses" resource and the "list of students" resource.

Processing Steps

When an endpoint accessor receives a request, it examines the request to determine the context and specifics of the request. It can examine the requested URI and the names and URI addresses of each parameter. The following table illustrates the processing steps taken by a browser and a NetKernel accessor upon receiving a request.

Browser NetKernel
Examine Request Browser parses the URL entered by the user. Accessor examines URI and parameters.
Determine if sub-requests are required. Browser always makes at least one sub-request. Most accessors make sub-requests.
Issue sub-request(s)
  1. Browser parses URL to obtain domain name.
  2. Queries DNS with domain name.
    1. Send DNS query with domain name.
    2. DNS resolves domain name to IP address.
    3. DNS returns IP address for endpoint to browser.
  3. Browser issues sub-request to endpoint.
  4. Endpoint returns representation to browser.
  1. Sub-request created with URI address and parameters.
  2. Sub-request issued to microkernel in current address space.
    1. Microkernel resolves to endpoint.
    2. Microkernel issues request to endpoint.
    3. Endpoint returns representation.
  3. Microkernel returns representation to Accessor.
Create representation Browser performs a page-flow layout using all returned resource representations. Accessor performs its service and creates a representation.
Return representation Browser displays a visual representation of the resource. Accessor creates a response referencing a representation and returns it to microkernel.

The only significant difference between a browser and an accessor in NetKernel is the way endpoint resolution is accomplished. A browser uses DNS to locate an endpoint in a single, global address space while an accessor relies on NetKernel's microkernel to resolve an address in a set of related address spaces.

It is important to note what is not occurring - the NetKernel accessor is not referencing physically linked objects, data or code. If it requires additional information it issues a sub-request for a logically identified resources back into the logical address space. This is significant and extremely important.

The following code illustrates a BeanShell script that requests the students and course information:

void main() {
  // Request students and courses resources
  repStudents = context.source("ffcpl:/enrolled_students");
  repCourses  = context.source("ffcpl:/2008/spring/courses");

  ...
}

Because information is modeled as resources and resource addresses are bound to an endpoint only at the moment of each request, NetKernel applications take on the properties of the World Wide Web:

  • Flexibility - Resource endpoints can be mapped to different endpoint implementation for each request. This means that operational systems can be modified, upgraded and down graded at any time while handling requests.
  • Scaling - Requests know nothing of threads. The microkernel is free to asynchronously schedule requests from its carefully managed pool of threads onto any available CPU core. As cores are added, performance increasing almost linearly.
  • Generational growth - clients and endpoints can evolve independently of each other. Because they are logically coupled, a change in either does not require code recompilation of the entire system.

These and other properties accrue to NetKernel systems. The idea of building information systems at the logical level of information instead of the physical level of objects and data has been validated by the World Wide Web as well as the numerous applications built and running on NetKernel.

20Oct
Sat2007
Thoughts about NetKernel release 3.3

NetKernel 3.3 will be released soon. The changes reflect input from our user community and our focus on making it easier for Java (and other) developers to learn about resource oriented computing. I will comment on some of these changes.

Based on input from users we have modified the HTTP transport to support if-modified-since and integrated the support with the internal cache. The result has been a significant if not dramatic improvement in performance of Web applications such as the 1060 Research forum. If you investigate the performance of the forum keep in mind it is running on a virtual server running Xen on a 2 Ghz machine. The NetKernel installation is using 64 Meg of Heap and runs the forum, blogxter, and bugxter. If you have experience with Java J2EE servers I think you'll appreciate just how lean and fast NetKernel has become.

Ruby is now an officially supported language in NetKernel. We have included the JRuby 1.0.1 release and we are dedicated to tracking updates an including them as we do with other languages. We also have a new language integrated with NetKernel in experimental form - we will make an announcement about that soon.

For me the most significant set of changes are based on our experience working on the NetKernel 4.0 release. While not yet released NetKernel 4 is impacting the NetKernel community in a number of ways. The two most significant are the new Request Visualizer Tool and the changes we have made to the documentation.

The Request Visualizer allows you to capture root requests (requests injected by a transport) and all sub-requests spawned during processing. Once captured, the Visualizer tool displays root requests in a folding tree structure along with all information about each request. The beauty of this is that for the first time you have easy access to all of the details of a request in one easy-to-use display. You can run a request multiple times and see how subsequent requests draw much if not all of the resource representations from cache. If resources are not being cached, you can see that immediately and then drill down to determine why.

Our work on NetKernel 4 has also impacted the documentation for NetKernel 3. We have learned to explain resource oriented computing and the core NetKernel abstractions using much clearer and more consistent language. We have also had the opportunity to speak with more people about NetKernel and in doing so we have learned how to better articulate concepts.

In the Getting Started book we now introduce NetKernel starting at the physical level and work upwards to the logical level. (In the past we started somewhere in the middle). After we reach the logical level we turn around and head down towards the physical level again. I tried this approach when I presented NetKernel to the Phoenix Java User Group recently and it seemed to work well.

We also reorganized the documentation and added two books - one that focuses on resource models and the other on client, server and general services tools. The Tutorial book has been reworked to make the examples more consistent both with each other and in the way we are explaining resource oriented computing.

NetKernel keeps getting better. NetKernel 3.3 will be our best release and as we look forward to NetKernel 4 we can say that NetKernel will become even more powerful and simpler.

I want to thank everyone in the NetKernel community who provided feedback and suggestions for improvements. There are a number of exciting projects using NetKernel that will be released soon. We have worked closely with these teams and suggestions they made have resulted in new features and capabilities in NetKernel.

9Jul
Mon2007
Project Zero - a second look

Today I'll take a look at IBM's Project Zero ("Zero") in more detail and compare what I find to NetKernel.

Zero is described as an event-based system that passes state outside of the event message. This means that event messages do not have parameters and event handlers request all state for each invocation. Zero uses a concept called a zone to contain state. The documentation lists the App, Request, User, and Event zones. The App zone contains information valid for the life of the application. The Request zone is visible to the thread processing an HTTP request and is valid up until the response is sent back to the client. The User zone is visible to all threads processing requests for a specific user's session. The Event zone is visible only to the thread dispatched to an event handler and is valid until the event handler returns.

In Zero, code can access state within these zones using a URI based hierarchical naming system. In Java code the GlobalContext is used to get and put information. For example:

GlobalContext.get("/request/uri")

will retrieve the original HTTP request URI and

GlobalContext.put("/request/status",200)

sets the return status. In Zero, the first part of the path is a key into each of the zones - these examples are using the request zone. New zones can be created by writing a ZoneHandler. The second part of the path is the sub-context which provides a limited set of operations such as dump() and a keys() operation.

Zero provides a mechanism that parses HTTP request and provides the information in a normalized URI address space within the request zone. Here are some example URIs that are supported:

/request/protocol
/request/scheme
/request/serverPort
/request/params

A full listing is available in the Zero documentation.

Events in Zero are handled by event handlers. Event handlers are registered in the configuration information and are associated with events that can occur in the system. For example, a request for the (partial) URL /foo/Hello.groovy could be registered as:

[/app/handlers/GET[]]
handler=acme.handler.GetHandler.class
conditions=[/request/path matches /foo(/.*)?]

Handlers then implement a set of standard method signatures such as onGet(), onPOST(), onPUT(), onDELETE(), etc. Zero supports a standard set of events and developers can add new ones. Events are fired using static methods on the EventEngine:

public class EventEngine {
 public static EventContext fire(String eventName, Map eventData) {...}
 public static EventContext fire(String eventName, Map eventData, EventDispatcher dispatcher) {...}

There is much more to Zero, but I will wait and explore more later. Next I want to compare this much of Zero with NetKernel.

Comparison with NetKernel

The opening description for Zero in its documentation establishes a key difference between Zero and NetKernel. Zero is an event-based system and NetKernel is a resource-oriented computing system. Yes, there is a concept of resources in Zero, but they play a supporting role whereas in NetKernel it is the central concept.

Resources

Resources in Zero are defined within the context of a virtual directory system. Each resource uses a virtual directory and is mapped into the /rest/... sub-space of the global context. For example, if people is a resource, then /rest/people is the location of the resource and the collection.groovy and item.groovy scripts within the virtual directory handle the aggregate and individual "items". The collection script is responsible for list and member creation while the item script is responsible for retrieve, update, and delete operations. This approach to resources has a distinct CRUD (Create, Retrieve, Update, Delete) feel to it, similar to Ruby on Rails and other result-set centric database-grounded frameworks.

Resources in NetKernel are very different. They are abstract, typeless and take on any form required by a system. Resources in NetKernel are fundamental and pervasive; they can be transformed by services and composed into more complex resources. For example, the result of using the sqlQuery service in NetKernel is a resource that can be transformed using the xslt service into HTML. Resources in NetKernel can flow through a chain of services (just like chaining the Unix toolkit- cat, grep, awk, etc.) to create the desired final information representation. Applications written in NetKernel focus on the necessary information model (and the supporting URI address structures), the services required, plumbing and routing and finally the application of constraints.

While Zero provides surface level support for resources it does not seem to be a deeply implemented concept. For example, I have not yet found how one would build services which take resources as parameters and provide resources as the result such as can be done in NetKernel with this URI:

active:xslt+operator@ffcpl:/style.xsl+operand@ffcpl:/data.xml

It is not clear if Zero could support a system-wide resource cache like NetKernel's.

Address Spaces and Context

Zero stipulates that event-handlers are stateless and all state exists globally with zones. Each zone has a different operational context (such as being related to the application lifetime or the user's session) and new zones can be created by the developer.

In NetKernel the location of state in system is up to the architect. Stateless or stateful services can be created to suit system requirements. However, stateless services is the preferred design approach and state can then be passed in parameters or through environmental contexts.

While Zero has a single global context, NetKernel uses a general idea of address spaces. NetKernel can support an unlimited number of address spaces and within and between each address space, mappings can be defined to perform address transformations. The ability to create multiple address spaces and to establish relationships between them is a very powerful capability that leads to high-level architectural patterns. For example, it is very easy in NetKernel to wrap an address space with a security gatekeeper or a session manager - adding new application capabilities in an AOP like manner.

Events

Events and event handlers are core elements of Zero's structure. NetKernel is a request-response system in which a request for a resource results in a resource-representation in the response. At first glance it may seem the two are similar. However, NetKernel implements a computational model while Zero does not. In NetKernel the resource request is the operation code (op-code) of the model. This has important implications.

One implication is the use of an engine - implemented as an operating-system caliber micro-kernel - to execute the URI op-codes. An execution engine provides a clear boundary between the logical computation model using resources and the physical implementation details. By hiding the details not only is the model pristine, the implementation details can be changed without impacting existing programs. This is not the case in an event / event handler design in which programs are written to a physical level API. Any subsequent changes to the implementation API will necessitate coding changes in applications.

A second implication derives from the textual form of the op-codes. In NetKernel the op-codes are literally a sequence of ASCII characters. For example the op-code "ffcpl:/index.html" is presented as the string of characters: 'f' 'f' 'c' 'p' 'l' ':' '/' 'i' 'n' 'd' 'e' 'x' '.' 'h' 't' 'm' 'l' to the engine. Because the op-codes are strings, they can be manipulated within the logical model. The execution engine supports pattern matching against op-codes and substitutions via re-write mapping rules. For example, a mapping rule can change all op-codes of the form:

ffcpl:/stock/price/closing/(.*)/(.*)

to

active:fetchClosingPrice+date@$1+symbol@$2

which would transform the op-code "ffcpl:/stock/price/closing/2007-06-15/IBM" to "active:fetchClosingPrice+date@2007-06-15+symbol@IBM". Another mapping from

(active:.*)

to

active:gk+function@$1

routes all service requests through a security gatekeeper service. This is model-level aspect-oriented programming, layering, and more all within an execution model that can be dynamically modified at run time.

This extraordinary flexibility and control - a dream of object-oriented developers - is possible within a simple, consistent execution model. Unfortunately, Zero only provides resources at the surface level and is hence constrained in what it can offer.

Conclusions

There is more to examine before reaching any final conclusions about Zero, and I may have missed some points about the design in this analysis. However, it is clear that the fundamental premise for Zero and NetKernel are very different. Zero is an event-based system with a simple and clean configuration and points of extensibility. NetKernel is a logical computing model that uses resources to model information. All development in NetKernel is focused on building, using, and constraining compositions of resources to provide the information requested by the user.

Ultimately, all computer systems no matter how designed and built, must provide requisite services and information. When view that way, NetKernel and the resource-oriented computing model seems fresh, direct, and empowering. Zero seems grounded in a procedural, physical-world way of thinking.