NetKernel 4 development is progressing well and the series of monthly "private preview" releases will wrap up with the preview 5 release due out in about a week.

The current series of preview releases has been private only because we knew the APIs were subject to change as we gained experience with the new, more flexible abstraction and incorporated feedback.

We anticipate a final major refactoring to occur in mid-April and when completed, NetKernel will finally be ready for public previews.

While I have not published a posting to this blog in quite a while, that will change as we start the public preview program for NetKernel 4. For very up-to-the-minute information about NetKernel follow my Twitter micro-blog at http://twitter.com/rsk1060

NetKernel 3.3.1 includes a new feature in the Resource Request Trace Tool which, combined with a simple configuration, provides an easy way to build and verify portions of an application.

The Resource Request Tool is available from the Developer tab in the NetKernel management interface. This tool allows you to specify a resource address and to make a request with the source verb delivered to either to the module's external interface or the inner private address space.

You may choose the "Resolve" button, which only performs the address resolution phase of request processing, to confirm that the module address mappings are set up correctly. Or you may choose the "Resolve and Execute" button to also cause the resolved-to Endpoint to receive the request. New in NetKernel 3.3.1 is the Display Representation button which will display in the browser the representation returned for the request.

The tip is to set up a <rewrite> rule in the <mapping> section of your module's module.xml file to map the address ffcpl:/test to the portion of the application or the service you are developing. For example:

<rewrite>
  <match>ffcpl:/test</match>
  <to>active:dpml+operand@ffcpl:/resources/programs/myProg.idoc</to>
</rewrite>

Then you simply request the URI ffcpl:/test and select the "inner" option and your request will be mapped to the program you are working on. You can quickly verify that the mappings are correct, run the program and see the result without setting up a separate path from a browser through a fulcrum, the modules export section and rewrite sections.

In a refinement of this approach you could use a mapper with a links.xml resource to define the mapping between ffcpl:/test and the code you are developing. By using a mapper you avoid the need to hot-restart your module each time you make a change and you could develop a family of test mappings such as ffcpl:/test/program1, ffcpl:/test/program2 if you are developing a set of capabilities.

Scala is a relatively new language that incorporates features of object-oriented and functional programming into a strongly typed, fully compiled language that has the "feel" of a scripting language. Scala compiles to Java byte codes and the language is designed to integrate with Java programs and libraries.

Scala incorporates a lot of features that could be useful for developers writing extensions to NetKernel and NetKernel applications. To start my investigation of both the language and its suitability for work with NetKernel I started with a simple accessor. This posting describes how I got Scala to run on OS X and work with NetKernel 3.3.

First, the Scala code. The following code implements a simple accessor in Scala.

import org.ten60.netkernel.layer1.nkf.impl.NKFAccessorImpl
import org.ten60.netkernel.layer1.nkf.INKFConvenienceHelper
import org.ten60.netkernel.layer1.nkf.INKFRequestReadOnly
import org.ten60.netkernel.layer1.representation.StringAspect

class HelloWorld extends NKFAccessorImpl(NKFAccessorImpl.NOT_SAFE_FOR_CONCURRENT_USE, INKFRequestReadOnly.RQT_SOURCE) 
  {

  override def processRequest(context: INKFConvenienceHelper)
  { val message  = "Hello World from Scala"
    val sa       = new StringAspect(message)
    val response = context.createResponseFrom(sa)
    response.setMimeType("text/plain")
    context.setResponse(response)
  }
}

Scala import statements look and operate like Java import statements. Scala classes do not need explicit constructors, the line starting with class defines the class name and indicates it extends the Java class NKFAccessorImpl and calls the superclass constructor with two parameters.

Overrides of methods must start with the keyword override. In Scala the combination of functional and imperative styles can be see when variables are declared. A var indicates a mutable variable for imperative programming while a val indicates an immutable value seen in functional programming. It is natural to use val for NetKernel code as a fundamental tenet of resource oriented computing is that all responses contain immutable representations.

Scala is a fully compiled, type-checked language - however notice that I do not need to state the type of each of the val objects (message, sa and response). This is due to the Scala compiler's ability to infer types. Scala was designed to minimize the amount of code typed and to make it have the "look-and-feel" of a scripting language.

Scala also supports some level automatic type conversion. While I have not yet investigated it, I believe this feature would allow me to reduce the accessor method code to:

override def processRequest(context: INKFConvenienceHelper)
  { val response = context.createResponseFrom("Hello World from Scala")
    response.setMimeType("text/plain")
    context.setResponse(response)
  }

As I explore Scala I'll investigate this feature and report on it and other discoveries I make.

Getting Scala to run and compile on OS X is straight forward. I run OS X Leopard on a PowerBook G4 but I would imagine Intel Macs would work just the same. I downloaded Scala as an IzPack Java installer, ran the installer and placed Scala at ~/scala. Next I wrote the accessor in a file HelloWorld.scala in the root directory of a module and compiled it with the following from the command line.

~/scala/bin/fsc  -classpath .:[install]/modules/ext-layer1-1.3.4.jar:[install]/1060netkernel-2.8.4.jar  HelloWorld.scala

Note: "[install]" should be replaced by the path to the directory on your machine where you installed NetKernel and the JAR file names are for NetKernel 3.3.

I used the fast Scala compiler (fsc) but the regular Scala compiler (scalac) could be used. The fast Scala compiler installs an instance of itself in memory and listens on a port for compile requests. The regular Scala compiler is slower because it starts up a JRE instance each time it is called.

The compiler creates the file HelloWorld.class that contains Java byte codes. At this point integrating the Scala HelloWorld accessor with NetKernel is just like integrating a compiled Java accessor. Map a logical address to the class with an entry such as this one from the mapping section of my module's module.xml file:

<ura>
  <match>ffcpl:/helloworld</match>
  <class>HelloWorld</class>
</ura>

One of the best ways to explain how NetKernel works is to compare a NetKernel Accessor with a browser operating in the World Wide Web.

Browser ABCs ...

The service provided by a web browser is the (usually visual) presentation of a resource located in the World Wide Web. We use web browsers so often we usually don't take time to ponder how they perform their service. It is instructive however, as a browser functions in basically the same way as an accessor does in NetKernel.

Fundamentally, a browser accepts a request for a URL, retrieves that resource, performs a page-flow layout and presents the visual representation to a user. As we examine these steps we'll also see reasons why the World Wide Web is flexible and scalable.

First, consider the nature of a resource. When we request a resource identified by a URL such as "http://www.google.com" we are specifying the address for a resource within the World Wide Web address space. Resources are used to model abstract information. For example, the resource "http://www.1060research.com" is an abstract notion - the "home page" for the 1060 Research web site. It may seem strange to think about this, but the home page doesn't actually exist until it is requested. The act of requesting "http://www.1060research.com" causes the endpoint responsible for it to create and return a physical representation. All representations are immutable - they cannot be changed, they can only expire and become invalid. Architecturally, immutability is wonderful for scaling. In the World Wide Web immutable representations can be stored in various locations to speed up processing such as in a browser or a proxy server or cache. As long as the representation has not expired, it can be provided by any of these caches, saving a trip to and work by the endpoint.

Second, consider what a browser does when it receives an HTML encoded resource representation. The browser interprets the HTML codes and if it finds resource references (such as for CSS, image, sound or other resources), it must issue sub-requests for them before it can perform its service. Once it has gathered all the referenced resource representations it can perform the page-flow layout work and build a visual representation. (Notice that a browser is free to issue the sub-requests asynchronously thereby creating parallel processing requests)

Third, consider the mapping of the logical address to an endpoint. For each request, the browser relies on the Domain Name Service (DNS) to map a logical address to an endpoint. While DNS is heavily cached and uses reasonably long expiry times there is no fundamental reason why an IP address couldn't be different for each request. In fact, IP round-robin assignment is a technique used to distributed requests across several endpoint IP addresses. For very large sites an IP load balancer is used to map a single IP address to one or potentially thousands of servers. In these horizontally scaled designs there is no notion of threads - all that a user and browser sees are requests.

A browser working with the World Wide Web is an amazingly flexible system. All addresses are logical and are bound to an endpoint for each request. Because of this indirection, web sites can appear and disappear, expand or contract at any time. Web servers can be upgraded or changed to different technologies and there is no impact on browsers - there is no physical coupling. Processing can occur in parallel without requiring knowledge of threads and immutable representations can be cached increasing the responsiveness of the whole system.

These are great properties! What if we could use this approach to develop software itself? Would we see the same properties from the web evident in software?

Turns out, we can ... and we do ...

NetKernel ABCs ...

In NetKernel all information is modeled as resources and resources are identified by URI addresses.

When a resource is requested, the address is resolved to an endpoint known as an Accessor. Like the browser, an accessor returns a representation of a resource. Also like a browser, an accessor may interpret codes and request additional resources. Unlike a browser, an accessor can return any form of representation. You can think of an accessor in NetKernel as a browser without the visual presentation surface.

Again, let's consider resources. In NetKernel resources represent abstract information such as "the list of customers" or "currently enrolled students". These resources are identified by URI addresses such as "ffcpl:/resources/customers" and "ffcpl:/enrolled_students". When a resource is requested, an endpoint returns the current state of that resource as a physical, immutable resource representation. A resource may be fundamental and atomic or it may be composed of other resources. For example, an application that assigns students to courses would need to compose a presentation page from the "list of courses" resource and the "list of students" resource.

Processing Steps

When an endpoint accessor receives a request, it examines the request to determine the context and specifics of the request. It can examine the requested URI and the names and URI addresses of each parameter. The following table illustrates the processing steps taken by a browser and a NetKernel accessor upon receiving a request.

The only significant difference between a browser and an accessor in NetKernel is the way endpoint resolution is accomplished. A browser uses DNS to locate an endpoint in a single, global address space while an accessor relies on NetKernel's microkernel to resolve an address in a set of related address spaces.

It is important to note what is not occurring - the NetKernel accessor is not referencing physically linked objects, data or code. If it requires additional information it issues a sub-request for a logically identified resources back into the logical address space. This is significant and extremely important.

The following code illustrates a BeanShell script that requests the students and course information:

void main() {
  // Request students and courses resources
  repStudents = context.source("ffcpl:/enrolled_students");
  repCourses  = context.source("ffcpl:/2008/spring/courses");

  ...
}

Because information is modeled as resources and resource addresses are bound to an endpoint only at the moment of each request, NetKernel applications take on the properties of the World Wide Web:

• Flexibility - Resource endpoints can be mapped to different endpoint implementation for each request. This means that operational systems can be modified, upgraded and down graded at any time while handling requests.
• Scaling - Requests know nothing of threads. The microkernel is free to asynchronously schedule requests from its carefully managed pool of threads onto any available CPU core. As cores are added, performance increasing almost linearly.
• Generational growth - clients and endpoints can evolve independently of each other. Because they are logically coupled, a change in either does not require code recompilation of the entire system.

These and other properties accrue to NetKernel systems. The idea of building information systems at the logical level of information instead of the physical level of objects and data has been validated by the World Wide Web as well as the numerous applications built and running on NetKernel.