1060 Forum - Recent Entries

Hrm

Thu, 11 Mar 2010 18:01:00 GMT

Ok, but ...

This is really confusing and unintuitive. I suspected the answer was something like this, but I could find nothing in the documentation that suggests the implication of something as simple and common as getting parameters.

But more importantly, why is this useful? The result of this is that getting the parameters anywhere in your call stack renders everything above it completely non-cachable, regardless of whether the value is different, important, or even used. I thought the benefit of the 'httpRequest' space was that you could easily get at request info from a very deep level without needing to pass things down at every level, but now my advice would have to be don't use it except at the very highest level.

In addition to that, why does the value even get (and stay) cached if there's no way to ever get to it? It just uselessly fills up the cache with inaccessible junk. (And confuses anyone who might try to understand what's going on.)

It the scope

Thu, 11 Mar 2010 12:47:00 GMT

Hi Jeff, although the resource representation isn't expired in your example the scope of response is different each time it is requested. This is because you have forced inclusion of the http request address space by resolving a resource from there. This in effect means that you are actually computing the representation in a different place and therefore the value you have in the cache cannot be validly used. For more details and example see my blog post:

http://durablescope.blogspot.com/2009/11/caching-how-and-why-it-works.html

One way to change the scoping and hence the cachability is to have a layer in your system which extracts the params from the http request space and puts them onto the request. (you could use the mapper to do this). That way for an identical value of a parameter you would be able to retrieve the response representation from cache (all other factors willing)

Cheers, Tony

Why doesn't this cache?

Thu, 11 Mar 2010 08:41:00 GMT

A confusing piece of behavior I ran into.


context.source("httpRequest:/params");
response=context.createResponseFrom("xyz:"+Math.round(Math.random()*1000));
response.setExpiry(response.EXPIRY_DEPENDENT);
context.setResponse(response);

This is close to as simple a groovy script as you can have. The expected behavior is that it will return a string like "xyz:123" once, cache that value, and after that keep returning the same result. It doesn't. You get a new result every time. My first thought was that 'httpRequest:/params' is always expired, which would prevent something dependent on it from getting cached. So I tried instead


context.source("httpRequest:/params");
response=context.createResponseFrom("xyz:"+Math.round(Math.random()*1000));
response.setExpiry(response.EXPIRY_NEVER);
context.setResponse(response);

But the result is the same - a new random number every time. I checked with the visualizer, and the uppermost resource indicates that it is not expired, yet it is not retrieved from cache. And the representation cache shows many instances of the same resource.

So what's going on? Does any access of http request parameters simply render a request completely non-cacheable, counter to both reasonable expectations and an explicit declaration that it is cacheable forever?

Request Clone - Change URI

Sun, 07 Mar 2010 16:51:00 GMT

There is no immediate need given your suggested work around (Thank you).

The scenario is when you want to fork a process into separate threads of work (via subrequests) and you want to clone the existing requests (verb, parameters, headers, primary representation) but obviously just need to change the URI.

Sun, 07 Mar 2010 15:21:00 GMT

Hi Jeremy,
yes it could. The INKFRequest interface used to have a setIdentifier() method but that was removed because of the issues wrt to constructing requests targeting endpoints and the valid that does against the endpoint interface specification. Your suggested method of:

INKFRequest INKFRequestReadOnly.createIssuableClone(String aIdentifier)

would work. What are you timescales for needed this enhancement? As a workaround you can use the code inside the method to achieve the same result:

NKFRequestImpl req= (NKFRequestImpl)mContext.createRequest(this.getIdentifier());
req.setVerb(this.getVerb());
req.setRepresentationClass(this.getRepresentationClass());
INKFResponseReadOnly primaryResponse = this.getPrimaryAsResponse();
if (primaryResponse!=null)
{   req.addPrimaryArgumentFromResponse(primaryResponse);
}
IRequestResponseFields headers=mRequest.getHeaders();
for (int i=0; i<headers.size(); i++)
{   req.setHeader(headers.getKey(i), headers.getValue(i), headers.isSticky(i));
}

Cheers, Tony

NetKernel News Volume 1 Issue 17

Sun, 07 Mar 2010 13:32:00 GMT

[This was posted to the mailing list 1 week ago]

What's new? A series of enhancements to detect and prevent commission-time space circularities. Detailed analysis of the insides of the Portal application.

NKSE Repository

NetKernel uses the ROC abstraction to bootstrap the commissioning of address spaces. The dynamic import capability means that spaces can be commissioned and discovered in any order. In architectures with multiple tiers of dynamic import it is therefore possible to have very exotic discovery. To prevent recursive behaviour there is circularity detection at commission time. The following enhancements were made...

layer0: detection and stable handling of misconfigured modules containing circular dependencies. Fix to HDSUtils.inOrderTraversalNext()

standard-module: Fixes to startup stability during space commission of certain multiple tiered dynamic import spacial configurations.

layer1: Changed hds aggregate/simple dynamic import to avoid circular dependencies.

Other changes...

kernel: changed some interfaces to enable thread profiling

nkse-control-panel: added pds config so that it doesn't rely on finding another in fulcrum

http-server: Just when you think you've got Jetty's encoding sorted out you launch a portal and get someone registering with an umlaut in their name. Turns out, that last week we fixed the encoding when Jetty was guessing, but now find that when the client is well behaved (JQuery AJAX) and provides the encoding on the Content-Type header, then Jetty actually uses it correctly and mangling is not needed. So this week we trapped this corner case too and relaxed the mangling. (Thanks to Thomas).

lang-xrl: Documentation typos fixed.

NKEE Repo

All of the above NKSE updates are available in the NKEE 4.0.0 preview-3.1 repository along with a feature enhancement...

nkee-dev-tools: Enhanced health check, now provides a space integrity check and reporting tool.

Portal Architecture

We've had some great feedback from people trying out the portal - we think we've fixed all the issues that were reported to us - so we'll be going public next week. If you've not tried it you can get to it (and join) here...

https://cs.1060research.com/csp/

We thought it would be interesting to provide some deep and detailed statistics of this codebase now that its in production. A high-level architectural overview is available here...

http://resources.1060research.com/docs/2010/02/CSPPortalArch.pdf

It shows the outline of the portlet pattern we're using and shows how we actually have a double level architecture - there is a second "1060-in-house" portal that dynamically imports a large set of 1060 portlets (70% of the system is "dark-matter" in house stuff). The 1060 portal is itself deployed as a portlet into the main services portal! Not shown are the infrastructural components: virtual host space, async HTTP transport, throttle, sessions, double-gatekeepers, decorating XRL template wrapper.

We've had a number of people asking if we can talk through the design in detail. So we'll host a webmeeting session for a deep dive through the code, as well as tour of the dark matter side of the system. Send us a support ticket expressing your interest and we'll arrange a time that covers the most timezones.

Finally some statistics, the following spreadsheet provides a very detailed analysis of the code, static resources, database, spacial topologies and development time...

http://resources.1060research.com/docs/2010/02/CSP-analysis.ods
http://resources.1060research.com/docs/2010/02/CSP-analysis.xls

A few extracts:

* There are 14 modules to the solution deployed from a custom application repository with hot-updates
* There are 33 rootspaces with a further 38 spaces contained within.
* There are 192 application channels (= distinct REST/URL interfaces) of which 64 are AJAX micro-services (we use JQuery).
* There are 145 custom accessors all of which are written in groovy with the mapper used to bind them to the address space grammar.
* There are 49 tables in the RDBMS
* There are 1182 unique points where requests are issued.
* The average number of requests per application channel is 6.
* The average NetKernel response time (transport request to transport response) per channel is 10ms (network and browser rendering latency dominate) (The NKEE visualizer provides high resolution/high granularity request timing data).
* It took 1 person (guess who) 288 hours to develop with a 50:50 split between data model development (working out just what we needed and developiong the DB schema) and actual physical "coding" (144hrs - spacial architecture, pattern implementation, implementing channels, cutting resource endpoints).
* The development rate translates to about 8 endpoints per hour.
* The system was written from scratch since the start of this year.
* This week in production it has taken an average 5 minutes end-to-end to fix a bug, cut an update, deploy to repository, sync and deploy to production.(NKEE has ARP a repo building tool which you can do the same with).

********
Advertorial Alert: We are available to solve your problems. No job too small (after all, with ROC most jobs are small). Great rates and access to the horses mouth (is that a benefit?).
********

ROC News

Randy Kahle will be speaking at the Denver Open Source group meeting on Tuesday next week:

http://denveropensource.org/upcoming

Still no time to blog - sorry.

Have a great weekend!

The 1060 Team

Request Clone - Change URI

Sun, 07 Mar 2010 02:11:00 GMT

The API allows you to clone the existing request and update the verb.


INKFRequest request =  context.getThisRequest().getIssuableClone();

But the does not allow you to change the URI. Could this be added to the API?
For example:


String uri = "res:/blah/foo"
INKFRequest request =  context.getThisRequest().getIssuableClone(uri)

NetKernel News Volume 1 Issue 16

Thu, 25 Feb 2010 19:10:00 GMT

Hi ROCers,

What's new this week? Several performance optimizatons including a 40% faster kernel. Some optimization updates to core modules.

NKSE Repository Updates
-----------------------

kernel: We are preparing an NKperf system benchmarking tool. This work resulted in some enhancements to the kernel (the first significant changes for nearly 2 years). Firstly we removed a very narrow, but potential, race condition when receiving an asynchronous exception response - we'd never seen this but its fixed anyway.

The second update is a redesigned data structure for the scheduler request queue - this eliminates a Java level contention that limited throughput (especially with asnyc requests). Net result of these changes is that the kernel is faster, for async requests this can be up to 40% in some scenarios.

layer0: removed synchronization from RegexRewriteRule this improves concurrent throughput of the http bridge and filesets indirectly.

standard module: fix to overlay base classes to enable metadata for non-absolute logical endpoint identifiers to be resolved correctly. For example. this update eliminates the possibility of prototype discovery failure inside a mapper.

http-server: Another performance enhancement, we increased throughput of http sessions by optimising stale session reaping.

nkse-control-panel: made endpoints threadsafe to increase throughput.

system-core services: updated to account for kernel changes

nkse-unit-tests: updated with tests for new kernel etc.

layer1: active:exec accessor fixed stderr processing and has improved documentation.

nkse-docs: Javadoc generation now works again after a screw up when we removed a cross-package dependency last week.

Short and sweet this week but lots of stuff will break cover next week. Watch this space...

NetKernel News Volume 1 Issue 15

Thu, 25 Feb 2010 19:09:00 GMT

Hi ROCers,

What's new this week? Refinements to the request nesting feature in NKF, tweaks to http-client and server, a few minor fixes. The first appearance of NetKernel Enterprise Edition preview 2 and NKEE repository. CSP updates.

NKSE Repository Updates
-----------------------

layer0 - refinement to request nesting to make scope of inner request follow the scope of the outer request when using mapper to define wrapped request chains.

http-client - now exports its Apache commons http classes.

http-server - Session overlay now has a configuration option to set the name of the Cookie to use for correlating the session.

layer1 - the BinaryStream2String transreptor now handles streams that don't report their length.

nkse-dev-tools - control panel search indexer updated, granularity of the request throughput chart was enhanced.

visualizer - a fix was made to prevent possible NPEs when saving a visualizer trace.

nkse-control-panel - now eliminates dependencies in the search results when they are not local resources.

CSP
---
We're still in pre-release testing but getting close to a public launch. For those who are in the test programme there are two new portlets...

A wiki portlet has been added. So now every project has a shared team wiki. A license portlet has been added for license management

https://cs.1060research.com/csp/

NetKernel Enterprise Edition - Preview 2
----------------------------------------
We've started a limited access programme for NKEE preview 2 with a corresponding long-term enterprise repository. Once initial shakedown is complete we'll be widening access and sending out invitations to join the portal and get hold of the NKEE release.

ROC News
--------
Sorry - we've been flat out to get NKEE ready so too busy to blog. Next week! Promise!

Have a great weekend!

The 1060 Team

NetKernel News Volume 1 Issue 14

Thu, 25 Feb 2010 19:08:00 GMT

Hi ROCers,

What's new this week? Plenty of updates, mostly enhancements - such as better search results etc.

Probably the most important new feature is an enhancement to NKF and declarative requests to support request chaining. This enables lazy evaluation of functional f(g(x)) style requests and introduces the potential of "micro-overlay" patterns.

Repository Updates
------------------

The following packages are updated in the repository - to update use Apposite->admin->sync then look at packages and "install all updates".

http-server: Thanks to Miles, Jeff and Menzo's diligence it was discovered that Jetty 6.1.x has been repeatedly changing its mind between release versions about what it thinks the default content encoding of a POST should be. To take control we've added a configuration property that will allow you to definitively set this for an individual transport instance...

<config>
...
<defaultPostParamEncoding>UTF-8</defaultPostParamEncoding>
</config>

If this parameter is not set we ensure that the NK default is always UTF-8 - we've introduced a trick to convert Jetty's "flavour of the day" when necessary.

layer0:

When transrepting a binary stream XML resource to HDS, the SAX handler now correctly deals with namespaces and, like with the existing DOM to HDS, represents the XML in the HDS tree.

The module change detection now recommissions dependent modules even when they have no public spaces. This update resolves the Cookie class cast on http-server update problem from last week - it turned out that the Fulrcums were not being recommissioned since they had private spaces.

NKF f(g(x)) request nesting - requests can now be added as first class arguments on a constructed request. Therefore it is now possible to create lazily evaluated request chains similar to how DPML's lazy evaluation model works. This enhancement to NKF enables a raft of new patterns - for example, with it, you can now create a local wrapper function around any endpoint without needing the static structure of overlay's and nested spaces. In short you can do micro-overlays. Tony has promised to blog in detail about this new capability (but you know what that means).

For completeness, the declarative request also supports nested requests - so you can create overlays just using the mapper alone.

nkse-control-panel:

Added indexing/resolving to control panels - so that they come up in searches.

nkse-cron: A fix to the control panel so that buttons work properly. The onclick() events had stopped working after changes to the control panel rendering (lesson here, watch out for attribute case sensitivity with XHTML).

nkse-dev-tools:

Class trace tool now works for modules with no public spaces (eg Fulcrums).

nkse-docs:

Updated with better glossary and more keywords to help search. Doc indexing was also tweaked so that the book name is shown in a search result - to provide better search context.

nkse-visualizer:

Primary arguments (ie SINK or TRANSREPT payloads) are now captured and visible for inspection.

pds-core:

Tweak to error messages so that you can see the zone if you have a resolution failure.

NetKernel Servlet Beta 2
------------------------

A Beta 2 update version of the The Servlet embedded version of NetKernel is available incorporating some compatibility tweaks based on user feedback. We're still looking for feedback and coverage in containers.

Download it here...

http://resources.1060research.com/aux/nkse/4/servlet/beta/netkernel-servlet-beta-2.zip

The solution is self-contained and includes a complete NKSE system ready to go within a servlet environment.

There are three files in this distribution, netkernelservlet.jar, web.jar and timeservice.war. The timeservice.war file is a complete ready-to-use web application which implements a RESTful time web service.

The netkernelservlet.jar file contains the NetKernel Servlet. This is a traditional servlet that can be installed in the WEB-INF/lib/ directory of a web application and configured in the web.xml

A detailed installation and usage guide is provided here...

http://resources.1060research.com/aux/nkse/4/servlet/beta/servlet-documentation.pdf

1060 CSP Testing
----------------

We've started public testing of the new Customer Services Portal (CSP). We'll be progressively broadening access over the next week or so, after which we will declare it officially open. Thanks very much to our early access users for their feedback.

Production tip: We discovered during the staging cycle that updating the Sun JVM to the most recent 1.6.18 build has solved a lot of GC segfault bugs we were seeing on the 64bit linux JVM.

Yes!

Tue, 23 Feb 2010 12:10:00 GMT

In addition, if I defined an pluggable overlay in a module (urn:infrastructure:overlay) and wrapped all my other modules in that space, could I then dynamically import the overlay into the front-end fulcrum rather than dynamically importing each of the wrapped modules?

Yes you certainly can. That is a good way to achieve the result you want. The control panel uses that pattern to ensure each tool gets wrapped with a template.

Tony

Tue, 23 Feb 2010 12:08:00 GMT

Hi, the pluggable overlay only supports a single preProcess and/or postProcess. There are three ways to achieve the pipes&filters pattern.

Firstly you can use the branch-merge accessor which instantiates a set of pipes for different subsets of the address space and channels all request/responses through a set of overlays. This is the best approach if you have lots of logical channels which want different overlay behaviours.

Secondly you can just nest pluggable overlays in the module definition. This is simplest if every request wants the same behaviour (I think this is what you need for this example)

Thirdly you can created nested requests using the new nested request capability of declarative request. This approach doesn't use overlays so it a more dynamic but might not be suitable for your example.

Tony

Pluggable Overlay Questions - Pipes & Filter & Dynamic HTTP Fulcrum Import

Mon, 22 Feb 2010 13:39:00 GMT

This is example is taken from the REST tutorial.


    <pluggable-overlay>
      <preProcess>
        <identifier>active:java</identifier>
        <argument name="class">tutorial.rest.part4.part4b.HTTPVerbTranslationAccessor</argument>
        <argument name="request">arg:request</argument>
      </preProcess>
      <space>
        <mapper>
          <config>res:/tutorial/rest/part4/part4b/mapperConfig.xml</config>
          <space>
            <fileset>
              <regex>res:/tutorial/rest/part4/part4b/.*</regex>
            </fileset>
            <import>
              <uri>urn:org:netkernel:ext:layer1</uri>
            </import>
          </space>
        </mapper>
      </space>
    </pluggable-overlay>

Can a pluggable overlay have many pre-processors (Pipes & Filters pattern)? In other words could I add another audit pre-process and it would be called next (with the results from the first pre-process) before calling the wrapped space?


    <pluggable-overlay>
      <preProcess>
        <identifier>active:java</identifier>
        <argument name="class">tutorial.rest.part4.part4b.HTTPVerbTranslationAccessor</argument>
        <argument name="request">arg:request</argument>
      </preProcess>
      <preProcess>
        <identifier>active:java</identifier>
        <argument name="class">tutorial.rest.part4.part4b.AuditAccessor</argument>
        <argument name="request">arg:request</argument>
      </preProcess>
      <space>
        <mapper>
          <config>res:/tutorial/rest/part4/part4b/mapperConfig.xml</config>
          <space>
            <fileset>
              <regex>res:/tutorial/rest/part4/part4b/.*</regex>
            </fileset>
            <import>
              <uri>urn:org:netkernel:ext:layer1</uri>
            </import>
          </space>
        </mapper>
      </space>
    </pluggable-overlay>

NK4.0.2 (updated as of 2/19/2010) on Windows XP

Mon, 22 Feb 2010 13:23:00 GMT

Grammar:


<grammar>
    <group>
        res:/cogito/business/employee/
        <group name="employeeID"><regex type="alphanum"/></group>
        <regex type="anything"/>
    </group>
</grammar>

Target URI:
res:/cogito/business/employee/1234

Each time I parse this URI using the grammar above ever increasing white-pace is added.

Mon, 22 Feb 2010 11:36:00 GMT

Hi Jeremy, I can't reproduce this. I remember the grammar kitchen used to have this issue for me but it was fixed a while back. What platform/browser are you using? And can you give me an example grammar?

Cheers, Tony

Mon, 22 Feb 2010 11:25:00 GMT

Hi Jeremy sorry about the sparseness of the documentation for JMS still it's great that you found the answer in the source code. We are going to be having another round of documentation refinement soon which will increase the coverage of reference documentation.

The HDS you've found will transrept to XML. What the transreptor does in this case is insert a <hds> root node so you end up with this:

<hds>
  <name>value</name>
  <name>value</name>
  <name>value</name>
</hds>

Mon, 22 Feb 2010 11:20:00 GMT

Hi Jeremy, good question! We don't discussion the tradeoffs much in the documentation. The design intent of HDS is to be a lightweight replacement for DOM based XML (which DOMXDA is a wrapper for) for representing hierarchical data. That has always been the most common usage of XML in processing with NetKernel. It allows random access to the structure and is much lighter weight in terms of memory and processing time than DOM because it has less complexity and corner cases. It also allows for the carriage of POJOs rather than strings. HDS is also immutable which is a big positive too. HDS supports forests! Not a statement about it's green credentials but rather it's ability to have more than one root node - this makes it useful for representing lists.

There are two negatives with HDS vs. DOM XML which may or may not be an issued depending up what you are doing. Firstly HDS doesn't support full XPath so you need build up expressions using a combination of the simple path selection XPath subset it does support and the HDSPredicate based filtering. You can do most stuff with this but if you do need complex expressions you'd be better off using DOM. The second negative is that it isn't designed for representing documents with mixed content, basically each node in the tree has a single value, not values dispersed with children.

Yes transreptors exists to convert between XML and HDS. Most permutations are covered.

Cheers,
Tony

NK4.0.2: JMS Transport Documentation

Sun, 21 Feb 2010 22:21:00 GMT

In the JMS Transport Documentation

/book/view/book:tpt:jms/doc:tpt:jms:transport

it is stated that the header and properties can be sourced from the context as HDS or XML. But an example of the XML is not provided. I had to unpack urn.org.netkernel.tpt.jms-0.3.0.jar to see that it is valid HDS but not well formed XML (no root element):


HDSBuilder b=new HDSBuilder();
      for (Enumeration e=aMessage.getPropertyNames(); e.hasMoreElements(); )
      {   String propertyName = (String)e.nextElement();
         String propertyValue = aMessage.getStringProperty(propertyName);
         b.addNode(propertyName, propertyValue);
      }
      properties=b.getRoot();

The code above produces:


<name>value</name>
<name>value</name>
<name>value</name>

Which I believe would fail if you attempted to source this as a document.

NK4.0.2: HDS vs XDA

Fri, 19 Feb 2010 20:53:00 GMT

What is the relationship between HDS and XDA (e.g. immutable/mutable, performance/flexibility)?

And is there automatic transrepretion between the two?

For example an overlay may would use HDS to XML log data but the ultimate endpoint might use XDA to modify the XML values.

Grammar Kitchen Issue (NK4.0.2)

Thu, 18 Feb 2010 16:57:00 GMT

Every time I parse grammar it adds unnecessary white space between each of the elements. This makes iteratively building grammar a pain.