Ok, but ...

This is really confusing and unintuitive.  I suspected the answer was something like this, but I could find nothing in the documentation that suggests the implication of something as simple and common as getting parameters.

But more importantly, why is this useful?  The result of this is that getting the parameters anywhere in your call stack renders everything above it completely non-cachable, regardless of whether the value is different, important, or even used.  I thought the benefit of the 'httpRequest' space was that you could easily get at request info from a very deep level without needing to pass things down at every level, but now my advice would have to be don't use it except at the very highest level.

In addition to that, why does the value even get (and stay) cached if there's no way to ever get to it?  It just uselessly fills up the cache with inaccessible junk.  (And confuses anyone who might try to understand what's going on.)

Hi Jeff, although the resource representation isn't expired in your example the scope of response is different each time it is requested. This is because you have forced inclusion of the http request address space by resolving a resource from there. This in effect means that you are actually computing the representation in a different place and therefore the value you have in the cache cannot be validly used. For more details and example see my blog post:

http://durablescope.blogspot.com/2009/11/caching-how-and-why-it-works.html

One way to change the scoping and hence the cachability is to have a layer in your system which extracts the params from the http request space and puts them onto the request. (you could use the mapper to do this). That way for an identical value of a parameter you would be able to retrieve the response representation from cache (all other factors willing)

Cheers, Tony

A confusing piece of behavior I ran into. 

context.source("httpRequest:/params"); response=context.createResponseFrom("xyz:"+Math.round(Math.random()*1000)); response.setExpiry(response.EXPIRY_DEPENDENT); context.setResponse(response);

This is close to as simple a groovy script as you can have.  The expected behavior is that it will return a string like "xyz:123" once, cache that value, and after that keep returning the same result.  It doesn't.  You get a new result every time.  My first thought was that 'httpRequest:/params' is always expired, which would prevent something dependent on it from getting cached.  So I tried instead

context.source("httpRequest:/params"); response=context.createResponseFrom("xyz:"+Math.round(Math.random()*1000)); response.setExpiry(response.EXPIRY_NEVER); context.setResponse(response);

But the result is the same - a new random number every time.  I checked with the visualizer, and the uppermost resource indicates that it is not expired, yet it is not retrieved from cache.  And the representation cache shows many instances of the same resource.

So what's going on?  Does any access of http request parameters simply render a request completely non-cacheable, counter to both reasonable expectations and an explicit declaration that it is cacheable forever?

There is no immediate need given your suggested work around (Thank you).

The scenario is when you want to fork a process into separate threads of work (via subrequests) and you want to clone the existing requests (verb, parameters, headers, primary representation) but obviously just need to change the URI.

Hi Jeremy,
yes it could. The INKFRequest interface used to have a setIdentifier() method but that was removed because of the issues wrt to constructing requests targeting endpoints and the valid that does against the endpoint interface specification. Your suggested method of:

INKFRequest INKFRequestReadOnly.createIssuableClone(String aIdentifier)

would work. What are you timescales for needed this enhancement? As a workaround you can use the code inside the method to achieve the same result:

NKFRequestImpl req= (NKFRequestImpl)mContext.createRequest(this.getIdentifier()); req.setVerb(this.getVerb()); req.setRepresentationClass(this.getRepresentationClass()); INKFResponseReadOnly primaryResponse = this.getPrimaryAsResponse(); if (primaryResponse!=null) {   req.addPrimaryArgumentFromResponse(primaryResponse); } IRequestResponseFields headers=mRequest.getHeaders(); for (int i=0; i<headers.size(); i++) {   req.setHeader(headers.getKey(i), headers.getValue(i), headers.isSticky(i)); }

Cheers, Tony

[This was posted to the mailing list 1 week ago]

What's new?  A series of enhancements to detect and prevent commission-time space circularities.  Detailed analysis of the insides of the Portal application.

NKSE Repository

NetKernel uses the ROC abstraction to bootstrap the commissioning of address spaces.  The dynamic import capability means that spaces can be commissioned and discovered in any order.  In architectures with multiple tiers of dynamic import it is therefore possible to have very exotic discovery.  To prevent recursive behaviour there is circularity detection at commission time.  The following enhancements were made...

layer0:   detection and stable handling of misconfigured modules containing circular dependencies. Fix to HDSUtils.inOrderTraversalNext()

standard-module: Fixes to startup stability during space commission of certain multiple tiered dynamic import spacial configurations.

layer1: Changed hds aggregate/simple dynamic import to avoid circular dependencies.

Other changes...

kernel: changed some interfaces to enable thread profiling

nkse-control-panel: added pds config so that it doesn't rely on finding another in fulcrum

http-server:  Just when you think you've got Jetty's encoding sorted out you launch a portal and get someone registering with an umlaut in their name.  Turns out, that last week we fixed the encoding when Jetty was guessing, but now find that when the client is well behaved (JQuery AJAX) and provides the encoding on the Content-Type header, then Jetty actually uses it correctly and mangling is not needed.  So this week we trapped this corner case too and relaxed the mangling.  (Thanks to  Thomas).

lang-xrl:  Documentation typos fixed.

NKEE Repo

All of the above NKSE updates are available in the NKEE 4.0.0 preview-3.1 repository along with a feature enhancement...

nkee-dev-tools:  Enhanced health check, now provides a space integrity check and reporting tool.

Portal Architecture

We've had some great feedback from people trying out the portal - we think we've fixed all the issues that were reported to us - so we'll be going public next week.  If you've not tried it you can get to it (and join) here...

https://cs.1060research.com/csp/

We thought it would be interesting to provide some deep and detailed statistics of this codebase now that its in production.  A high-level architectural overview is available here...

http://resources.1060research.com/docs/2010/02/CSPPortalArch.pdf

It shows the outline of the portlet pattern we're using and shows how we actually have a double level architecture - there is a second "1060-in-house" portal that dynamically imports a large set of 1060 portlets (70% of the system is "dark-matter" in house stuff).  The 1060 portal is itself deployed as a portlet into the main services portal! Not shown are the infrastructural components: virtual host space, async HTTP transport, throttle, sessions, double-gatekeepers, decorating XRL template wrapper.

We've had a number of people asking if we can talk through the design in detail.  So we'll host a webmeeting session for a deep dive through the code, as well as tour of the dark matter side of the system.  Send us a support ticket expressing your interest and we'll arrange a time that covers the most timezones.

Finally some statistics, the following spreadsheet provides a very detailed analysis of the code, static resources, database, spacial topologies and development time...

http://resources.1060research.com/docs/2010/02/CSP-analysis.ods
http://resources.1060research.com/docs/2010/02/CSP-analysis.xls

A few extracts:

* There are 14 modules to the solution deployed from a custom application repository with hot-updates
* There are 33 rootspaces with a further 38 spaces contained within.
* There are 192 application channels (= distinct REST/URL interfaces) of which 64 are AJAX micro-services (we use JQuery).
* There are 145 custom accessors all of which are written in groovy with the mapper used to bind them to the address space grammar.
* There are 49 tables in the RDBMS
* There are 1182 unique points where requests are issued.
* The average number of requests per application channel is 6.
* The average NetKernel response time (transport request to transport response) per channel is 10ms (network and browser rendering latency dominate) (The NKEE visualizer provides high resolution/high granularity request timing data).
* It took 1 person (guess who) 288 hours to develop with a 50:50 split between data model development (working out just what we needed and developiong the DB schema) and actual physical "coding" (144hrs - spacial architecture, pattern implementation, implementing channels, cutting resource endpoints).
* The development rate translates to about 8 endpoints per hour.
* The system was written from scratch since the start of this year.
* This week in production it has taken an average 5 minutes end-to-end to fix a bug, cut an update, deploy to repository, sync and deploy to production.(NKEE has ARP a repo building tool which you can do the same with).

********
Advertorial Alert: We are available to solve your problems. No job too small (after all, with ROC most jobs are small). Great rates and access to the horses mouth (is that a benefit?).
********

ROC News

Randy Kahle will be speaking at the Denver Open Source group meeting on Tuesday next week:

http://denveropensource.org/upcoming

Still no time to blog - sorry.

Have a great weekend!

The 1060 Team

The API allows you to clone the existing request and update the verb.

INKFRequest request =  context.getThisRequest().getIssuableClone();

But the does not allow you to change the URI. Could this be added to the API?
For example:

String uri = "res:/blah/foo" INKFRequest request =  context.getThisRequest().getIssuableClone(uri)

Hi ROCers,

What's new this week?  Several performance optimizatons including a 40% faster kernel.  Some optimization updates to core modules.

NKSE Repository Updates
-----------------------

kernel:  We are preparing an NKperf system benchmarking tool. This work resulted in some enhancements to the kernel (the first significant changes for nearly 2 years).  Firstly we removed a very narrow, but potential, race condition when receiving an asynchronous exception response - we'd never seen this but its fixed anyway.

The second update is a redesigned data structure for the scheduler request queue - this eliminates a Java level contention that limited throughput (especially with asnyc requests).  Net result of these changes is that the kernel is faster, for async requests this can be up to 40% in some scenarios.

layer0: removed synchronization from RegexRewriteRule this improves concurrent throughput of the http bridge and filesets indirectly.

standard module: fix to overlay base classes to enable metadata for non-absolute logical endpoint identifiers to be resolved correctly. For example. this update eliminates the possibility of prototype discovery failure inside a mapper.

http-server:  Another performance enhancement, we increased throughput of http sessions by optimising stale session reaping.

nkse-control-panel: made endpoints threadsafe to increase throughput.

system-core services: updated to account for kernel changes

nkse-unit-tests: updated with tests for new kernel etc.

layer1: active:exec accessor fixed stderr processing and has improved documentation.

nkse-docs:  Javadoc generation now works again after a screw up when we removed a cross-package dependency last week.


Short and sweet this week but lots of stuff will break cover next week. Watch this space...

Hi ROCers,

What's new this week?  Refinements to the request nesting feature in NKF, tweaks to http-client and server, a few minor fixes.  The first appearance of NetKernel Enterprise Edition preview 2 and NKEE repository.  CSP updates.

NKSE Repository Updates
-----------------------

layer0 - refinement to request nesting to make scope of inner request follow the scope of the outer request when using mapper to define wrapped request chains.

http-client - now exports its Apache commons http classes.

http-server - Session overlay now has a configuration option to set the name of the Cookie to use for correlating the session.

layer1 - the BinaryStream2String transreptor now handles streams that don't report their length.

nkse-dev-tools - control panel search indexer updated, granularity of the request throughput chart was enhanced.

visualizer - a fix was made to prevent possible NPEs when saving a visualizer trace.

nkse-control-panel - now eliminates dependencies in the search results when they are not local resources.

CSP
---
We're still in pre-release testing but getting close to a public launch.  For those who are in the test programme there are two new portlets...

A wiki portlet has been added. So now every project has a shared team wiki. A license portlet has been added for license management

https://cs.1060research.com/csp/

NetKernel Enterprise Edition - Preview 2
----------------------------------------
We've started a limited access programme for NKEE preview 2 with a corresponding long-term enterprise repository.  Once initial shakedown is complete we'll be widening access and sending out invitations to join the portal and get hold of the NKEE release.

ROC News
--------
Sorry - we've been flat out to get NKEE ready so too busy to blog.  Next week! Promise!

Have a great weekend!

The 1060 Team

Hi ROCers,

What's new this week?  Plenty of updates, mostly enhancements - such as better search results etc.

Probably the most important new feature is an enhancement to NKF and declarative requests to support request chaining.  This enables lazy evaluation of functional f(g(x)) style requests and introduces the potential of "micro-overlay" patterns.

Repository Updates
------------------

The following packages are updated in the repository - to update use Apposite->admin->sync then look at packages and "install all updates".

http-server: Thanks to Miles, Jeff and Menzo's diligence it was discovered that Jetty 6.1.x has been repeatedly changing its mind between release versions about what it thinks the default content encoding of a POST should be.  To take control we've added a configuration property that will allow you to definitively set this for an individual transport instance...

<config>
...
<defaultPostParamEncoding>UTF-8</defaultPostParamEncoding>
</config>

If this parameter is not set we ensure that the NK default is always UTF-8 - we've introduced a trick to convert Jetty's "flavour of the day" when necessary.

layer0:

When transrepting a binary stream XML resource to HDS, the SAX handler now correctly deals with namespaces and, like with the existing DOM to HDS, represents the XML in the HDS tree.

The module change detection now recommissions dependent modules even when they have no public spaces.  This update resolves the Cookie class cast on http-server update problem from last week - it turned out that the Fulrcums were not being recommissioned since they had private spaces.

NKF f(g(x)) request nesting - requests can now be added as first class arguments on a constructed request.  Therefore it is now possible to create lazily evaluated request chains similar to how DPML's lazy evaluation model works.  This enhancement to NKF enables a raft of new patterns - for example, with it, you can now create a local wrapper function around any endpoint without needing the static structure of overlay's and nested spaces.  In short you can do micro-overlays.  Tony has promised to blog in detail about this new capability (but you know what that means).

For completeness, the declarative request also supports nested requests - so you can create overlays just using the mapper alone.

nkse-control-panel:

Added indexing/resolving to control panels - so that they come up in searches.

nkse-cron:  A fix to the control panel so that buttons work properly. The onclick() events had stopped working after changes to the control panel  rendering (lesson here, watch out for attribute case sensitivity with XHTML).

nkse-dev-tools:

Class trace tool now works for modules with no public spaces (eg Fulcrums).

nkse-docs:

Updated with better glossary and more keywords to help search.  Doc indexing was also tweaked so that the book name is shown in a search result - to provide better search context.

nkse-visualizer:

Primary arguments (ie SINK or TRANSREPT payloads) are now captured and visible for inspection.

pds-core:

Tweak to error messages so that you can see the zone if you have a resolution failure.

NetKernel Servlet Beta 2
------------------------

A Beta 2 update version of the The Servlet embedded version of NetKernel is available incorporating some compatibility tweaks based on user feedback.  We're still looking for feedback and coverage in containers.

Download it here...

http://resources.1060research.com/aux/nkse/4/servlet/beta/netkernel-servlet-beta-2.zip

The solution is self-contained and includes a complete NKSE system ready to go within a servlet environment.

There are three files in this distribution, netkernelservlet.jar, web.jar and timeservice.war. The timeservice.war file is a complete ready-to-use web application which implements a RESTful time web service.

The netkernelservlet.jar file contains the NetKernel Servlet. This is a traditional servlet that can be installed in the WEB-INF/lib/ directory of a web application and configured in the web.xml

A detailed installation and usage guide is provided here...

http://resources.1060research.com/aux/nkse/4/servlet/beta/servlet-documentation.pdf

1060 CSP Testing
----------------

We've started public testing of the new Customer Services Portal (CSP).  We'll be progressively broadening access over the next week or so, after which we will declare it officially open.  Thanks very much to our early access users for their feedback.

Production tip: We discovered during the staging cycle that updating the Sun JVM to the most recent 1.6.18 build has solved a lot of GC segfault bugs we were seeing on the 64bit linux JVM.