1060 Forum - Recent Entries

ok ... so how do I ...

Mon, 06 Oct 2008 06:38:00 GMT

Greetings Randy and Tony,

Thanks for taking the time to track down the problem. I have done the same in DPML and that worked, so I was stumped for quite some time there. Hopefully you will advise me how best to tackle my problem then (without having to resort to actually managing my files myself).

I'm trying to build a resource tree. The database is very simple, one record which contains the Bill of Material : ASSETBOM (shortdesc_owner, shortdesc_member);

I enter my program with (obviously) a shortdesc_owner. In one run of the program I read all the shortdesc_members and in a loop start the program recursively for all of them (going down the tree). I've got the recursion covered. What I don't have covered is that I need to keep all the results. Let me give an example output :
Level Shortdesc
0 SVLIPC16
1 SVLIPC16_BOOTDISK
1 SVLIPC16_CPU
1 SVLIPC16_NETWORK
2 SVLIPC16_NETW0
2 XW010101_P001
1 SVLIPC16_SAN
(and then think some nice XSLT-transformation for the eye-candy).
Now, since I have recursion covered, those levelnumbers are easy. I just pass a param down the line, increase /nvp/level at each pass. Piece of cake. For my data I thought to be smart as well and pass the URI as /nvp/uri. In DPML this worked great (but relatively slow). In Python ... I ran into the optimisation problem (but with "ordinary" files I outperform DPML 3/1).

So Tony, while I do agree that this may not occur often in a real application, it does occur in mine ;-).

Any suggestions ?
Regards,
Tom

The cause

Fri, 03 Oct 2008 16:19:00 GMT

Hi Tom,
I've looked into the test case you have and confirm that it is working as expected (at least by NetKernel if not you ;-)

What is happening is that NKF API contains an optimisation to not re-request when it already has something and the response hasn't expired. So in this situation what happens is that the second source request isn't performed. (this explains why the visualizer doesn't show the request) Even if it was performed it would be plucked from the cache and the same result would ensue.

I suspect the reason why the response from the source isn't expired is because ffcpl module resources use a poll period to avoid excessive file IO that drains performance. This poll period can be changed on the "Kernel Parameter Manager" page on the control panel. However it isn't good to rely upon this because it is entirely possible that the granularity of the filesystem timestamp isn't high-enough to detect the change anyway.

Sorry this has tripped you up though I suspect this kind of behavior wouldn't happen much in a real application.

Cheers,
Tony

Confirmed behavior

Fri, 03 Oct 2008 14:35:00 GMT

Tom,

Thank you for the program!

I can confirm that only that first SOURCE request is showing up in the visualizer on my system. I changed the program a bit to see if some initial ideas about the problem would resolve it and they do not.

We will now take a look at the issue and get back to you promptly.

Thanks!

-- Randy

the source

Fri, 03 Oct 2008 06:35:00 GMT

Greetings Randy,

Here it is :

import sys
import time
sys.add_package("org.ten60.netkernel.layer1.representation")
sys.add_package("org.ten60.netkernel.xml.representation")
sys.add_package("org.ten60.netkernel.xml.xda")
sys.add_package("org.ten60.netkernel.xml.util")
sys.add_package("org.ten60.netkernel.layer1.nkf")
sys.add_package("org.ten60.rdbms.representation")
from org.ten60.netkernel.layer1.representation import StringAspect   
#######################################################
temp_uri="ffcpl:/tmp/test_randy"
# Write initial output 
outputs="BEGIN"
context.sinkAspect(temp_uri,StringAspect(outputs))
#######################################################
output=context.sourceAspect(temp_uri,StringAspect)
outputs=str(output.getString())
outputs+="\nTHIS LINE GOES LOST"
context.sinkAspect(temp_uri,StringAspect(outputs))
#######################################################
output=context.sourceAspect(temp_uri,StringAspect)
outputs=str(output.getString())
outputs+="\nEND"
context.sinkAspect(temp_uri,StringAspect(outputs))
#######################################################
resp=context.createResponseFrom(StringAspect('RETURN'))
resp.setMimeType("text/xml")
context.setResponse(resp)

And my output in /tmp/test_randy :
BEGIN
END

Regards,
Tom

Re: NK 4 Preview 1

Thu, 02 Oct 2008 16:25:00 GMT

Thanks Peter.

I'd like to point out that the Preview 1 .jar file is exactly the same one that was distributed on the key at the Architect's Weekend (the source code for the lib modules is new).

So to install from this .jar file, follow the instructions from the readme.txt file (on the key only), which are as follows:

1060 NetKernel v4.0 Preview 1
-----------------------------

The NetKernel preview is packaged in an executable jar. NetKernel is
pre-built to run from the jar - like a Linux-Live-CD.

There's an installer tool that allows NetKernel to install itself to a
local disk.

System Requirements:
--------------------

NetKernel 4 requires Java 5 or higher.

To view the SVG diagrams in the documentation you will need an XHTML/SVG capable
browser such as Firefox or Opera.

Boot
----

To boot NetKernel:

java -jar 1060-NetKernel-4.0.0-preview-1.jar

and point your browser at

http://localhost:1060

Install
-------

To install NetKernel, when booted, point your browser at

http://localhost:1060/installer/

Follow the prompts to tell NetKernel where to install itself.

enjoy,
-tom

Python program

Thu, 02 Oct 2008 15:57:00 GMT

Tom,

I am not a python programmer, however, I am now setup with a debug module in which I want to replicate your problem with sink/source.

Would you mind posting a complete, short python script that replicates the problem with some simple strings? I'll use that in my module and see if I can figure out what is going on.

Thanks -- Randy[/i]

NK 4 Preview 1

Thu, 02 Oct 2008 14:58:00 GMT

I just sent email with details of where to get the limited access preview. If you didn't get it (or I screwed up the mailing list ;-) please drop me a note and I'll point you to it.

Cheers and enjoy.

Peter

jdbc indeed

Thu, 02 Oct 2008 14:22:00 GMT

Greetings Randy, Tony,

Its indeed a jdbc problem. It does work in HSQLDB. So I'm considering it a limitation of sqlite (you can't have everything in so small an engine).
Thanks for the suggestions that put me in the right track ... moving on to the other fish I got to fry with this application ;-)

Regards,
Tom

P.S. Randy, if you still want to have a look at the code your welcome to it (but I presume there's plenty of problems waiting at the moment). Its the same code I have the sink/source problem with.

non threadsafe issue in jdbc driver

Thu, 02 Oct 2008 12:54:00 GMT

Hi Tom,

I'd say it looks like your jdbc driver isn't threadsafe and you are hitting it with concurrent requests.

A solution might be to look for options on it's configuration or look at limiting the pool size for connections to 1.

If none of that works maybe you could create a data layer in your application to do database ops and place a throttle over it.

Cheers,
Tony

DB Problem

Thu, 02 Oct 2008 12:35:00 GMT

Tom,

The exception report seems to be coming from your database, if so my comments may help...

In your code snippet you are iterating on the results of a query. Since the DOM has already been created there should be no active connection to the database for that query. So the exception can't be caused by that code.

I'll presume that the code "impact_tree.py" is connecting to your database also. Since you are calling the code asynchronously you may be getting the database into a state it doesn't like. I don't know SQLLite. However I do know that some databases require column access in a certain order if BLOB or CLOB columns are involved. (The columns must be accessed in the order they are mentioned in the SQL statement - this has to do with buffering, etc in JDBC and the driver).

If you can provide the code (or a summary of it) from "impact_tree.py" maybe I can be of further assistance.

Randy

async processing

Thu, 02 Oct 2008 07:02:00 GMT

Greetings,
-- I seem to monopolise the forums at the moment ... ah well :-)

This is a snippet of my code ...

   counter=0
   resp={}
   for i in resultdom.getNodesFor('/results/row/shortdesc'):
      counter+=1
      newdesc=resultdom.getText('/results/row[' + str(counter) + ']/shortdesc',0)
      sat.setText('/nvp/description',newdesc)
      sr=context.createSubRequest("active:python")
      sr.addArgument("operator", "ffcpl:/resources/impact_tree.py")
      sr.addArgument("param",DOMXDAAspect(sat))
      resp[counter]=context.issueAsyncSubRequest(sr)

   counter=0   
   for i in resultdom.getNodesFor('/results/row/shortdesc'):
      counter+=1
      resp[counter].join()

# Write final output
if level==0:
   # Here the code for after all processing

As you can see I submit requests in a loop. Actually this is recusive, this is a snippet from impact_tree.py itself. I'm not really interested in the results ... however, I do want to know when all the requests in the loop have ended since I do have some final processing to do (only at the lowest level, but I know when that is).

I'd say that logically speaking my code is correct, however it throws following exception :

Message: call() called in inappropriate state 
Callstack: org.ibex.nestedvm.Runtime.call() line:655 
org.ibex.nestedvm.Runtime.call() line:647 
org.sqlite.NestedDB.call() line:451 
org.sqlite.NestedDB.call() line:433 

org.ibex.nestedvm.Runtime$ExecutionException: Jumped to invalid address at (unknown)

Seems that I'm pushing something to the limit ... any pointers as to what that can be ?

Regards,
Tom

read write works ... source sink does not

Thu, 02 Oct 2008 05:34:00 GMT

Greetings,

I replaced the source/sinks with "normal" python file access ... and that works. However I don't like it. I've got to specify where my files go, need specific routines for copying and deleting ... Somehow this does not feel like what I've gotten used to when using Netkernel.

So ... still looking for a solution to my "source"-problem ...

Regards,
Tom

Re: Proxy my Accessors?

Wed, 01 Oct 2008 17:38:00 GMT

Hi Shane,

Because of the loose coupling inherent in ROC, adding AOP-like behavior is easy. It is most often a matter of "wrapping" existing services by intercepting or translating their logical addresses.

So, instead of thinking about extending accessors or altering code, imagine the logical requests flowing around the system. The logical requests are all specified by URIs. You probably want to "intercept" only a identified subset of these URIs and pass the rest through. This is the ROC equivalent of identifying your joinpoints in AOP.

You are correct in identifying this pattern as an overlay in NK4, but the Functional Overlay pattern is also implementable in NK3:

http://docs.1060.org/docs/3.3.0/book/architectguide/doc_tutorial_patterns_functional_overlay.html

regards,
-Tom
[/quote]

Proxy my Accessors?

Wed, 01 Oct 2008 17:05:00 GMT

Is it possible to proxy my accessors for AOP types of behavior? I have two problems when I look at this. One, I will need some means to control the creation process by NetKernel and second, the lack of interfaces at this level. I realize I could probabaly implement my own base accessor and in turn achieve some of the same goals but I like the idea of wrapping them post implementation.

I also realize there will be overlay opportunites in 4.x but we are currently working in 3 and will be there for a while. In either case wrapping the accessor would work in both platforms.

Where is the download?

Wed, 01 Oct 2008 10:59:00 GMT

Hi Peter,

Hope the architects weekend went fantastic.
I am itching to see NK4. How do I get it?

Warm regards,
Satya

well ...

Tue, 30 Sep 2008 14:30:00 GMT

Randy,

No worries. Take your time to unpack ...
If I put only one source-request (I tried both source and sourceAspect) in my code it works as I expect :

if level==0:
   outputs="{\r\n identifier: 'shortdesc',\r\n label: 'shortdesc',\r\n items: ["
   context.sinkAspect(temp_uri,StringAspect(outputs))

# So this first source request works
resp=context.source(temp_uri,StringAspect)
output=resp.getAspect(StringAspect)
outputs=str(output.getString())
outputs+= "\r\n  { shortdesc:'" 
outputs+= description 
outputs+= "', numberOfItems:0,"
context.sinkAspect(temp_uri,StringAspect(outputs))
#
# And here I now comment the source request ... 
#resp=context.source(temp_uri,StringAspect)
#output=resp.getAspect(StringAspect)
#outputs=str(output.getString())
outputs+="\r\n  }" 
context.sinkAspect(temp_uri,StringAspect(outputs))
#
# Write final output and again I comment out the source request
if level==0:
   #resp=context.source(temp_uri,StringAspect)
   #output=resp.getAspect(StringAspect)
   #outputs=str(output.getString())
   outputs+= "\r\n ]\r\n}"
   context.sinkAspect(temp_uri,StringAspect(outputs))
   context.sinkAspect(save_uri,StringAspect(outputs))

This works perfectly.
However. You'll notice that I've still got way too many sinks.
This is because this is just skeleton code. In between the two main requests I want to do a lot more processing, amongst other things some asynchronous subrequests that will in turn access the same uri.

So, before each sink I want to make sure I have the latest source (I'm going to add locks and unlocks too). But beyond the first source request none of the others (when I uncomment them) shows up in the visualizer.

Before you show me the error of my ways, let me explain what I want to accomplish. I'm building up a recursive tree (so my script will call itself recursively). I've managed this perfectly in DPML where I build up the tree in a temporary file. I don't have management of that file, curi takes care of that perfectly (and I pass the uri in the param of each recursive request). Now, I want to speed-up my code by using Python (am thinking of asynchronous requests handled by a table). But now the filehandling seems to be my problem ...

Long story. Like I said, take your time, in the mean time I'm learning lots by going to all the netkernel code and trying to find similar stuff.

Regards,
Tom

Visualizer

Tue, 30 Sep 2008 13:39:00 GMT

Tom,

We are just now getting back from the Architects' Weekend and it may take us a little while to unpack and setup in our offices before we can answer your question.

While you are waiting I'll ask this quick question - have you viewed this request processing in the visualizer? You might want to do so to confirm that all sub-requests are being issued, the arguments are as you think they should be, etc.

-- Randy

request visualizer

Tue, 30 Sep 2008 10:13:00 GMT

Greetings,

-- Additional information

I used the Request Visualizer on this code ...
All the "sinks" are there ... but only the first "source" is listed ...
So my "result" is logical ... but I still don't understand why the other "sourceAspect" requests are not done ...

Regards,
Tom

Goodies

Tue, 30 Sep 2008 08:59:00 GMT

Greetings,

Come on Peter ... where can we find the goodies ... :-)

Regards,
Tom (who is getting bad habbits from playing beta versions from online games)

python filehandling sinkAspect sourceAspect newbie question

Tue, 30 Sep 2008 08:55:00 GMT

Greetings,

It's probably a newbie question, but why would this work :

# Write initial output 
if level==0:
   outputs="{\r\n identifier: 'shortdesc',\r\n label: 'shortdesc',\r\n items: ["
   context.sinkAspect(temp_uri,StringAspect(outputs))

#
output=context.sourceAspect(temp_uri,StringAspect)
outputs=str(output.getString())
outputs+= "\r\n  { shortdesc:'" 
outputs+= description 
outputs+= "', numberOfItems:0,"
context.sinkAspect(temp_uri,StringAspect(outputs))

And would this not work :

# Write initial output 
if level==0:
   outputs="{\r\n identifier: 'shortdesc',\r\n label: 'shortdesc',\r\n items: ["
   context.sinkAspect(temp_uri,StringAspect(outputs))
#
# output from this block goes lost
output=context.sourceAspect(temp_uri,StringAspect)
outputs=str(output.getString())
outputs+= "\r\n  { shortdesc:'" 
outputs+= description 
outputs+= "', numberOfItems:0,"
context.sinkAspect(temp_uri,StringAspect(outputs))
#
# output from this block goes lost
output=context.sourceAspect(temp_uri,StringAspect)
outputs=str(output.getString())
outputs+= "\r\n  }" 
context.sinkAspect(temp_uri,StringAspect(outputs))

# Write final output (this output I get)
if level==0:
   output=context.sourceAspect(temp_uri,StringAspect)
   outputs=str(output.getString())
   outputs+= "\r\n ]\r\n}"
   context.sinkAspect(temp_uri,StringAspect(outputs))

These are just preliminary blocks, so yes, I know that its not the most efficient code possible. I basically want to do multiple writes (sinks ?) to the same fileresource at different stages in my code. But I seem to run into some sort of caching (?) problem. Can somebody explain how I should best do this ?

Regards,
Tom