Module.xml as a "first class" (transreptable) resource

Poster Content
nk4um User
Posts: 112
February 3, 2012 18:21alternate syntaxes

Allow me to quietly add my +1 to this idea. The complexity and verbosity of even a minimal module.xml does turn people off.

FWIW, my approach is to have a consistent implementation pattern where a particular uri is mapped to a particular script implementing the behavior by a naming convention, and generally importing the same set of modules. This pattern covers about 90% of my needs, so I just wrote a simple script that takes 1 or 2 parameters and generates the boilerplate module.xml. It's sort of like the new module wizard in concept, but customized for my use :)

Like · Post Reply
nk4um Moderator
Posts: 595
January 20, 2012 10:59

camps that are probably going to be closed minded about a graphical programming approach. I should shut up maybe I'm betraying my own bias and doing some kind of freudian "projection" here. :)

Yes, like Tom, I think you are right. That's why we have been a bit cautious pushing nCoDE. There are a lot a very limited and effectively useless tools put at to satisfy a marketing and sales need. I hope nCoDE is not that, I've always used Labview as my benchmark and that is definitely not a useless tool.

nCoDE certainly hides many of the details of working with NetKernel but certainly not all of them. Also to be clear nCoDE doesn't do any of the space orchestration stuff that module.xml does. What it is is a visual representation of process flow/ service composition and this respect it hides none of the power of the ROC abstraction. The long term vision is to have unification of space explorer structure diagrams being editable combined with nCoDE into a full graphical IDE but of cause that is non-trivial and we need to work in an incremental way.

Tony

Like · Post Reply
nk4um User
Posts: 131
January 20, 2012 08:19Freudian

Greetings Darren,

camps that are probably going to be closed minded about a graphical programming approach. I should shut up maybe I'm betraying my own bias and doing some kind of freudian "projection" here. :)

Nothing to be ashamed of. I don't know how many times you've been handed the "ultimate development/management/..." gui tool that would solve all your problems, but I lost count along the way and I've been disappointed every single time. Hence my own bias and late discovery of nCoDE.

If follow the reasoning for a more concise module.xml. A dynamic one is - because of the Chicken-Egg problem - not likely at the moment, but if we could have a single transrept to and from a limited number of representation types ... I'll definitely have to take a look :-)

Thank you for the kind words on the book.

Regards, Tom Geudens

Like · Post Reply
nk4um User
Posts: 60
January 19, 2012 20:42

Posted by Tom Hicks (View)

Interesting! I played around with this same idea a couple of years ago, too, but quickly decided that the Groovy Builder mechanism allowed the programmer to co-opt too little of the syntax (really just the call syntax) to make a nice DSL. I've been wondering lately if the ever-expanding Groovy AST capabilities would allow one to utilize a wider range of syntax to build one's own DSL?

Not being serious but only because it sounds like you'd appreciate it... I was googling yesterday and stumbled on this thing I'd never heard of, an add on for Common Lisp that uses macros to actually make XML really and truly be executable/modifiable right within LISP programs because it really is LISP just with this syntax sugar allowing XML syntax instead of S-Expressions (they call them X-Expressions).

I'm not really a LISP guy I just thought that's really cool they could do that.

XMLLisp

X-Expressions

Like · Post Reply
nk4um User
Posts: 60
January 19, 2012 20:31

Thanks Tom congrats on your book btw I have been reading it I was looking at your nCode chapter last weekend.

Though I confess I have not made myself sit down and try it hands on I know I need to.

I didn't withhold my comments only from the perspective which you touched on too - that this stuff is subjective and more about certain groups of developers having certain preferences and biases and things they like and don't like.

Which is to say that what I've seen of nCode I am super impressed with I don't know if the guys found some great tool that made that somehow easy to build or how they pulled it off... and yet as you suggested - there are certain camps that are probably going to be closed minded about a graphical programming approach. I should shut up maybe I'm betraying my own bias and doing some kind of freudian "projection" here. :)

But I feel like the node.js guys would want this stuff in JSON, the Rails guys would likely want more of a "convention over configuration" approach, some of the java Spring guys would probably hope for annotations in the code nowadays... as wonderful as nCode looks it seems like not from a technical but from a political and sales/business point of view it feels like the market would be bigger if that was complemented with a concise textual/code approach.

But I hear yeah I will have to try nCode. And I promise to buy your book btw. As opposed to pirate it or something. :) I figure I want to support the cause! :)

Like · Post Reply
nk4um User
Posts: 9
January 19, 2012 20:20

Posted by darrencruse (View)
Posted by tab (View)
It's very hard for Peter and I to see perspective to refine the syntax of many of these technologies. We're usually more focused on the semantics of what we create than the syntax. It's the semantics that make NetKernel unique. The syntax is always playing catchup.

I had written groovy scripts a couple years ago that could convert module.xml to groovy "markup builder" syntax and vice versa, along with a bit of DSL'ish treatment to like the grammar stuff. I named them "module.gml" files. The idea was to edit the module.gml in lieu of the module.xml and simply convert them to module.xml independent of NetKernel.

Interesting! I played around with this same idea a couple of years ago, too, but quickly decided that the Groovy Builder mechanism allowed the programmer to co-opt too little of the syntax (really just the call syntax) to make a nice DSL. I've been wondering lately if the ever-expanding Groovy AST capabilities would allow one to utilize a wider range of syntax to build one's own DSL?

I agree with you about the verbose XML syntax and you've given some good examples of places where a good DSL could be much more concise.

   -tom H
Like · Post Reply
nk4um User
Posts: 131
January 19, 2012 19:54nCoDE to the rescue

Greetings Darren,

For being a bit rusty that is an impressive list of - valid - suggestions. I had a hard time keeping up :-).

The very first non-typo feedback I got on the Practical NetKernel book was : "Chapter 2 is way too long, can't you do something simple first ... like a Hello World?". The reader had lost the thread in the module.xml before even noticing Chapter 2 IS a Hello World example.

It is true, module.xml is a big unwieldy thing. On the other hand, once you get through it, you are practically done with your module. However, that doesn't help if you're doing a demo and the guy before you pulled out some node.js/ruby rails/... rabbit from the hat. It is comparing apples to oranges, but the audience doesn't (want to) see it that way.

(maybe something like) nCode is an uber-example of this kind of thing

Yes, it is. And it works. And I heavily underestimated it. Because once you get to the point of being a "poweruser" you are quick to see it as a gui toy. nCoDE is an answer to many (if not all) of your questions. Other answers can be build (and I'm looking into the pointers Tony gave to see if I can help with some of your questions), but nCoDE is very close to being the ultimate compose tool and I for one am rather ashamed I did not recognise it as such right from the start. Use nCoDE in a demo and I dare say the playing field is better than level.

I also think - my personal opinion - recent NetKernel developments (simple grammar, antlr, ...) show that your concerns are shared and (being) addressed.

Regards, Tom

P.S. XML, JSON, ..., there will always be a text based (human readable) data interchange format. It is first and foremost important that we can use them all inside NetKernel (and we can).

Like · Post Reply
nk4um User
Posts: 60
January 19, 2012 14:24

Posted by tab (View)
It's very hard for Peter and I to see perspective to refine the syntax of many of these technologies. We're usually more focused on the semantics of what we create than the syntax. It's the semantics that make NetKernel unique. The syntax is always playing catchup.

I had written groovy scripts a couple years ago that could convert module.xml to groovy "markup builder" syntax and vice versa, along with a bit of DSL'ish treatment to like the grammar stuff. I named them "module.gml" files. The idea was to edit the module.gml in lieu of the module.xml and simply convert them to module.xml independent of NetKernel.

I'm the first to admit it was an experiment though - and possibly a failed one. :) i.e. I had not dramatically shortened the line count and the groovy markup builder syntax is an acquired taste - it might have appealed to groovy/grails types of folks but been off-putting to other people who hadn't been exposed to groovy markup builder before.

This is the flavor of what I'd been able to do at the time:

   mapper {
     config {
       endpoint {
         grammar { text 'res:/book/velocity/'; any 'value'; }
         request {
           'SOURCE' identifier: 'active:groovy',
           	operator: 'res:/resources/velocity/hello.groovy',
           	valueAsString: 'arg:value'
         }
       }
     }
     space {
       fileset 'res:/resources/velocity/.*'
       importSpace 'urn:org:netkernel:lang:groovy'
       importSpace 'urn:org:netkernel:lang:velocity'
     }   

Like I said - I'd originally hoped that these techniques could really help shorten things up and mostly I did't feel I succeeded and I'm not 100% sold on the groovy markup builder syntax myself.

But just looking at the above for discussion sake, I do like how short the imports are. I've always felt NK imports would be related to by new newbies as analogous to #include in C, import in java, etc. I feel like having those short/sweet single lines reinforces that notion and makes things feel a little more familiar.

But putting these wacky ideas aside and looking back at module.xml, I feel like the space import stuff is really verbose:

<space>
  <fileset>
    <glob>resources/tmplt/*</glob>
  </fileset>
  <import>
    <uri>urn:org:netkernel:ext:layer1</uri>
  </import>
</space>

And I did come to recognize that there are options (private versus public, regex versus blog, etc) that explain why it is this way, but in my case I was rarely using those options. Would it be out of the question to add some options to use xml attributes instead e.g.:

<space>
  <filesetglob="resources/tmplt/*" />
  <importuri="urn:org:netkernel:ext:layer1" />
</space>

Even though it's still xml the shorter version feels more readable to me.

This also illustrates why I'd started thinking of some kind of "xml rewriting"/"xml macro"/xsl-transform kind of approach to the module.xml. i.e. The attribute approach could obviously be rewritten to the standard expanded-out tags.

I think the grammars are another area where the xml syntax had felt verbose to me, e.g.:

<grammar>res:/bb/book/section/
  <groupname="isbn">
    <regextype="anything" />
  </group> /
  <groupname="eid">
    <regextype="anything" />
  </group>.html
</grammar>

But maybe I'm forgetting - you do provide an alternative non-xml regex type of syntax too (I think? I've forgotten - I was doing the above as it seemed the recommended way right).

From my server side javascript exposure I'm noticing a lot of these "Sinatra-inspired" web frameworks that do a :placeholder syntax that seems to be popular now i.e. it would be like:

<grammar>res:/bb/book/section/:isbn/:eid.html</grammar>

As I said I feel like this is kind of applying that "80/20 rule", i.e. I know NK has more power than the above alone supports (e.g. you let the type of the placeholders to be specified like number versus string right), but maybe if this works for 80% of the uses and is what newbies first see, then maybe the xml grammar approach could remain more like the "power user" approach? Just thinking out loud...

Lastly I'll mention one more thought that I'm guessing is frought with technical difficulties that I'm just too naive to see at the moment, but when you consider the module.xml for submitting requests e.g.:

<request>
  <identifier>active:java</identifier>
  <argumentname="class">resources.java.PDSUpdatingBookCrawler</argument>
  <argumentname="isbn">arg:isbn</argument>
  <argumentname="eid">arg:eid</argument>
</request>

Other than the verbosity, it feels like what I love most about NetKernel - that it's a generalization of the web and a generalization of http based REST, is getting - kind of - "obfuscated".

And I vaguely remember that the above xml style is a different way of conveying your "active url" syntax right e.g.:

     active:xslt+operator@file:/style.xsl+operand@file:/document.xml

But again, there's something about my brain that is thrown by the pluses and the at signs.

I feel like the really great strength that NetKernel is like "writing programs the way you wrote web apps" is really hard to see with those syntaxes I think newbies and typical programmer types would be incredibly helped by shoe-horning this into what looks familiar to them in their common use of urls, i.e. something like:

   java://resources.java.PDSUpdatingBookCrawler.class?isbn=:isbn&eid=:id
   xslt://style.xsl?operand=/document.xml

Where the above is just rough ideas and I know I'm overlooking issues, encoding, the fact that you can nest URIs within URIs etc.

Maybe the nesting (and some of the encoding and even some of the verbosity) issues would best be addressed with some kind of aliasing you have the notion of endpoints having ids already right so my brain would parse easily the idea of @endpoint-id as the value of a query parameter as as a form of nesting:

  xslt://style.xsl?operand=@bookTransform

Sorry the above is probably naive but it seems to me if something that looks more like a regular url could be made to work at least for some of the most common use cases, at least as an optional alternative to the others, it feels like to me it helps to make NK much more relatable and approachable.

And it seems like all three of these things could even help to make module.xml shorter and easier even remaining as xml.

Like · Post Reply
nk4um Moderator
Posts: 595
January 18, 2012 14:09

Posted by darrencruse (View)
would it be possible for the "main"/"master" module.xml file to be static and yet these xml files included by it be actual resolved requests (i.e. what might be dynamic requests)?
The short answer is no. I expect what you remember is examples of overlays or accessor that take configuration parameters pulling those parameters from resources rather than inline literals within the module.xml

And lastly I swear I'm a NetKernel fan it's not my intention to sound negative, but it is true that when I gave my NetKernel talk the most critical feedback I got was that (some) people felt overwhelmed by the length and amount of the module.xml configuration.
Understood. XML is a verbose way to express things. However it is easy to consume by NetKernel and that allowed use to be very free to explore capabilities and technologies rapidly. In turn that allowed us to create some crazy ideas, the ideas that make ROC different and valuable.

We do understand that it's now starting to become time for us to invest more resources in syntax and ease of use.

Like · Post Reply
nk4um Moderator
Posts: 595
January 18, 2012 14:00

Posted by Tom Hicks (View)
I've always thought it would be even better if the processes of Space management and the module lifecycle (especially instantiation) were abstracted into a clean, solid API.
This already exists, see org.netkernel.layer0.boot.IModule and IModuleFactory. It is completely possible to create and deploy alternate module implementations. We have an unreleased "Classic" module implementation that partially implements NetKernel3 modules, also the encrypted modules used by NetKernel 5 Enterprise uses a subclass of StandardModule to support PKI encrypted jars.

However implementing modules and factories is non-trivial. You can argue that is should be trivial and I would agree. Unfortunately implementing the standard module to support arbitrary interdependence of spaces whilst maintaining robust initialization is complex.

The best pathway that I see in the medium term to non module.xml modules and/or dynamically instantiated spaces is have a space representation that is created from module.xml in the way branch-merge does it and create transreptors to alternate compact and noise-free syntaxes.

BTW I'd love to see you or anyone else create a working space definition language. It's very hard for Peter and I to see perspective to refine the syntax of many of these technologies. We're usually more focused on the semantics of what we create than the syntax. It's the semantics that make NetKernel unique. The syntax is always playing catchup.

Like · Post Reply
nk4um User
Posts: 60
January 17, 2012 20:11

Thanks for your comments.

Just one quick follow up.

I apologize again that I'm a little rusty with me NetKernel skills so the following is a wee bit vague, but I remember a number of the NK examples broke their module.xml files up such that a main module.xml file was pulling in other xml files. I can't remember now if that was specifically for the xml "configs" used with some of the accessors or if it was something even more general. My real question: If I'm remembering that right, would it be possible for the "main"/"master" module.xml file to be static and yet these xml files included by it be actual resolved requests (i.e. what might be dynamic requests)?

Also I do understand the concern about this opening up crazy changes to what is thought of as the Netkernel bedrock (the format of module.xml). I'll just mention the thought that this reminds me a little of Lisp macros, which I don't claim to really understand. :) But from what I've heard Lisp macros are something that once you understand them make Lisp into something like a "language toolkit" where you can effectively raise and reshape the language to match your particular problem domain.

I think this idea is similar in that it'd be a very powerful tool that would place Netkernel in a similar kind of realm. (ok I admit I'm really selling it I know but you got to try right :).

And lastly I swear I'm a NetKernel fan it's not my intention to sound negative, but it is true that when I gave my NetKernel talk the most critical feedback I got was that (some) people felt overwhelmed by the length and amount of the module.xml configuration.

For me the real goal of a capability like this would be that module.xml as it's expressed today would be almost like "xml assembly language" that could be abstracted/simplified.

Last thought (going back to the Lisp macro analogy): Lisp has the notion of "reader macros" that happen specifically at load time of the source code and not later when the program is running.

An analogy here would be something like a bootstrap module that implements a "templating"/"xml macro" type of capability prior to the other modules being loaded, with just this limited capability applied to the module.xml files as they are loaded.

Like · Post Reply
nk4um User
Posts: 9
January 17, 2012 19:10Space/Module instantiation API

Rather than just a resource, I've always thought it would be even better if the processes of Space management and the module lifecycle (especially instantiation) were abstracted into a clean, solid API. Given such an API, one could then build new types of modules in whatever languages one wanted; eschewing the verbose, noisy XML of the Standard Module altogether.

Like · Post Reply
nk4um Moderator
Posts: 595
January 17, 2012 16:36Chamber of Secrets

Hi Darren and Gregoire,

I see the desire for this and I feel it too! It is as you identify, Gregore, a chicken and egg, bootstrapping issue indeed.

That said we do have the capability to dynamically instantiate space based on the module.xml schema dynamically at runtime. If you look at the internals of the branch-merge accessor this is how it operates. It dynamically creates XML to instantiate the inner spaces it manages.

That said we've held back on exposing this functionality publically. It does have limitations and complexities. The lifecycle of spaces is complex and must be managed carefully. Also spaces need class loaders and Java doesn't work to well with this degree of dynamism - a workaround it to take a surrogate classloader from another static space. It is also, and this is the main reason, quite a complexity leap for this capability to be unleashed on applications, we don't know what kinds of bad and crazy non-maintainable patterns might be created. It's something that really needs to be packaged into a safe space runtime accessor and for a set of basic patterns to be explored and documented. The reason this hasn't happened is priorities. We think that maybe we've already exposed to many crazy new concepts to absorb to be productive quickly with ROC and making one of the last stable bedrocks at the moment dynamic makes things even mind-warping.

However now I've told you the location of the secret sauce have a dig and see what you can create. I'll be interested to see what you come up with!

Cheers, Tony

Like · Post Reply
nk4um User
Posts: 155
January 17, 2012 16:14

Hi Darren,

As Tony wrote in this discussion, this is indeed a chicken and egg problem. Quote :

(View)

The module.xml is a static resource which is loaded by the module infrastructure and doesn''t support loading resources directly because it is actually loaded before the address space comes into life based directly on it''s contents (chicken and egg) There it would not be possible to import like functionality in it.

But I agree that it could be useful if "module.xml" was just another resource.

Gregoire

Like · Post Reply
nk4um User
Posts: 60
January 17, 2012 07:55Module.xml as a "first class" (transreptable) resource

Guessing this might be a chicken and egg problem technically, but It'd be awesome if the information "behind" module.xml could be generated dynamically i.e. if module.xml were able to be generated as a resource like any other.

This would allow DSL's or alternative syntaxes for module.xml to be created (maybe something like nCode is an uber-example of this kind of thing).

A concrete example of what this could allow would be to represent the module.xml information as JSON instead of XML.

e.g. Tom's book shows the use of Mongo with NetKernel and a groovy endpoint, but I've been exposed more this past year to the node.js community which basically embraces JSON as an "anti-XML".

I think it's a testament to the design of NetKernel ROC that it's pretty easy to imagine using Mongo with NetKernel and Rhino, and with all representations and configuration done using JSON instead of XML, yet all the concepts still apply.

With such a "module.json", and a little help e.g. a module for using mustache/handlebars templating, JSONPath, etc. NetKernel might be more appealing to the JavaScript "community".

And it would be pretty neat if such a thing were doable without rewriting NetKernel's core - it seems like the inability to treat module.xml as any other resource (and do transreption on it) is what prevents this today.

Like · Post Reply