nk4um

Yes things look really good and no it''s not critical path for me to have your "cluster manager" support if it comes to pass that initial project only uses a two machine cluster like I mentioned (we already use WebSphere admin pages to do deployments separately for each machine in that cluster so if we used apposite to update each machine we''re no worse off than today).

When I say "things look really good" I should confess that I didn''t originally appreciate this "fine grained modularity" that NetKernel supports via it''s modules and URI "spaces", i.e. how powerful it is when you can create these "vertical slices" through your application (as you put it in your ROC Architecture tutorial), with the ability to individually deply just the UI portion of your app as a module, or just the persistence portion (of course I''m talking "horizontally" now aren''t I), or even just a portion of the UI for the app (e.g. one tab of the overall application) - that''s getting more at what you mean by a "vertical slice" right.

I truly think this is very good stuff.  We''ve struggled with WAR files and how they don''t let us easily do these things, and I know that OSGi/SpringDM/Grails plug-ins etc. are all speaking to similar problems, but my impression of OSGi/SpringDM (and this may be unfair I haven''t spent a lot of time with them) is that they''re less elegant/more complicated than NetKernel''s notion of modules. 

On the Grails plug-in side they do have a very simple/clean design that is closer to what NetKernel has, but I became a little worried when I tried some plugins written for a prior version of Grails that didn''t seem to work under the new version, and because of an implementation decision they made where they organize the "plugged-in" files outside your project''s working area in what to me was a confusing way. 

Although I have to say outside of NetKernel Grails is also very exciting to me - e.g. the way it''s using the Rails-like "convention over configuration" philosophy to dramatically simplify things esp. for newbies.

But at the moment certainly considering the heavy of use of XML/XSL, as well as "SOA" at my company, along with some of the challenges we have with deployments of our (pretty darn complicated!) systems, I tend to feel NetKernel brings more to the table than anything else I''ve seen.

Hi Darren,

You are keen!  To answer the following...

Would something like this be even remotely possible?  And assuming our needs are unique and that''s not standard NetKernel functionality, is Apposite possibly written in a way that I could write scripts to go around the UI and create something like I described myself?

Yes this is completely possible - the core of the NKSE system is written to be operated both as a user application (apposite-client) and as a remotely managed service (via a cluster manager/monitor) (In fact everything you see as an application in NKSE is the same - that''s just a natural consequence that these apps are themselves using ROC).

We are in the process of working through the feature set of the cluster manager now - if this is on your critical path we can get you early access.

Hopefully you saw from the video that 90% of the iceberg is in place, packaging, package management, repository development, apposite client.  A multiply redundant cluster manager, although mission critical, really is a relatively small part of the whole solution.

Cheers,

Peter

I just noticed this thread spoke to part a. from my questions above:

http://www.1060.org/forum/topic/54/1

I just watched the videos they were very helpful I see there''s a lot more to Apposite and the package manager stuff than I''d seen before...

I guess the only questions I can think of are:

a.  The Cocoon system you and I spoke of runs on a 2 machine cluster, and there''s another system that potentially/maybe/possibly down the road could see use of NetKernel - that runs on a 6 machine cluster.  Without some custom/special process developed, deploying to such a cluster does require going to Apposite running under the 1060 admin pages of each machine in the cluster?

The two machine cluster obviously doesn''t seem much of an issue but thinking of the 6 machine system, in an ideal dream wish-list world, I was thinking it would be really helpful if there was a way to e.g. use a single machine as kind of a "staging" machine, follow the process you demoed to upgrade that machine''s modules, test that machine to make sure all is okay, and if/when all checks out, click a button that causes all the other machines in the cluster to be automatically updated so their module versions match that of the initial test machine.

Would something like this be even remotely possible?  And assuming our needs are unique and that''s not standard NetKernel functionality, is Apposite possibly written in a way that I could write scripts to go around the UI and create something like I described myself?

b.  Briefly to clarify a little why a feature like above is attractive - it''s *mainly* because we have so much data in Oracle that it''s not at all uncommon for code to test out when it''s hitting our test databases only to find when it''s deployed in production against the full-blown database things don''t work the same.

And this leads me to another question:  With so many of our services front ending Oracle, though we often make changes in the course of upgrading (our equivalent of modules) - changes that are backward compatible from the database level (e.g. the addition of a new nullable column in a table), there are of course times when the database changes are not backward compatible.

I was trying to figure out whether/how the nice deployment and rollback capability could work in these cases where the database changes are being made independent of NetKernel''s module deployment process.  i.e. Potentially "rolling back" such database changes may be a challenge even with NetKernel''s nice features having made the "code" part easy to roll back.

Do you have suggestions/advice regarding this database aspect of deployments?  e.g. Have you heard of clients using NetKernel maintaining duplicate databases where they always have one version back of their database schema in case they need to rollback?   That seems extreme and a lot of work for the dba to mirror the database with each deploy, but is there an alternative?

c.  Maybe too broad a question and not really on the topic of deployments, but I''ve been real curious if the focus of REST and ROC on "resources" like XML etc. as "documents" encourages designs that truly reduce the use of databases like Oracle, i.e. to avoid some of the headaches I alluded to in b. ?? 

But there are cases where flat files representing the resources aren''t sufficient right, e.g. the database does give you nice querying/searching abilities...  but is this where your support for Lucene fills a gap? 

Just wondering - relational databases are such a staple at big companies... are you finding people using NetKernel moving at all away from storing everything in them?  And/or are they sometimes just storing "resource" data as a BLOB/CLOB in relational tables?

Even better with Dolby surround sound!

The NetKernel''s "Apposite" repository architecture is very powerful.These videos should help see the end-to-end process...

Part 1
Part 2
Part 3
Part 4

(Best viewed in HD)

Don''t know that I followed all that you said (the part about stem cells :), but I''ll mention that the way you''ve set up your scripting languages using e.g. the Groovy compiler as the "transreptor" applied to the original groovy source code "resource" put me in mind of e.g. a book from a few years ago that I don''t got much attention called "Programming Mobile Objects with Java", or ideas I''d read about long ago about software "agents" that were little programs that could move around a network server to server.

More specifically:  Do you guys literally do anything where in a clustered environment, instead of people "pushing" the software out to the servers, you actually put e,g, groovy scripts or java jars, or complete modules in a networked location and let the nodes in the server cluster "pull" the changes to them?

This seems in many ways ideal right?  The biggest concern I could see is if by having all the nodes in a cluster reference a single machine for their deployment you''ve made that a single point of failure.  But that''s not true during the duration of a program''s run, if the code pulled over is in the NetKernel cache, and it wouldn''t be true even on multiple runs if NetKernel was using an L1/L2 cache where the L2 cache is a persistent (e.g. file based) cache.  And you said that''s real easy to do anyway...

Is this what you''re saying you''re already able to support?  Or the direction you''re headed?

You got it.  This is exactly what the infrastructure in NKSE is designed for.  FYI Later this week I''ll be posting a video of the end-to-end cycle: module creation, packaging, package management, repository design, publish, deploy, production server update.

We''ve been managing our in-house systems with these tools for about a year now - we''ve got our RTM process down to about 2-mintues.

Watch this space...

P.

PS As far as cluster management - we have some cool stuff here.  We view cluster (cloud) deployment/management as like biological stem-cell differentiation. A cluster node starts life as a stem-system. Either by explicit instruction, or through local context,it is directed to become differentiated (ie take on (deploy) a software image).  Monitoring control nodes can manage failover, where a standby stem-system is hot-differentiated to replace a failing node.

The repository model provides the foundation for this and is actually a resource-oriented model of providing the software package resources.  We have a full end-user package management application that manages package versions, supports multiple repositories, versions and repository targets (test, staging, production etc) and which will publish a full, securely signed (with each package also individually signed) repository image.   This will be shown in the end-to-end video later this week.

Newbie questions about deploying NetKernel when used in a cluster:

a.  The idea of apposite is that an enterprise can have their own repository of modules and they can be installed via apposite just the way NetKernel modules get installed - right?

b.  Assuming a is true, in a clustered environment running multiple (identical) NetKernel instances, does an admin have to use the 1060 control panel individually on each machine?  Is there any slick way to update multiple machines all at once?  Or maybe not?  Or if apposite is "RESTful" itself, can you write something to go around it''s UI and remotely invoke it''s functions from a script?  i.e. To automatically update modules on a whole cluster of machines?