nk4um

Thanks for your work on this Peter!

This is starting to look like a really useful feature! (I can also confirm it''s working on OS X).

If the system already has a module installed with the same version or higher than the same module in the package then the installation halts.

It would be really helpful if this "fail-safe" was optional. Our deployment process is to build our own distribution zip, which we can then deploy to our servers. If an existing version of a module is already installed then it''ll still be in our distribution zip. At the moment, we''re automatically giving every module the same version number (based on SVN version) to get around this problem. Ideally, however, we''d like to give each module its own specific version number (based on the svn version of that folder in our repository) - as I see it, this isn''t possible at the moment.

Another problem that we encountered with building our own distributions, is that if folders are included in the zip then the deployment fails (we get a "Not a folder" error). We got around this by including

filesonly="true"

in our ant zip task.

If anyone is interested, our build target looks like:

<targetname="build-deployment" depends="deploy">   <zipdestfile="${basedir}/dxml-${revision}.zip" filesonly="true">     <filesetdir="${build}" excludes=".*" includes="**/*" />   </zip> </target>

Hi Peter,

I agree that the *nix platform has some advantages. Only a week or two ago
we had a Java developer asking if we (the *nix System Engineers) couldn''t just
convert all our systems into using "\\" instead of "/" since combining both
was playing hell on his (presumed) cross-platform application.

We had a good laugh with that at the time, but in retrospect its not all
that funny ...

Anyway, I''ll do some more testing on a Windows Vista system this afternoon,
but I don''t expect that to behave all that differently (although it behaves
differently in almost everything ... not one of Bill''s beter accomplishments).

Thank you for the work !
Tom

P.S. You''ll find this hard to believe, but I hadn''t actually noted the fact
that I could give a name to the export. I sort of thought I could enter the
name of one module there and found the pick-and-fire list a lot easier.
Just goes to show I have a lot to lear !

Ah thanks Tom. You instantly honed in on an unanticipated use case.  I forgot to consider what to do if a package has been built with no name.

In the packager, if no name is given, then a default package called netkernel-deployment-package.zip is created but the manifest will have a blank name.

In the deployer everything is done with URIs (what else!) which is platform neutral (good to hear it works out of the box on Win32!).  For the target directory it does a URI resolution by turning the package name into a relative directory URI by adding plus a trailing slash and resolves it against the base uri of the installation modules/ directory.  If the package name is blank then the target will be file:///[installpath]/modules/ which always exists so the ''target existence test '' will fail safe.

Is this a flaw or a feature? Well as I said the deployer is very conservative unless everything is sensible it always fails safe, and even if the deployment succeeds but the app itself is not satisfactory it can be instantly rolled back.

Its probably a very good idea for a production process that you should give a name in the package, but I didn''t enforce it in the packager. I''ll add some better reporting for this and modify the packager.

Glad it works for you!

P.

PS file: URIs are very handy since they are guaranteed to be platform neutral especially when being processed by apps like a deployer.  On windows you need to include the drive letter. So, say you had a directory.

C:\\dir1\\dir2\\somefile

Then the file URI is...

file:///C:/dir1/dir2/somefile

You need to remember the 3 slashes at the front (technically the reason for this is that the windows colon : has to be moved up to a point in the URI where it won''t look like a hostname:port pair).

Life is easier on *nix since we have a logically abstracted filesystem and URIs are closely related to *nix file paths anyway.

/dir1/dir2/somefile

is...

file:/dir1/dir2/somefile

You can also use the 3 slash version: file:///dir1/dir2/somefile if you like total consistency.

Greetings,

I had a little testdrive this morning. Here''s what I tried (on Windows XP):
- Yesterday I had been studying the forum modules (and been
  playing with the forum-lite), so my export was those three modules
- Then I ''accidently'' :-) dropped those modules.
- For good measure I also deleted the three .jar files from the modules
  directory (I really don''t know what came over me :-).
- I wasn''t quite awake enough to completely figure out how to get the
  file:// uri to work, so after two attempts I dropped the zip into
  C:\\ and deployed file:///netkernel-deployment-package.zip. A ''browse''
  option would be nice to have (yes, I''m lazy on sunday morning :-)
- Deployment gave me some static, claiming the modules-directory already
  existed. I recalled that was indeed the failsafe check ...
  I quickly found where this was located in the javascript and allowed
  an exception if it is actually the toplevel directory (this was just
  for testing, need some clearer brains to know if that''s actually the
  correct course in all cases).
- Deployment went smoothly and after reboot everything worked again, my
  testposts were still there.

Very very nice, the functionality is there and even most of the polish
is already on. Still a miracle !

Regards,
Tom

A new version is available with package and deploy:

http://1060.org/upload/ext-install-1.1.3.jar

The deployer is very conservative...

If it finds a directory in [install]/modules/ with the same name as the package then the installation is halted.  So always create incremental names like mypackage23.4.etc for packages that are for updating existing systems.

If any of the module dependencies specified in the package are not found in the system or as members of the package then the install halts.

If the system already has a module installed with the same version or higher than the same module in the package then the installation halts.

The deployment tool provides a full report for successful and halted installs.

The deployer writes a transaction rollback record - so if the worst happens you can return the system to the pre-deploy state by using the existing rollback feature.

I''ve tested on Linux but please let me know how you get on, especially on win32 and OSX.

Cheers,

Peter

PS Install is the same procedure as described earlier in this topic.
PPS It took a little longer to develop than the create since the conditions are more thorough and you have to test every step very carefully.  I didn''t develop in one session but I started yesterday afternoon and I suppose all told it took about 5hrs.

That is ... amazing. I definitely must get to the bottom of this. Just for my personal information, how long did this take to develop ?

It took about 2 hours. Most of that was figuring out the java.util.zip package to create zip package!  The actual information comes from existing NetKernel system resources and was just a bit of composing some existing services and slicing and dicing the resources.

I did a quick testdrive and there''s some minor quirks (cosmetic), but it works exactly as I would expect such an application to work (and a lot better than a very specific rpm package menu that comes to the front of my mind now :-).

Good. Please let me know if you have ideas for improvement.

There''s one thing I couldn''t get to work ... you stated that I''d be able to see the dependencies inside the manifest ? I tried this with several modules (the forum ones actually) that I''d expect to have some dependencies to other modules (in case of the forum I expected the other forum modules and the database modules), but I didn''t see this in the manifest. Am I overlooking something ?

The data in the package is normalized - each module contains its dependencies in its module.xml file.  The manifest just provides the extra information needed to install and deploy the set.  The deployer will extract each of the module.xml and manifest to do the dependency reconciliation.

One thing you might not be familiar with from Linux admin is that because NetKernel applications are logically composed resource oriented systems, the dependence on physical implementation is generally decoupled.  Our experience is that dependence on low-level libraries really doesn''t creep into applications.  Applications modules tend to work together and are pretty much impervious to changes in the NK underlying service implementations.

An example is that the 1060.org servers were updated to 3.2 recently (we eat our own dog food!).  Not one of our production applications needed any modification.   In fact apps like http://1060.org/blogxter/ and http://1060.org/bugxter/ haven''t been touched since NK 1.0

Insofar that you still had to prove the flexibility to me ... you did ! Even if you started coding the very minute I wrote the first post on this, it would be impressive ... and I bet you didn''t.

Ah go on.  You''ll make me blush.  I have a running joke with a friend of mine, the catchphrase is: "It is not a miracle, It is science!".

Hi Peter,

That is ... amazing. I definitely must get to the bottom of this.
Just for my personal information, how long did this take to develop ?

I did a quick testdrive and there''s some minor quirks (cosmetic),
but it works exactly as I would expect such an application to work
(and a lot better than a very specific rpm package menu that comes
to the front of my mind now :-).

There''s one thing I couldn''t get to work ... you stated that I''d be
able to see the dependencies inside the manifest ? I tried this
with several modules (the forum ones actually) that I''d expect
to have some dependencies to other modules (in case of the forum
I expected the other forum modules and the database modules), but
I didn''t see this in the manifest. Am I overlooking something ?

Insofar that you still had to prove the flexibility to me ... you
did ! Even if you started coding the very minute I wrote the first
post on this, it would be impressive ... and I bet you didn''t.

Regards,
Tom

Hi Tom,

Thanks for the detailed insight into your business processes - this is very very helpful.  I added the first 50% of what you want to the installer application.  Please take try this update out...

http://1060.org/upload/ext-install-1.1.2.jar

You can this update installer using the existing installer:

http://localhost:1060/install/add

Select "Install a new module from a URL" and paste the URL in. Once you''ve restarted try the updated installer:

http://localhost:1060/ep+name@app_install_installer

You will see a new "Package Manager" option.  I''ve added a tool to create an exportable package.  This creates a zip containing the modules you want to package and a manifest.xml that shows the dependencies and fulcrum attachments. When I get chance I will add a deployment tool that will take this package structure and do an install/update with rollback.

I hope this is along the lines of what you were looking for.  If you''re curious about the code the source very small and is in /org/ten60/netkernel/ext_install/packager.

You''ll see from the code that NK has all the infrastructure to do what you want. It is all exposed as services and resources so you can quite easily manage and adapt NK using the regular tools.

All feedback very welcome.

Cheers,

Peter

Hello Randy,

No worries about the delay, it''s very kind of you to take time to look
into this. I''ll try to answer both of your questions :

1) The notion of "split" environments and "split" roles is deeply ingrained
in my company. So obviously when I look at a <trying to find one word that
describes Netkernel ... and failed> like Netkernel, ease of deployment of
applications, of moving applications between instances is always in front
of my mind.
Would this block adaptation ? No, we''ve written deployment systems before,
and we could do so again. But it would be a very nice to have.

2) If we take Netkernel itself as an application, it has dependencies
- It requires a Java runtime (and that must be higher than a certain level)
- It requires ...
So in order to "deploy" Netkernel (which comes in a nicely packaged form,
I installed it on a RedHat 4 update 4 VM earlier today in less than 20
seconds, had it running in another 5), those dependencies have to be
fullfilled.

I see two major points in this :
-> The application (autoinstall/autoconfiguration) package. Preferably
   one file for easy transport. Installs itself in the correct directories
   (creates these if not there), registers itself where needed (fulcrum,
   deployedmodules) ...
BUT
-> It only does the install if the dependencies of the package are
   fullfilled. You may laugh at this, but that RedHat system I mentioned
   above did not have a JRE. So I first had to push the rpm''s that
   installed the JRE and only then would Netkernel "talk" to me.
   When/if I re-package Netkernel for RedHat 4 in rpm form, I will
   therefore add the JRE requirement, so a System Engineer that does
   not know Netkernel in detail, will not have to search why the
   application does not work.

The keyword ... dependencies. Yes, I agree with your definition of
application. A dependency can be something in the Netkernel system
(I need the ruby 0.9.9 module to execute Ruby scripts for example),
or another self made module (my shop-application does not work
unless the atm-module is there).

Whether or not the shop-application package should include the
atm-module is of course another debate. Possibly yes (if you would
sell the shop-application as standalone for example), possibly no.



Hopefully I''m not boring you to death with these "musings" of mine,
I just found it strange (reason for that, see 1.) that I couldn''t
find it in Netkernel.

Regards,
Tom

Tom,

Sorry for the delay - I was checking the details to give you the most accurate answer possible.

NetKernel has very good grip on the meta information related to a module. In fact the Module Wizard and the XUnit testing tools use this to do their work. The Module Wizard and XUnit are written in NetKernel using standard code, which means these and similar tools can be written by a NetKernel developer.

If having a module-level facility that would move one or more modules from one running instance of NetKernel to another is something that is critical for your adoption of NetKernel, please let us know and we will add this capability.

I also want to be clear about your question(s) as they relate to the notion of an application. By application I mean a set of related modules which constitute a complete running functional process. NetKernel does not currently have meta data that understands this. Since you have raised this as a question, it is now on my list of desirable features for a future release of NetKernel.

Randy

Tom,

Thank you for your replies and the details in your messages.

The short answer is that we can guide you towards a deployment process that will work for you and your company. As you either know or may have guessed, NetKernel is built on NetKernel - so we have the interesting task of creating a version, testing it, and deploying it - in an installable set of JAR or TGZ files.

I want to make sure I answer your question with sufficient detail so I am taking a little extra time to check all of my facts.

Randy

Randy,

I believe the expression for the above reply is "I went completely overboard".
My apologies for that.

Lets state it in a simple way what I would like for Christmas :-) :
1) I want to be able to "export" a module/application,
   with its dependencies. Not only "I" want to be able to do that,
   an operator should be able to do so with a press of a button/
   simple script. This should not impact the module itself.
2) I want to be able to "import" that export in another NetKernel
   environment. If the dependencies are not fullfilled, it should
   not load (so that the environment does not have a disfunctional
   module in it). If the dependencies are fullfilled, it should
   load and be active with the next boot (preferably hot, possibly
   cold).

Ok, much beter ...

Regards,
Tom

Randy,

You''ve very accurately described our environment (how we roll out
java applications today) :

- A developer works on his own workstation. Very little in the way of
  data input available there.
- Once the application has passed testing it is then given in the
  hands of our Application Infrastructure group. They typically modify
  the datasources, determine the application server to be used, and
  off we go to the test environment. Still not a lot of data there,
  but this is where database-access is tested.
- Once the application has again passed testing, it goes off to the
  system test environment (this is automated, operations does it).
  Here the application is stresstested. Production load of data,
  production load of users (simulated at least), interaction with
  other application. Typically also the first decent batchtests.
- If the responsible System Architect gives the green light, the
  application then goes to production. Again, this is an automated
  process, invoked by operations.

The above may be a bit overkill in the way of explanation, but the
point I''m trying to make is that I need to know how to move/promote
one application :
- The module itself (easy enough I guess)
- The fulcrum files need to be altered
- The deployedmodules needs to be altered
- ...
Still doable I guess ... but now it gets murky
- Does the module depend on other modules
Or lets make it harder ...
- Does the module(version) depend on other module(version)s

What if we loaded a couple of extra modules in the test
environment (typically for database access) ... that we then
"forget" in production.

Deployment is hard. I do not have experience in java deployment
(although I have worked with our Application Infrastructure
group, hence the explanation above), but I have extensive
experience in rpm packaging and deployment ... and that''s no
piece of cake.

For myself I''d be happy to have an automated way to move an
application and update the necessary files. But ... if you
want the "promotion" process fully automated (and in big
environments you do) you''ll have to go further than that.

All of this is my personal opinion on the matter of course,
I''d love to hear your feedback on it.

Regards,
Tom

Tom,

This sounds interesting and I may want to write this up as a use case for the next version of NetKernel.

Are you suggesting that there is a workflow with multiple roles involved? Let''s use this:

    * Developer creates a module and needs to test it. Developer installs module in a test instance of NetKernel, runs tests, debugs, and then reaches the point where the module is ready for release.

    * System Tester then gets the module and tests it against the whole system in another instance of NetKernel. Once the whole system passes tests, the module is ready for release in the production instance of NetKernel.

    * System Administration then gets the module and is responsible for release into the full production system.

Is this close to your scenario? If not, please help me get this right before we move on to how this would be done in NetKernel 3.2 and if there are deficiencies, we''ll figure out how to make this work better in the next release.

Randy

Greetings,

Thank you for the information. I was thinking more along the lines of
"promoting" one module to a different netkernel instance. So there''s :
- an entry in [install]/etc/deployedModules.xml
- an entry (multiple entries ?) in the Fulcrum''s
- the [module] directory itself underneath [install]/modules
- ??? dependencies ???

I was also thinking of an "organised" way of promotion (sorry if this
is not the correct English expression for this), where a developer
can "promote" from his own instance to the test environment, but
promotion from test to production is done by a different person/group
(that does not even have to know how the module is build).

Regards,
Tom

It is very easy to transfer modules from one installation to another.

The [install]/etc/deployedModules.xml file contains a reference to all modules known to the NetKernel installation.

All you have to do is copy that file and your modules from one machine to the other.

Note: It would be best to do this when both instances are not running.

Note: If you put all of your modules in the [install]/modules directory, then you could probably automate this transfer with a script file, creating a ZIP of the modules directory.

Greetings,

There may/must be an obvious way of doing the below ... but I cannot find it ...

I''m currently developing at three different locations, three slightly
different machines (two XP, one Vista). Of course I''d like to
"carry" my applications with me, so I can develop wherever I am.

So I went looking for the "package application" and "deploy packaged
application" options ... But either I''m overlooking them completely
or they are not there (or there is another obvious system to do this
that I don''t know of).

Is there a way to do this ? How do I bring an application from a
test-environment to a production-environment ? Or what obvious point
am I missing here (this is very likely the correct question) ?