nk4um

On the one hand,it [URL=http://www.escortstorontodirectory.com]toronto asian escort[/URL] will make children lack independence and initiative,which is unfavourable to [URL=http://www.escortstorontodirectory.com]toronto escort[/URL] their development in the long run.On the other hand,in many cases parents'' decisions are not [URL=http://www.escortstorontodirectory.com]toronto asian escorts[/URL] always in accordance with children'' personal ideas and interests,this will [URL=http://www.escortstorontodirectory.com]toronto escorts[/URL] make children form resistence and naturely influence their growth.

I read this to mean you have more than just the services I''ve seen exposed to the front-end fulcrum.  In which case - yes the global transport throttle will cause all requests to slow because of the rogue set in this module.

So yes what you''re suggesting is right. Without knowing the details I can''t advise much about where to put this.  I don''t quite see why the configuration you have isn''t what you need - effectively you are only allowing 5 threads into this entire module. Of course any blocking service requests then deny access to every other service in this module. If you have a set of services that are the trouble makers then I''d isolate them in a throttled module of their own.

From what it looks like to me, the problem seems to stem from the fulcrum processing messages in the order they are received.  The throttle therefore causes a problem as every request is coming in through the same fulcrum and therefore the throttle actually slows processing down.
I think what I am likely going to need is a separate fulcrum for processing the bpm-service requests and then throttle those requests.  This way other requests coming in through the current fulcrum will not be queued behind these slow requests.
Does that sound about right?

OK so this is progress - the throttle is doing what its supposed to.

You could put a throttle in your top tier - I don''t know your architecture so can''t advise the best location - but the way you''ve done this example is correct and can be applied anywhere you need it.

However - you know you can always set the main kernel throttle (which sits between all transports and the kernel)...

http://localhost:1060/ep+name@app_ext_introspect_sysconfig

Its the first 2 options.  This will mean all external requests are safely queued on the top side of NK. Would this help?

We did set a timeout on the connection pooling.  The DB connection pool setting was to close after 60 seconds, and we had the NK kill after 90 seconds.  However, we would still have processes running for 5 minutes or more.

I did the sanity check, and the Accessor is throttled to a maximum count of 5.  I think what we are trying to do is restrict all incoming requests from HTTP sources for bpm services to be restricted to a limited amount so that there are processes for other applications to be serviced.  If a process is throttled, does it allow other processes to run even if the Active Requests page shows only bpm-service requests.
Or, do we need to set the throttle at the Fulcrum or in Jetty through the HttpConfig?

Gary

Thanks for posting this.  So far I don''t see anything unusual.  You will see multiple requests - as they will all be issued and redirected to the throttle.  It will allow through only as many as you specify in the config.  So you should only see n-requests to the uri on the other side - in this case that will be active:evaluateConditionalTransitions

Just to sanity check - can you put a counter in the class

com.fl.nk.bpm.accessor.EvaluateConditionalTransitionsAccessor

so that it increments on each received request and decrements prior to returning a response. For each inbound requests dump the count to standard out.  We need to know if you really are getting all these these threads down at this point.

Note that a request that reaches the throttle but doesn''t get through gets put into an asynchronous queue - so the logical request is blocked, but the underlying thread goes back to the kernel and gets new work to do. So far this scenario would give you what you posted.  ie you see active requests but they will won''t be tying up threads.

Please can you quickly report what the counter on the accessor gives you - ie it should never exceed the admittance value of the throttle

Cheers,

P.

PS Not sure if you saw Jeff''s comment above - but NK can''t kill IO blocked threads with the deadlock detector - its a limitation of Java.  So if you have pathological DB response you should set the timeout values on the connection pool configuration.  As he mentioned - if the DB is blocking you, they would then naturally timeout and not tie up all your threads - in which case the throttle may not be needed after all?

I used the Trace tool to verify that we are going to the Throttle.

Root Request   Timestamp   Duration
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .257   2293
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .258   2470
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .258   2391
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .259   15378
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .259   15635
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .259   2697
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .259   5242
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .270   14855
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .270   15077
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .276   15434
http://duckwood.westlan.com:9505/bpmapp/bpm-services/9.4/evaluateConditionalTransitions?FuegoInstanceString=%2fCustomFulfillment%23Default-1.2%2f874667%2f2   13:44:07 .277   19989

Notice the timestamp.  I have at least 7 simultaneous requests.  All the requests show up in the Active Requests page as well.
Each of them is going to the Throttle Accessor:
HTTPBridgeAccessor
urn:org:ten60:netkernel:tpt:http   SOURCE   http://duckwood.west lan.com:9505/bpmapp/ bpm-services/9.4/eva luateConditionalTran sitions?FuegoInstanc eString=%2fCustomFul fillment%23Default-1 .2%2f874667%2f2   
104

VoidAspect$1
(from cache)
urn:org:ten60:netkernel:tpt:http   SOURCE   ffcpl:/etc/HTTPBridg eConfig.xml   
208

HTTPBridgeConfigAspect
ThrottleAccessor
urn:org:ten60:netkernel:ext:sys   SOURCE   jetty://duckwood.wes tlan.com:9505/bpmapp /bpm-services/9.4/ev aluateConditionalTra nsitions+param@liter al%3Aparam3   
104

StringAspect
(from cache)
urn:com:findlaw:nk:bpm:services   SOURCE   ffcpl:/etc/bpmThrott le.xml   
72

DOMXDAAspect, ModuleResourceRepresentation
EvaluateConditionalTransitionsAccessor
urn:com:findlaw:nk:bpm:services   SOURCE   active:evaluateCondi tionalTransitions+pa ram@var%3Aparam   
32

StringAspect

This is the current module.xml

<module>   <identity>     <uri>urn:com:findlaw:nk:bpm:services</uri>     <version>9.4.0.${buildNumber}</version>   </identity>   <info>     <name>Findlaw BPM Services</name>     <description> Module for BPM services </description>     <dynamic />   </info>   <publisher>     <name>FindLaw</name>     <uri>http://www.findlaw.com</uri>   </publisher>   <license>     <name>Internal</name>     <uri />   </license>   <export>     <uri>       <match>ffcpl:/bpm-services/9.4/.*</match>       <match>ffcpl:/bpm-services/LATEST/.*</match>       <match>ffcpl:/etc/.*</match>     </uri>   </export>   <rewrite>     <rule>       <match>ffcpl:/bpm-services/9.4/(.*) </match>       <to>active:throttle+id@throttle:bpmThrottle+configuration@ffcpl:/etc/bpmThrottle.xml+uri@active:$1</to>     </rule>     <rule>       <match>ffcpl:/bpm-services/LATEST/(.*) </match>       <to>active:throttle+id@throttle:bpmThrottle+configuration@ffcpl:/etc/bpmThrottle.xml+uri@active:$1</to>     </rule>   </rewrite>   <mapping>     <this>       <match>ffcpl:/resources/sql/.*</match>       <match>ffcpl:/resources/.*</match>       <match>ffcpl:/etc/bpmThrottle.xml</match>       <match>ffcpl:/etc/.*</match>     </this>     <rewrite>       <match>ffcpl:/fuegoInstanceString/(.*)/activityName/(.*)</match>       <to>active:getSubscriptionDetails+SubscriptionId@data:text/plain,254796</to>       <!--<to>active:validatePassCode+Xml@data:text/xml,</to>-->     </rewrite>     <ura>       <match>active:assignActivityParticipant.*</match>       <class>com.fl.nk.bpm.accessor.AssignActivityParticipantAccessor</class>     </ura>     <ura>       <match>active:canCancelProcess.*</match>       <class>com.fl.nk.bpm.accessor.CanCancelProcessAccessor</class>     </ura>     <ura>       <match>active:cancelProcess.*</match>       <class>com.fl.nk.bpm.accessor.CancelProcessAccessor</class>     </ura>     <ura>       <match>active:canCompleteActivity.*</match>       <class>com.fl.nk.bpm.accessor.CanCompleteActivityAccessor</class>     </ura>     <ura>       <match>active:createBPMInstance.*</match>       <class>com.fl.nk.bpm.accessor.CreateBpmInstanceAccessor</class>     </ura>     <ura>       <match>active:createProcessInstance.*</match>       <class>com.fl.nk.bpm.accessor.CreateProcessInstanceAccessor</class>     </ura>     <ura>       <match>active:createInstanceFromInstance.*</match>       <class>com.fl.nk.bpm.accessor.CreateInstanceFromInstanceAccessor</class>     </ura>     <ura>       <match>active:evaluateConditionalTransitions.*</match>       <class>com.fl.nk.bpm.accessor.EvaluateConditionalTransitionsAccessor</class>     </ura>     <ura>       <match>active:evaluateConditionalTransition.*</match>       <class>com.fl.nk.bpm.accessor.EvaluateConditionalTransitionAccessor</class>     </ura>     <ura>       <match>active:getDueTransitionDeadline.*</match>       <class>com.fl.nk.bpm.accessor.GetDueTransitionDeadlineAccessor</class>     </ura>     <ura>       <match>active:getValidAdditionalProcesses.*</match>       <class>com.fl.nk.bpm.accessor.GetValidAdditionalProcessesAccessor</class>     </ura>     <ura>       <match>active:getValidReclassifyTypes.*</match>       <class>com.fl.nk.bpm.accessor.GetValidReclassifyTypesAccessor</class>     </ura>     <ura>       <match>active:getValidReclassifies.*</match>       <class>com.fl.nk.bpm.accessor.GetValidReclassifiesAccessor</class>     </ura>     <ura>       <match>active:getValueForKey.*</match>       <class>com.fl.nk.bpm.accessor.GetValueForKeyAccessor</class>     </ura>     <ura>       <match>active:mapCloneArgsToThread.*</match>       <class>com.fl.nk.bpm.accessor.MapCloneArgsToThreadAccessor</class>     </ura>     <ura>       <match>active:onActivityEntry.*</match>       <class>com.fl.nk.bpm.accessor.OnActivityEntryAccessor</class>     </ura>     <ura>       <match>active:onActivityExit.*</match>       <class>com.fl.nk.bpm.accessor.OnActivityExitAccessor</class>     </ura>     <ura>       <match>active:onExceptionEntry.*</match>       <class>com.fl.nk.bpm.accessor.OnExceptionEntryAccessor</class>     </ura>     <ura>       <match>active:onProcessBegin.*</match>       <class>com.fl.nk.bpm.accessor.OnProcessBeginAccessor</class>     </ura>     <ura>       <match>active:onRunInteractiveComplete.*</match>       <class>com.fl.nk.bpm.accessor.OnRunInteractiveCompleteAccessor</class>     </ura>     <ura>       <match>active:processPapiToDoItems.*</match>       <class>com.fl.nk.bpm.accessor.ProcessPapiToDoItemsAccessor</class>     </ura>     <ura>       <match>active:processEmailToDoItems.*</match>       <class>com.fl.nk.bpm.accessor.ProcessEmailToDoItemsAccessor</class>     </ura>     <ura>       <match>active:processPendingMessages.*</match>       <class>com.fl.nk.bpm.accessor.ProcessPendingMessagesAccessor</class>     </ura>     <ura>       <match>active:runAutomatic.*</match>       <class>com.fl.nk.bpm.accessor.RunAutomaticAccessor</class>     </ura>     <ura>       <match>active:unleashTheClones.*</match>       <class>com.fl.nk.bpm.accessor.UnleashTheClonesAccessor</class>     </ura>     <ura>       <match>active:getUIComponents.*</match>       <class>com.fl.nk.bpm.accessor.GetUIComponentsAccessor</class>     </ura>     <ura>       <match>active:createInstanceFromParameterMap.*</match>       <class>com.fl.nk.bpm.accessor.CreateInstanceFromParameterMapAccessor</class>     </ura>     <ura>       <match>active:createDeepLinkableInstanceFromParameterMap.*</match>       <class>com.fl.nk.bpm.accessor.CreateDeepLinkableInstanceFromParameterMapAccessor</class>     </ura>     <ura>       <match>active:getSubprojectMetadataByProjectId.*</match>       <class>com.fl.nk.bpm.accessor.GetSubprojectMetadataByProjectIdAccessor</class>     </ura>     <ura>       <match>active:getSubprojectMetadata.*</match>       <class>com.fl.nk.bpm.accessor.GetSubprojectMetadataAccessor</class>     </ura>     <ura>       <match>active:getQualifyingSubscriptions.*</match>       <class>com.fl.nk.bpm.accessor.GetQualifyingSubscriptionsAccessor</class>     </ura>     <ura>       <match>active:canCreateProcessInstance.*</match>       <class>com.fl.nk.bpm.accessor.CanCreateProcessInstanceAccessor</class>     </ura>     <ura>       <match>active:getPage.*</match>       <class>com.fl.nk.bpm.accessor.GetPageAccessor</class>     </ura>     <ura>       <match>active:submitPage.*</match>       <class>com.fl.nk.bpm.accessor.SubmitPageAccessor</class>     </ura>     <!-- <ura> <match>active:refreshPapiConnection.*</match> <class>com.fl.nk.bpm.accessor.RefreshPapiConnectionAccessor</class> </ura> -->     <ura>       <match>active:refreshParticipantRoleMapping.*</match>       <class>com.fl.nk.bpm.accessor.RefreshParticipantRoleMappingAccessor</class>     </ura>     <ura>       <match>active:getVideoOptions.*</match>       <class>com.fl.nk.bpm.accessor.GetVideoOptionsAccessor</class>     </ura>     <ura>       <match>active:monitor.*</match>       <class>com.fl.nk.bpm.accessor.MonitorAccessor</class>     </ura>     <!-- Internal services and resources -->     <import>       <uri>urn:com:findlaw:nk:appcore</uri>       <version-min>9.2.0</version-min>       <version-max>9.2.99</version-max>     </import>     <super />   </mapping> </module>

Are you using the latest mod-db that supports timeouts on database queries?  You still can''t kill the request from the NK process list, but you can have it die after 10 (or whatever) seconds of hanging.

For your throttle-protected resources, are you accessing them synchronously or asynchronously?  If you''re calling it as a synchronous subrequest it will still block the calling thread.

Are you seeing the 12 bpm processes in the NK process list, or on the backend server that you are attempting to protect?

Hi Gary,

You seem to be on the right track.  To understand what''s happening we need to see the module.xml.   The throttle does work as you believe and what you are doing, moving the request to the throttle and then reissuing a request (uri@ arg) with a new base URI, is sound.

Have you looked at a single request in the visualizer?  Is it definitely reaching the throttle?

If you can''t track this down easily - please send the module.xml and we''ll dig into it.

P.

We are having issues that we are trying to solve by implementing a throttle.  The problem is that we have some services that for some reason are experiencing database locks.  This then locks up a process thread.  We have tried a few things to kill the process but so far none work.  The major problem is that there is a particular set of services that cause the problem.  They then end up chewing up all available process threads so that no other services can be serviced.  This brings down all our applications making NK calls.  What we would like to do is limit the amount of processes/threads that the bpm services can make.  We did the following.
In the bmp services module.xml has the following rewrite rule.

<rule>             <match>ffcpl:/bpm-services/9\\.4/(.*) </match>             <to>active:throttle+id@throttle:bpmThrottle+configuration@ffcpl:/etc/bpmThrottle.xml+uri@active:$1</to>         </rule>

We have established in the same module, the following throttle config.

<throttle>   <concurrency>5</concurrency>   <queue>100</queue> </throttle>

We then have a JMeter test that kicks off a about 12 requests.  We then pull up the active request page.  We are expecting to only 5 bpm processes running.  Instead we see 12 active bpm processes.  When we experience our NK lock up in production, it is the active requests page that we see has filled up with long running processes - multiple minutes over the time that we had hoped NK would kill the process.

Is there something we are doing wrong with our throttle setup, or does a throttle not work as I expect it to work?

Gar