nk4um

Hi Jeff,

I do hear you! But I''m reluctant to add more behavior to the kernel with more subtle processing around caching unless there are no other solutions. I look at the permutations in HTTP around caching and I don''t want to recreate that. Yes adding the SKIP_CACHE, and DONT_CACHE wouldn''t be to hard. EXPIRED_OK has subtle implications on the operation of the cache which I''m not keen on tampering with it.

To this end I''ve create an example module for you which I have just emailed to you. It has a single unit test which executes a scenario of repeated resource requesting through a user caching layer implemented as a pluggable-overlay. This cache will always return a stale response if it exists but will also trigger an asynchronous update of the cached value. This simple example has no cleanup but demonstrates the concept.

Cheers,
Tony

before we are descend into the complexity of handling the semantics of stale and expired

There are some complexities involved here, but they''re of the "what is the right thing to do" type, not "how to do it".  At the kernel api level, the behavior is unambiguous and the changes are quite simple.  Simple to the point that I would just do it myself were I not concerned that I''d lose the changes with the next release.  I''m pretty sure it''s about 20 lines of code total (posibly a bit more because it''s not clear how to check if a particular request header is present) across 4 files for just adding in EXPIRED_OK, SKIP_CACHE, and DONT_CACHE; that grows somewhat if isStale() were added to IExpiry.  I''d be more than happy to implement my refreshing logic with max-stale, async request, force-refresh, and so forth as a local overlay/wrapper module, but I can''t do it without a little help.

I think maybe you are missing the subtlety of my solution 1.

I''m pretty sure I understand what you''re suggesting, although I was thinking in terms of NK3''s active:throttle accessor in which you specify a  uniqie id on the throttle so its easy to make lots of different throttles.  The NK4 throttle is an overlay so I''m not sure how that would work but I assume pretty similarly.  The problem remains the same - the first request when the resource is expired blocks for the full time, and all other requests for that exact same resource block too.  Other instances of the general resource are fetched normally so long as they are not expired. So the first 50 people who request "page 1" post-expiry all get a delay of up to 10 seconds.  Everyone requesting "page 2" is fine.  It''s that 10 second delay I want to eliminate.

This multi-throttle implementation is a key part of the solution in any case, but it is not a complete solution.

had a simple all wrapped resources have a simple 24hr (for example) expiry

For this specific case, I could do that, but I see this as an important use case that could be applied in many cases, so I''d prefer a more general solution.  Even a really simple deviation from this simple expiry (like, to add a golden thread to cut if the pages need to be updated off-schedule) would break the simple model.

a third solution ... maintain a local cache

Right, this is the fallback solution.  (Well, one of a few variants.)  It is how I would do it if I wasn''t working inside NK.  I''d prefer to have something that best leverages the NK resource model, and I don''t think implementing a local cache does that.  From my original question:

So now we''re looking at implemeting our own caching behavior completely outside the netkernel interfaces to get what we want.  It''s not overly complicated, but it''s wasteful to need to ignore a perfectly good cache that''s built in because it''s not flexible enough.

Hi Jeff,

before we are descend into the complexity of handling the semantics of stale and expired lets see if we can find a solution at the application level. I''m extremely reluctant to consider an approach which involves special handling unless there really is no other way to do it because of the optimisation that needs to be performed at the kernel level and the need to keep that code extremely simple.

I think maybe you are missing the subtlety of my solution 1. I''m not proposing the existing throttle is up to the job but rather a new throttle that has the ability to treat each distinct resource identifier independently. In this way all requests will not block when any resource expires but rather only requests for that specific resource. I still believe this approach will work for you unless there is something that I''m not seeing in your requirements.

I agree that solution 2 may get complex. I was under the impression that you had a simple all wrapped resources have a simple 24hr (for example) expiry. If there is a mix of timed and dependent and the wrapping layer cannot know what is what then this approach won''t work.

I''ll also throw in a third solution to the pot given that we''ve discounted solution 2. This again involves having a wrapper layer. But this time we maintain a local cache within this wrapper endpoint. By requesting for response
issueRequestForResponse() we get back an INKFResponseReadOnly which we can check the expiry of. So when a representation is expired we still have a handle on it an can return it. But also we can async issue another request for the resource and update our cache when we can a new response.

Tony

A note on the above proposal: either of the main suggested enhancements would enable a near-complete implementation.  If I could get the expired value then I could return it and issue a normal async request through the throttle which would get cached normally and thus refresh.  If I could force a refresh of a non-expired value then I could take the lying approach and set the expiry date in the future and the force-refresh the cache if the normally requested item is expired tomorrow. 

Obviously, I''d prefer all options for maximum flexibility, but who wouldn''t? :)

A slight problem with the expansion of the expiry notion into stale/expired is that it raises the question of how stale-ness propagates compared to expired-ness, and how does it extend to a golden thread for example (i.e., can you make a golden thread stale without completely cutting it).  While I think the notion makes alot of sense, it does complicate things somewhat.

Hey Tony,

Both of your suggested solutions fall a bit short of my ideal solution.  The first (which I alluded to above) gives my "middle case" behavior - the backend does not get overloaded with lots of useless work, but all requests post-expiry block until the work is completed.

The second would work, but it is made alot more complex because of two specific limitations in the caching behavior: there is no way to retrieve an expired value from the cache (the workaround is to lie about the expiry time but there is no workaround for dependent expiry), and there is no way to force a refresh of non-expired value in the cache (the workaround is to attach a golden thread, but this forces you to change your underlying accessor).

I agree that the "allow-stale-refresh" behavior I outlined is fairly complex and probably does not deserve to be a core function.  However, it could be built much more effectively out of some simple pieces, if those pieces were available. These pieces are exactly the limitations mentioned above, and I believe they could both be implemented in very contained manners.

First, retrieving an expired value, or more specifically retrieving a cached value regardless of its expiry status.  This would be a request header (say, EXPIRED_OK) that, if present, would skip the call to meta.getExpiry().isExpired() in the cache get() function.  It would only affect the cache get() function and have no effect anywhere else.

Second, force-refreshing a value.  This is half of the behavior of the HEADER_NO_CACHE, which I think unfortunately combines two distinct behaviors:  it skips checking the cache but also skips possibly caching the response.  These two should be doable separately (say, SKIP_CACHE to not check the cache and DONT_CACHE to not cache any response;  ideally NO_CACHE would be the first but it already has an established meaning in NK).  These would only affect whether - not how - the reqeust processor interacts with the cache, with no effect anywhere else.

Going beyond this, two additional extensions that I would not specifically need but would nonetheless be useful are
- a way to retrieve a resource ONLY from the cache (there is obviously a way to do this already, as shown by the cache viewer, but it''s not clear how to do it elsewhere)
- an enhanced notion of expiry, allowing for ''soft'' and ''hard'' expiry.  This would only be meaningful with EXPIRED_OK, which could then allow soft-expired values but never hard-expired ones.  Or to give them more distinct terms, "stale" intstead of ''soft-expired'' and "expired" instead of ''hard-expired''.  If this existed, then the first extension above becomes STALE_OK and the change is something like if (!meta.getExpiry.isExpired()) becomes
if (!meta.getExpiry.isExpired() && (!meta.getExpiry.isStale() || requestHeaders.STALE_OK)) -- again, only affecting one specific test.

Hi Jeff,
this is an interesting problem but not one that is too hard to find a solution too. I am thinking of two approaches. I don''t think creating a custom header and having complex behaviour interactions between the kernel with new caching semantics is going to fly. We are not looking to make anything at that level more complex. :-)

So the two solutions I see are:

1) create a throttle where each unique request identifier is throttled by it''s own logical throttle. So each of the 200 resources could be treated independently without needing to specify 200 throttles. If needed it could even contain a configuration to specify the unique keying of throttles to resource identifiers.

2) create a wrapper layer that wraps it''s own expiry function over your resources and gives them their 24 hour expiry but internally will implement the semantics you specify. I.e return stale values but re-request the underlying resource asynchronously.

I''d prefer to go down path 1.

Thoughts?

I have a problem with an interesting solution, and I''m trying to figure out how to implement that in NK.

The problem: I have a resource that is heavily requested (say, 5 times a second) that takes significant time to compute (say 10 seconds).  The result, once computed, is cachable for some time (1 day or until otherwise expired). 

Once the resource is hit once and cached, everything works well. Until it expires.

When it expires, then the best-case behavior is that there will be a single request that will take 10 seconds.  Middle case is that the first request will take 10 seconds and all other concurrent requests will block until the first completes, at which time they will all complete nearly instantly.  This could be accomplished by putting the accessor behind a throttle with concurrency=1 but this is troublesome because there are some 200 variants of the resource so any one blocking would make all of them block, no matter how fast they might respond (unless a separate throttle is used for each combination of parameters).  The worst case is that 50 concurrent requests for the same resource are issued (5 per second for 10 seconds, until it gets cached) hammering the system (and the backend database behind it) with redundant work.  None of these are really desirable.

The interesting solution I came up with is to allow a stale cached value to be returned, but if the cached value being returned is stale then also issue the request normally so that a fresh value can be cached, but only if it is not already being done.

The ideal way I would want to implement this is to add a request header, "allow-stale-refresh", possibly with a timeout value indicating just how stale a value is allowed.

However, it''s not clear that this is possible at any level, because there is no way to get a stale value out of the representation cache in the first place.  A partial workaround I thought of is to have the "slow" resources set their timeout to some time in the future (the allowed stale amount, say a day) and then a wrapper could get the resource normally and check the response metadata to see if it will be expired tomorrow and if so then return it normally but also issue a request for the resource with the ''NO_CACHE'' header to get the fresh value, but NO_CACHE bypasses the cache in both directions so it cannot be used to update a cached value.  The resource could attach a golden thread to expire it manually and then issue the request normally, but now this is getting very complicated (and cannot be implemented as a simple wrapper around an arbitrary resource) instead of the nice clean exotic caching behavior I would like to see.

So now we''re looking at implemeting our own caching behavior completely outside the netkernel interfaces to get what we want.  It''s not overly complicated, but it''s wasteful to need to ignore a perfectly good cache that''s built in because it''s not flexible enough.

Any suggestions?