Using ICacheManager in Orchard with Expensive Factory Code

The ICacheManager abstraction in Orchard is a very cleverly written piece of code that can be used to cache frequently used data, within as well as across requests. It supports an advanced invalidation mechanism whereby you can associate your cache entry with one or more IVolatileToken instances which can cause the cache entry to expire based on arbitrary events or conditions. There are a few built-in implementations of IVolatileToken, such as the AbsoluteExpirationToken and FileToken, and you can also implement your own if you want to control expiration/invalidation based on something else.

Because of this nice token-based expiration mechanism, ICacheManager cannot easily be made farm-aware, and as a result, it is typically only used for data that can be considered "node local", i.e. data where neither the generation nor the expiration/invalidation need to be synchronized across farm nodes, but rather can happen on all nodes independently without negative consequences.

The API is pretty straight-forward: you call ICacheManager.Get() and provide it with a cache key and a factory function. If a cached value exists, it is immediately returned to you. If not, your factory function is called to create the data, the result is stored in the cache for subsequent callers, and then returned to you. Optionally, your factory function can also associate the data with one or more IVolatileToken instances if you want the data to expire at some point.

Let's look at a simple example:

ICacheManager _cacheManager; // Injected dependency.
IClock _clock;  // Injected dependency.

var someCachedValue = _cacheManager.Get("SomeKey", context =>
{
    context.Monitor(_clock.When(TimeSpan.FromSeconds(30))); // Expire in 30 seconds.
    string result = null;
    // TODO: Do some expensive work to generate the result value here.
    return result;
});

Here, the ICacheManager.Get() function is called with the cache key "SomeKey" and a factory lambda. The factory is provided with a context parameter, and calls context.Monitor() to associate the cache entry with a volatile token. The IClock.When() function is a convenience method that in this case takes a TimeSpan and returns an AbsoluteExpirationToken representing a point in time 30 seconds from now.

The underlying implementation of ICacheManager basically depends on an internal ConcurrentDictionary for storage, and does not guarantee thread safety for the factory function. In other words, if multiple threads call ICacheManager.Get() with the same cache key while the value is not in the cache, the factory function might very well be executed multiple times, and only one of the values kept.

For the most part this is fine, but in certain situations it's undesirable. For example, if the factory code is very expensive (e.g. very CPU- or database-intensive) and time-consuming to execute, and your site is under very high user load, this can lead to serious problems. Typically, the unfortunate sequence of events goes like this:

  1. Some expensive value expires from the cache.
  2. The first request after that finds the value missing and the factory function is called to generate the value.
  3. Since the generation is expensive and takes a while, and new requests come in very frequently, in the mean time the next request also finds the value missing, and it also starts generating the value. Now both requests are fighting for the same scarce CPU and database resources, and as a result, both will now likely take even longer to complete.
  4. More and more requests come in, and the snowball keeps growing. The more requests start generating the value, the longer they all take to complete because they compete for the same resources, and as a result even more requests start generating the value.
  5. In the best case, at least one of the requests eventually succeed in generating the value, the ones that fail do so in a graceful manner, and your site recovers (at least until the next expiration). In the worst case, your site crashes and burns.

The same problem can occur not only with sudden expiration, but also on the initial construction of the value if you already have significant user load at this point. I have seen this happen frequently when a passive node is reinserted into a running cluster when there is already heavy traffic, and the shit hits the fan running, to use a nice mixed metaphor.

So what can we do to make our code more resilient?

Well, the desired behavior is that only one thread gets to regenerate the value, while other threads block while waiting for that to be finished. Blocked threads are just waiting idle, so they do not consume the critical resources. The first thread can use any and all available resources to generate the value as quickly as possible, and all subsequent threads get it when it's done. In the grand scheme of things, all threads get the result much faster, with much less impact on system resources. Cooperative synchronization at its best - everybody wins and goes home happy.

The easiest way to accomplish this is to cache a Lazy<T> instance rather than the generated value itself, and rely on the thread synchroniation of Lazy<T> for the blocking part. Let's take a look at a revised example:

ICacheManager _cacheManager; // Injected dependency.
IClock _clock;  // Injected dependency.

var someCachedValue = _cacheManager.Get("SomeKey", context =>
{
    context.Monitor(_clock.When(TimeSpan.FromSeconds(30))); // Expire in 30 seconds.
    return new Lazy(() =>
    {
        string result = null;
        // TODO: Do some expensive work to generate the result value here.
        return result;
    });
}).Value;

Here, instead of returning the result from the factory, we are returning a Lazy<T> wrapping the factory logic. We use the Lazy<T> constructor that takes neither a LazyThreadSafetyMode mode parameter nor a bool isThreadSafe parameter, which in effect gives us a Lazy<T> instance that is fully thread safe for both execution and publication of the value, and as a result, Lazy<T> guarantees that the factory executes only once.

This simple change dramatically changes how the operation works. Now, if two threads call ICacheManager.Get() simultaneously, instead of constructing two result values in parallel, we only construct two Lazy<T> instances. The Lazy<T> instance is extremely fast and cheap to construct; we can safely afford to do this multiple times and throw all but one of the instances.

When we get back the Lazy<T> instance from ICacheManager.Get() we read its Value property. At this point, ICacheManager.Get() (by virtue of its underlying ConcurrentDictionary implementation) will always return the same Lazy<T> instance. Since Lazy<T> doesn’t execute the actual factory until requested, the value is only constructed once.

Another advantage of this code compared to the first example, is that exceptions are cached by the Lazy<T> class, so we are protected against the snowball effect even if the factory code throws for some reason (for example, if it depends on an external resource that is not responding). Depending on your scenario and factory logic, this may or may not be desirable; you can control the exception caching behavior by using one of the other overloads of the Lazy<T> constructor.

10 comments

  • ... Posted by Jean-Thierry Kéchichian Posted 03/04/2015 08:32 PM [http://www.evolutiveautomation.com/]

    Wow, many thanks, I like this kind of post that is so clear even it goes in depth into the code. This because I prefer fewer things but better functioning

    Would it be possible to integrate this in the Orchard core? Thus the factory would be automaticaly wrapped into a Lazy instance

    Regards

  • ... Posted by Daniel Stolt Posted 03/04/2015 10:42 PM (Author)

    Jean-Thierry: glad you like! As to inclusion in core, I am leaning towards including this in core as an extension method on the ICacheManager interface. Partly to avoid breaking existing code, but primarily because it makes more sense - this synchronizing "wrapping" code is probably not something you'd want different implementations doing differently, it's really just a convenience layour on top of the ICacheManager abstraction.

  • ... Posted by Alex Posted 03/05/2015 12:04 PM

    Great post and really great writing style. Very easy to follow and understand. You and Sipke could make the half of world use Orchard ;-)

  • ... Posted by harmony7 Posted 04/17/2015 02:14 PM

    Hey, great post. I want to make sure I understand the Lazy<T> constructor properly --- you are setting isThreadSafe to be false, but the documentation says that when you want to create a Lazy<T> object that is fully thread safe (one that uses locking to ensure that only one thread initializes the value) then you would want the default setting where isThreadSafe is true. Can you help me understand why you are using false?

    Cheers!

  • ... Posted by Daniel Stolt Posted 04/18/2015 10:51 AM (Author)

    Alex: Many thanks. ;)

    harmony7: I apparently made a mistake on that one. I interpreted the documentation (and in fairness was also misled by a sample which made the same interpretation) to mean that isThreadSafe = false indicates that your value factory function is not thread safe, thus causing Lazy<T> to ensure the thread safety for you. Thanks for pointing this out! I have corrected the post.

  • ... Posted by Donald Boulton Posted 05/18/2015 11:20 PM [http://donboulton.com]

    Orchard 1.9 -int build orchard-05faccd7226e0b15d926582994b0169ac1efb0e8 worked great. Then you merged it with dev and released Orchard.Web 1.9 and its caching is full of errors, profiles updates are so bad you cannot log out, will not use a sql session state database loses connections if you refresh the page. Session timeouts all the time = its basically trash, full of errors on caching and profiling. Really disappointed!!!

  • ... Posted by Daniel Stolt Posted 05/19/2015 10:58 AM (Author)

    Donald: Sorry you're having issues with the 1.9 release, but this is not the place to voice your concerns. Please open issues on https://github.com/OrchardCMS/Orchard/issues and we'll look into them.

  • ... Posted by harmony7 Posted 06/26/2015 03:45 AM

    Hey again.

    I found one more thing out relating to this (the hard way) so I thought I'd bring it up here.

    You have to be very careful when one cached item depends on another cached item. Orchard's cache has a bubbling mechanism so that, if reading a cached value for resource A misses and part of the lambda calls for getting the cached value for B, then item A's invalidation tokens are automatically set up so that they include those of B as well. This way, if a token would cause B to be invalidated in any way, item A will also simultaneously be invalidated.

    You already have a correct example that shows the context.Monitor() for signals happening in this way, outside the lambda of the Lazy<T>.

    However, because of the way context bubbling works, it's also important that things that attempt to fetch other cached items also happen outside of the lambda of the Lazy<T>. If this is not done, then the tokens for item B are not properly inherited by item A.

    Example:

        public IEnumerable<ExtensionDescriptor> GetModulesThatStartWithA() {
            return _cacheManager.Get("cachekey", ctx => {
                var extensions = _extensionManager.AvailableExtensions();  // Do this HERE
    
                return new Lazy(() => {
                    // var extensions = _extensionManager.AvailableExtensions();  // DON'T do this here
    
                    // Filter to just Modules
                    var modules = extensions
                        .Where(ex => DefaultExtensionTypes.IsModule(ex.ExtensionType));
    
                    // Make a list of just the modules whose IDs start with "A"
                    return modules
                        .Where(ex => ex.Id.StartsWith("A"))
                        .ToList();
                });
            }).Value;
        }
    
  • ... Posted by Arman Forghani Posted 07/09/2016 09:06 PM

    Although this article was written over a year ago, it is really nice. I 100% agree with Alex: "Great post and really great writing style..." Thank you so much. harmony7: Thnak you too.

  • Thanks for such great information, i love your blog posts. Keep the good work.

Leave a comment