WinDBG commands, configuration

How many times the problem you’ve faced was caused by ….. wrong configuration:

  • Environment-specific – TEST/UAT/PROD config is enabled in wrong environment
  • Function-specific configurations wrongly ON/OFF:
271 rows for each Sitecore role

Do you recall the pain of configuration spreadsheet with over 270 configuration files to be adjusted for every logical role for prior Sitecore 9 world?

Over 270 files to be manually changed just to show HTML in browser =\

When to blame config?

Configuration is worth inspecting as soon as you hear:

How to view Sitecore config?

Sitecore configuration file is built during system startup and cached inside

  • Sitecore.Configuration.ConfigReader.config (newer)
  • Sitecore.Configuration.Factory.configuration (old versions):

XmlDocument is NOT stored as plain text, hence should be assembled back into human-readable format somehow.

Kudos to NetExt extension for saving the day:

Loading NetExt into WinDBG

!wxml restores XML:

XML document restored

Only Sitecore configuration node is stored there, while native config parts could be read via !wconfig:

In-memory config parts

Alternative approaches

Showconfig tool

One could use SIM Tool, showconfig to re-build configs from web.config + App_Config copy. Are there any guarantees it uses exactly same algorithm as the Sitecore version you are using does?

Showconfig admin page

Requesting sitecore/admin/showconfig.aspx from live instance – the most trustworthy approach. However CDs have that disabled for security reasons, so not an option.

Support package admin page

Support Package admin page would ask all servers online to generate their configs:

Support Package admin page

Do you pay the performance price for non used features?

Intro

The general assumption is – not using a feature/functionality means no performance price is payed. How far do you agree?

Real life example

Content Testing allows verifying which content option works better. Technically it enables multiple versions of same item being published and injects processor into getItem pipeline to figure out which version to show:

      <group name="itemProvider" groupName="itemProvider">
        <pipelines>
          <getItem>
            <processor type="Sitecore.ContentTesting.Pipelines.ItemProvider.GetItem.GetItemUnderTestProcessor, Sitecore.ContentTesting" />
          </getItem>
        </pipelines>
      </group>

As a result, every GetItem API gets extra logic to be executed. What is the price to pay?

Analysis

The processor seem to roughly do following:

  • Check if result is there
    • yes -> quit
    • no – fetch item via fallback provider
  • If item was found – is context suitable for Content Testing to be executed:
    • Are we outside shell site?
    • Are we seeing site in Normal mode (neither editing, nor preview) ?
    • Are we requesting the specific version, or just latest?

Should all be true, a check for cached version is triggered:

  • String concatenation to get unique item ID
  • Append sc_VersionRedirectionRequestCache_ to highlight domain

As a result, each getItem call shall produce strings (strings example)

Question: How much excessive memory does it cost for a request?

Coding time

IContentTestingFactory.VersionRedirectionRequestCache is responsible for caching and defaults to VersionRedirectionRequestCache implementation.

Thanks to Sitecore being config-driven, stock implementation can be replaced with own:

using Sitecore;
using Sitecore.ContentTesting.Caching;
using Sitecore.ContentTesting.Models;
using Sitecore.Data.Items;
using Sitecore.Reflection;

namespace ContentTesting.Demo
{
    public sealed class TrackingCache : VersionRedirectionRequestCache
    {
        private const string MemorySpent = "mem_spent";
        public TrackingCache() => Increment(TypeUtil.SizeOfObjectInstance);  // self

        public static int? StringsAllocatedFor 
            => Context.Items[MemorySpent] is int total ? (int?) total : null;

        private void Increment(int size)
        {
            if (System.Web.HttpContext.Current is null)
            {
                return;
            }

            if (!(Context.Items[MemorySpent] is int total))
            {
                total = 0;
            }

            total += size;

            Context.Items[MemorySpent] = total;
        }

        protected override string GenerateItemUniqueKey(Item item)
        {
            var value  = base.GenerateItemUniqueKey(item);
            Increment(TypeUtil.SizeOfString(value));
            return value;
        }

        public override string GenerateKey(Item item)
        {
            var value = base.GenerateKey(item);
            Increment(TypeUtil.SizeOfString(value));
            return value;
        }

        public override void Put(string key, VersionRedirect versionRedirect)
        {
            Increment(TypeUtil.SizeOfObjectInstance); // VersionRedirect is an object
            base.Put(key, versionRedirect);
        }
    }
}

The factory implementation:

using Sitecore.ContentTesting;
using Sitecore.ContentTesting.Caching;
using Sitecore.ContentTesting.Models;
using Sitecore.Data.Items;

namespace ContentTesting.Demo
{
    public sealed class DemoContentTestingFactory: ContentTestingFactory
    {
        public override IRequestCache<Item, VersionRedirect> VersionRedirectionRequestCache => new TrackingCache();
    }
}

And the EndRequest processor:

using Sitecore;
using Sitecore.Abstractions;
using Sitecore.Pipelines.HttpRequest;

namespace ContentTesting.Demo
{
    public sealed class HowMuchDidItHurt : HttpRequestProcessor
    {
        private readonly BaseLog _log;

        public HowMuchDidItHurt(BaseLog log) 
            => _log = log;

        private int MinimumToComplain { get; set; } = 10 * 1024; // 10 KB

        public override void Process(HttpRequestArgs args)
        {
            var itHurts = TrackingCache.StringsAllocatedFor;

            if (itHurts is null) return;

            if (itHurts.Value > MinimumToComplain)
            {
                _log.Warn($"[CT] {MainUtil.FormatSize(itHurts.Value, translate: false)} spent during {args.RequestUrl.AbsoluteUri} processing.", this);
            }
        }
    }
}

Log how much memory spent on request processing by CT (sitecore.config):

    <httpRequestEnd>
      <processor type="Sitecore.Pipelines.PreprocessRequest.CheckIgnoreFlag, Sitecore.Kernel" />
      <processor type="ContentTesting.Demo.HowMuchDidItHurt, ContentTesting.Demo" resolve="true"/>
...

Wire up new CT implementation Sitecore.ContentTesting.config that logs memory consumed:

    <contentTesting>
      <!-- Concrete implementation types -->
      <contentTestingFactory type="ContentTesting.Demo.DemoContentTestingFactory, ContentTesting.Demo"/>
      <!-- <contentTestingFactory type="Sitecore.ContentTesting.ContentTestingFactory, Sitecore.ContentTesting" /> -->

Testing time

The local results are: 16.9 MB empty VS 3.2 MB warm HTML caches.

50 requests processed at a time would give extra 150 MB pressure, and that is only for a single feature out of many. Each relatively low performance price getting multiplied on number of features results in a big performance price to pay.

A system with 20 same price unused components would waste over 3GB of memory. That does not sound as cheap any more, right?

Conclusion

Even though you might not be using Content Testing (or any other feature), the performance price is still payed. Would you prefer not to pay the price for unused stuff?

Why are reports outdated?

Case study: Analytics reports were a few weeks behind.

Sitecore Reporting introduction

Narrow issue cause down

  • Visitor data is flushed into collection database as number of records increases
  • Interaction live processing pool grows – aggregation is not fast enough

How many visits are pending for aggregation?

The processing pool had over 740K records with oldest entry dated a few weeks ago (same as latest report data).

Throw in more hardware?

Unfortunately, that is the solution the majority decides to take despite:

  • There are no proves existing hardware is used efficiently
  • The current setup is properly configured
  • There are no proves additional hardware would help

Investigation plan

  • Restore prod dbs locally
  • Configure aggregation locally
  • Launch SQL Server profiler for ~60 seconds
  • Launch PerfView for ~30 seconds
  • Collect a few memory snapshots

Collection phase

Triggering aggregation locally with prod dbs killed laptop:

Task manager has shown 80% of CPU usage by SQL Server. Even though queries did not consume much CPU, they had huge duration:

Please tell me where it hurts:

RESOURCE_SEMAPHORE_QUERY_COMPILE is on top of the wait list highlighting query compilations are bottleneck. Good thing we could inspect already compiled plans to get a glimpse of queries:

Ironically, query plan cannot fit default XML output size.

SQL Server needs over 30MB solely for describing how query is to be executed. There are many ‘huge’ queries processed at a time causing compilation gate to hit the limit.

Inspecting memory snapshot

!sqlcmd -s open command gives us a list of all SQL requests:

No wonder MSSQL Server is dying as the first SQL command has 1.7M characters:

The command is run by 58th thread:

It turns out that huge commands are produced by Analytics SQL bulk storage provider. Instead of doing many lightweight SQL calls, it collects them into a batch to avoid network latency cost for each command.

Solution: Stored procedures would reduce the length of SQL query & allow reusing compiled query plans => resolve bottleneck.

Intermediate summary

The behavior was reported to Sitecore:

  • Performance issue 370868 has been registered
  • AD-hoc queries are converted into Stored procedures for by Support
  • Hotfix to increase the aggregation speed has been provided

As a result, aggregation speed increased two times at least.

Even though further improvements are on the deck, I’ll save them for future posts.

WinDBG commands, memory

What objects consume RAM?

!dumpheap -stat reports which objects live in heap:

  • First column is a type descriptor (method table) aka type unique identifier.
  • The second column shows how many objects of that type are in heap.
  • The third column shows own object size (without following nested references).

Use case: Comparing live stats between memory snapshots collected within a few minutes interval would flag growing types, and potential cause of a memory leak.

Hints

  • Free shows memory fragmentation, that would be addressed by compact phase of GC
  • -name include only types that contain string in name
  • -live (slow) leaves only objects that are alive
  • -dead shows only objects that are eligible for garbage collection
  • -short show only object addresses
  • -strings print strings only
  • -mt print objects only of given type by with method table ID

GC heap stats

!gcheapstat (or !gcheapinfo by MEX) the running size of each heap segment, as well as its fill ratio:

Use case:

  • Large objects end in LOH that is not compacted by default. As a result, large objects can provoke additional memory allocations in case existing blocks are insufficient = look like a memory leak.
  • What are the dead objects in ‘older’ generations:

Dump dead objects from generation start till segment generation end (? 0x0000022fadb4e6c0+0n1191488) = 0000022fadc71500:

Sum heximal (0x) format with decimal (0n)

Performing same operation for GEN2 from heap 5:

Strings are in top 3, getting the content of strings by adding -string option:

This specific example highlights Content Testing getItem cache key.

Who holds object in memory?

!gcroot shows why object is not removed from memory by GC:

Use case: Find out the reference chain preventing memory from being released.

Who references the object?

!refs -target by SOSEX attempts to find who references the object:

Use case: find parent collection/cache object is stored in.

ASP.NET cache content

!AspNetCache MEX shows the native ASP.NET cache content:

Use case: Find cached responses (System.Web.Caching.CachedRawResponse) as well as other cached data.

WinDBG commands, threads

Viewing ongoing operations (thread activity) in memory snapshot.

System-wide CPU usage

!threadpool gives average CPU usage during last second + thread pool stats:

Use case

  • Is CPU maxed?
  • How many concurrent operations are being processed?
  • Is there any work queued due to lack of CPU/worker threads?

Flags

  • Over 100 threads & low CPU likely flags a bottleneck in dependent resource (f.e. SQL, Redis, WebAPI calls)
  • 81 % CPU flags garbage collection is running
  • Huge Work queue flags operations are added faster than being processed

Notes

  • Command is overloaded by MEX, thereby unload extension prior to the call
  • CPU is system wide meaning other processes could steal the show

Ongoing operations performed by managed threads

!eestack -ee by SOSEX prints each managed thread call stack:

Use case: Figure out what activities are performed in the snapshot.

Notes

FileWatcher/LayoutWatcher (threads dedicated to certain activity) wake up once in a while (like every half second) to do the status check, they are not usually interesting.

Sitecore.Threading.ManagedThreadPool threads are persisted during application lifetime so that visible in output. Should no operations be added for processing, threads would be idle-ing, hence not consuming any CPU.

How much CPU did each thread consume?

!runaway shows how much CPU time did each thread consume since app start:

Use case: Comparing the numbers between two sequential snapshots would highlight top CPU-consuming threads in between. Stats from single snapshot rarely makes any use since data is aggregated per-thread since application start.

Non-managed stack traces

~* kb 20 outputs top N frames from every thread in application:

Use case: Figuring out what unmanaged operations are running (GC/finalization, attempts to enter critical sections).

Notes: 81% CPU usage is a magic number that likely indicates garbage collection is running, top frame would show phase (f.e. mark/compact). The MSDN article mentions dedicated threads for GC – you can see them on your own (the ones with gc_heap) in unmanaged call stacks.

List threads with exceptions and monitors owned

!threads by SOSEX would print all threads in app:

Use cases

  • Does the number of threads look expected?
  • What threads are inside lock statements?
  • Is there any active exception?

!threads -special would print thread role:

Switching to thread context

~[ThreadNumber]s would set [ThreadNumber] as context one, picture sets last ThreadPool worker thread as context one.

Use cases: inspect objects that live on thread stack, map them to executed frames.

Dump stack objects

!mdso by SOSEX prints objects that live in thread stack:

Use case: Get the data thread operates with (method args would live here).

Restoring thread call stack

Call stacks could make no sense from first sight as CLR can optimize code execution. !dumpstack & clrstack use different semantics to restore call stacks:

Optimizing resources for Content Testing

What is Content Testing?

Sitecore Content Testing finds most effective content variations. It is aimed to improve user experience via testing various content variations and picking top influencing ones.

How content optimization are decisions made?

Decisions are made by aggregating live site visitors behavior for each variant.

How does it know what content to test?

Sitecore_suggested_test_index has computed values for all of the eligible items in the content tree to be optimized.

What is the performance price to pay?

  • Index should be rebuilt periodically (daily) for fresh data
  • Aggregation extracts extra bits from tracked data
  • Item-level API always checks if item under test (a few versions of same content may exist simultaneously)

Room for improvement in every step

Suggested Test index

Content Testing index attempts to index everything under the /sitecore/content

...
            <locations hint="list:AddCrawler">
              <crawler type="Sitecore.ContentTesting.ContentSearch.SpecificSitecoreItemCrawler, Sitecore.ContentTesting">
                <Database>master</Database>
                <!-- Limit this parameter to the root of the content of the site. -->
                <Root>/sitecore/content</Root>
              </crawler>
            </locations>

Default config has a comment to narrow crawler root to actual stored content.

The ‘exclude‘ node allows some items to not be added into the index:

...
           <documentOptions ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration/documentOptions">
                <indexAllFields>false</indexAllFields>
                <exclude hint="list:AddExcludedTemplate">
                  <folder>{A87A00B1-E6DB-45AB-8B54-636FEC3B5523}</folder>
                  <mainSection>{E3E2D58C-DF95-4230-ADC9-279924CECE84}</mainSection>
                </exclude>

By default, everything would be indexed by Sitecore (except 2 templates).

Knowing which items are actually content ones (thereby are eligible for testing) enables us to exclude others, hence let system operate with less data.

Item by template content statistics

The query locates all items under site node and groups them by template id:

WITH ContentItems AS (
  SELECT
    DISTINCT(ID)
  FROM Items
  WHERE
    ParentID = '{0DE95AE4-41AB-4D01-9EB0-67441B7C2450}' -- Content item ID, change to relevant one
  UNION ALL
  SELECT
    m.ID
  FROM Items m
  JOIN ContentItems q ON m.parentID = q.ID),

PopularItemTemplates(TemplateID, Hits) AS (
  SELECT
    [TemplateID],
    Count(1)
  FROM Items i
  JOIN ContentItems u ON u.ID = i.ID
  GROUP BY
    TemplateID
  HAVING
    Count(1) > 300)

SELECT
  t.TemplateID,
  i.[Name] AS [TemplateName],
  t.[Hits] AS 'Based on template'
FROM PopularItemTemplates t
JOIN Items i ON t.TemplateID = i.ID
ORDER BY
  t.Hits DESC

A list of most popular templates under content root is given for a review.

Indexing results

3 times less documents getting stored in index (24K vs 8K) in our success story.

Aggregation and Reporting

Sitecore provides hotfix to make aggregation run faster.

Additionally, you might want to follow Azure query advisor recommendations as it analyzes your database workload and provides suggestions specifically for that load.

High SQL Server load when updating the sitecore_suggested_test_index has index optimization suggestions as well.

Get Item operation

Content testing influences getItem pipeline by adding own processor. That is needed to let different versions of item co-exist at the same time.

This functionality was optimized in Sitecore 9.1+ (ref. 215433).

Why am I visiting Buildstuff?

Simple – there will be people I admire. Their ideas had a tremendous influence and cultivated a specialist from me.

Tribute to Michael Feathers

Any improvements into Sitecore was a ‘fix here breaks there‘ roulette due to a lack of safety net harness.

Working Effectively with Legacy Code had all the solutions (practices) to follow. The book shows how to improve tangled code base. I’m proud to say Kernel safety net was developed thanks to Michael.

There are at least three books each developer is recommended to read:

Tribute to Roy Osherove

The Art of Unit Testing is one of the superlative books explaining TDD and the importance of having self-checking code. I have not met a single person who after reading the book would not write tests.

Tribute to Don Syme

Thanks for F# and most of the good things C# has.

Links and promo code

Here is a promo code for BuildStuff: BUILDBIZDEVA

See you there

WinDBG basic commands, part 3, SQL

This post shows WinDBG commands to analyse SQL queries produced by application. Answering the mystery what parts of program bombard SQL Server with queries.

Ongoing SQL commands

MEX !sqlcmd finds all SQL commands & their state & executing thread:

All SQL commands found in snapshot

Use case: Figuring out the nature of load produced on SQL Server. An owner thread call stack would show where the query is coming from.

SQL connectivity stats

MEX !sqlclientperfcounters displays the connection stats:

Use case: Number of pooled connections could be treated as max SQL commands executed concurrently. That is useful to understand if snapshot collected during peak pressure.

SQL Commands per database

!sqlcn groups commands per connection string:

Use case: Shows most-queried (connection strings)/databases application-wide.

Find all DataSets

!dumpdataset finds DataSet instances in memory with a possibility to view content:

Data set content

Use case: Find the data that mimics database content (either a caching layer, or collected data pending to be pushed). Sitecore Processing uses DataSets to populate reporting (one interesting investigation is coming).

WinDBG basic commands, part 2

How busy CPU is?

!theadpool command shows avg. CPU usage (during last second), number of active threads, and callbacks:

Threadpool

Use case: High CPU investigations, ensure snapshot was collected during maxed CPU. Shows pending callbacks (if any) as well as how many threads are there.

What threads are consuming CPU time?

!runaway shows aggregated CPU time taken by each alive thread:

Use case: A difference between a few sequential snapshots shows threads using CPU resource in between. Useful during high CPU investigations.

What does each thread do?

Each thread either performs some operation, or waits to do something.

!eestack -ee prints managed call stacks for all the threads:

Each thread call stack

Use case: Get an overview of ongoing activities, always useful:

  • Are threads waiting for exclusive access? F.e. lock (obj)
  • Are ASP.NET requests being processed? Trail – IIS7WorkerRequest class
  • What are the operations performed by most CPU consuming threads?
  • Similar/repeatable patterns?

What does each thread do (shorten version)

Mex extension powers !mthreads showing top frame/action performed by thread:

High-level overview of ongoing activities

Use case: High-level overview of ongoing operations in the snapshot

Group threads by call stacks

MEX !uniquestacks groups threads by operations they execute:

Threads grouped by call stacks

Use case: Locate repeatable patterns in one go.

Find currently executed ASP.NET requests

MEX !aspxpagesext gives stats for processed ASP.NET requests:

ASP.NET requests executed

Use case: Get the number of concurrent requests & how long they run for.

Tackling memory leak

In academia, there is no difference between academia & the real world.

In the real world, there is.

Nassim Nicholas Taleb

This post covers the real world case study on tackling memory leak.

Investigation strategy

Memory leak occurs if useless resources are not removed from memory – there has to be something rooting old data. Investigation plan:

  1. Collect memory snapshot once memory usage is beyond the expected range
  2. Collect another one in a minute or so
  3. Compare heap statistics: locate growing/weird types
  4. Find GC roots for these objects
  5. Determine construction mechanisms for these instances

Collecting memory snapshots

A few memory snapshots have been collected via Task Manager:

Collecting memory snapshot via Task Manager

Snapshot can be collected automatically once memory is higher than threshold.

Any anomalies in heap statistics?

Both snapshots contain a huge number of GlassMapper objects that contradicts the module usage pattern (create many short-lived objects):

We could either find !gcroots (who holds object in memory) for suspicious classes that have much higher count than expected or look for unexpectedly huge objects in the heap.

A few IDisposable[] arrays take as much space as 100K objects?! That is over 2200 elements in average (or ~18KB) inside array. There are over 700 huge arrays each having over two thousands disposable elements each, sounds as a code smell.

Finding bigger than average arrays with size over 20K bytes (via min param):

Huge array with over 2 million entries
Glass mapper objects stored inside

Two million elements long array (disposable GlassMapper objects inside) is rooted to ServiceProvider:

SitecorePerRequestScope module roots array in all snapshots

Same pattern is met in all the captured snapshots.

Hypothesis

GlassMapper disposable objects are resolved from global DI scope, hence getting disposed when application shuts down. Objects are stacked in memory until then.

Alternatively disposed scope is used to keep on resolving objects.

Who creates objects that are never disposed?

Although leaking classes can be modified to save birth call stack & ServiceProvider used to create them, not owned types cannot be modified to add that code.

Nevertheless, behavior could be simulated on a test disposable class and Sitecore DI mechanism.

Prediction

At least one way for creating disposable objects produces leak by rooting to either global DI scope (list of resolved IDisposable dependencies), or re-using disposed scope.

Verification steps

  1. Request disposable type via various Dependency Injection mechanisms
  2. Log object creation call stack
  3. Trigger full garbage collection a few times
  4. Attach to process with WinDBG and inspect alive instances
    • How where they created
    • Who roots them in memory

Demo code

The disposable dependency records its birth call stack and capture the ‘parent’ service provider:

using System;
using System.Threading;
using Sitecore.Abstractions;
using Sitecore.Diagnostics;
using Sitecore.Pipelines;

namespace Demo
{
    /// <summary>
    /// Registered in DI with transient scope.
    /// <para>Dispose is to be called upon scope end.</para>
    /// </summary>
    public class DisposableDependency : IDisposable
    {
        private readonly BaseLog _log;
        private readonly IServiceProvider _parent;
        private static int _instancesCount;
        private readonly string _birthCertificate;

        public int Number { get; }

        public DisposableDependency(BaseLog log, IServiceProvider parent)
        {
            Assert.ArgumentNotNull(log, nameof(log));
            _log = log;
            _parent = parent;
            Number = Interlocked.Increment(ref _instancesCount);
            _log.Warn($"Dummy processor was born: {Number} {Environment.NewLine} {Environment.StackTrace}", this);
            _birthCertificate = Environment.StackTrace;
        }

        public void Process(PipelineArgs args)
        {
            _log.Warn($"Dummy processor was executed: {Number} {Environment.NewLine} {Environment.StackTrace}", this);
        }

        void IDisposable.Dispose() => _log.Warn($"Dummy processor died: {Number} {Environment.NewLine} {Environment.StackTrace}", this);
    }
}

Injection to types created by Factory

Sitecore creates objects via DI if resolve=’true’ attribute set in config node.

New instance is created each time if reusable=’false’ attribute is set for processor:

<?xml version="1.0"?>

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
    <sitecore>
        <services>
            <!-- Register disposable service -->
            <register serviceType="Demo.DisposableDependency, Demo" implementationType="Demo.DisposableDependency, Demo" />
        </services>

        <pipelines>
            <initialize>
            <!-- Register controller route to test MVC constructor injection -->
                <processor type="Demo.RegisterRoute, Demo"/>
            </initialize>

            <disposableSample>
            <!-- Register processor to be created every time -->
                <processor type="Demo.DisposableDependency, Demo" resolve="true" reusable="false" />
            </disposableSample>
        </pipelines>
    </sitecore>
</configuration>

Define the disposableSample pipeline with non-reusable DisposableDependency so that a new instance is created for each call. Invoke it via dummy page:

<%@ Page Language="C#" AutoEventWireup="true" Inherits="Demo.PipelineCall" %>
using System;
using Sitecore.Abstractions;
using Sitecore.DependencyInjection;
using Sitecore.Diagnostics;
using Sitecore.Pipelines;

namespace Demo
{
    [AllowDependencyInjection]
    public class PipelineCall : System.Web.UI.Page
    {
        private readonly BaseCorePipelineManager _pipelineManager;

        protected PipelineCall()
        {
        }

        public PipelineCall(BaseCorePipelineManager pipelineManager)
        {
            Assert.ArgumentNotNull(pipelineManager, nameof(pipelineManager));
            _pipelineManager = pipelineManager;
        }

        protected void Page_Load(object sender, EventArgs e)
        {
            var pipeline = _pipelineManager.GetPipeline("disposableSample", string.Empty);
            pipeline.Run(new PipelineArgs());
        }
    }
}

Injection to WebForms and UserControls

ASP.NET ConstructorInjection.aspx form:

<%@ Page Language="C#" AutoEventWireup="true" Inherits="Demo.ConstructorInjection" %>
using System;
using System.Web;
using Sitecore.DependencyInjection;
using Sitecore.Diagnostics;

namespace Demo
{
    [AllowDependencyInjection]
    public partial class ConstructorInjection : System.Web.UI.Page
    {
        private readonly DisposableDependency _dependency;

        protected ConstructorInjection()
        {
        }

        public ConstructorInjection(DisposableDependency dependency)
        {
            Assert.ArgumentNotNull(dependency, nameof(dependency));
            _dependency = dependency;
        }

        protected void Page_Load(object sender, EventArgs e)
        {
            HttpContext.Current.Response.Write($"<h2>I was called {_dependency.Number} </h2>");
        }
    }
}

Injection and MVC

Controller that needs disposable dependency:

using System.Web.Mvc;
using Demo;
using Kerzner.Foundation.DependencyInjection;

namespace Kerzner.Project.BrandWebsites.Pipelines.ScScopeVerification
{
    [Controller]
    public class DummyController : Controller
    {
        private readonly DisposableDependency _dependency;

        public DummyController(DisposableDependency dependency)
        {
            _dependency = dependency;
        }

        public ActionResult Index()
        {
            return Content($"<h2>called {_dependency.Number} <h2/>");
        }
    }
}

Controller registration:

using System.Web.Mvc;
using System.Web.Routing;
using Sitecore.Pipelines;

namespace Demo
{
    public class RegisterRoute
    {
        public void Process(PipelineArgs args)
        {
            RouteTable.Routes.MapRoute(
                "DisposableDependency",
                "DummyController",
                new { controller = "Dummy", action = "Index" });
        }
    }
}

Test results

Non-disposed dependencies are created by Factory

Sitecore DefaultFactory (registered as singleton) captures scoped ServiceProvider. Any object is created from that service provider reuses that scope instead of one valid for request. Birth certificates proof objects created by Factory thereby leaking memory.

ASP.NET pages do not dispose some dependencies as well

AutowiredPageHandlerFactory creates ASPX pages with the respect to proper scope from the first sight. Nevertheless, it does not work as intended due to handler factory cache inside HttpApplication.

Even though correct service provider scope is used at first, it would still be re-used for further requests processed by the instance of HttpApplication.

Conclusion

The memory leak mystery was resolved thanks to a few memory snapshots. Once the cause is found, code adjustments could be done:

  • Do not resolve disposable objects from DI, prefer explicit creation (in case possible) to favor explicit dispose call
  • Consume per-request scope from Sitecore.DependencyInjection.SitecorePerRequestScopeModule.GetScope to resolve dependencies from request-bound scope so that disposed is called on EndRequest
  • Sitecore framework improvement to keep in mind potential nature of disposable dependencies

The behavior described in the article has been reported to Sitecore Support and accepted as 331542 bug report.