Kristof Rennen’s Blog

In my two previous posts, How An ASP.NET DataBind Can Cause A Memory Leak and How An ASP.NET PostBack Can Cause A Memory Leak, I showed you how a UserControl inside a Repeater could cause serious memory problems.

Well after working on the memory issues for several days already, a co-worker of mine finally found a solution to ensure all user controls and pages are disposed properly.

Davy explains how he solved our problems in his blog post about the topic: Guaranteeing Disposal Of UserControls In ASP.NET

 Well, thanks to Davy’s great technical knowledge, we finally managed to get the memory in this application and all our future applications stable, proven by a rerun of our stress test scenarios.

Thanks Davy!!

Kristof Rennen Development .NET, ASP.NET, Memory, Microsoft, Profiling

In my previous post How An ASP.NET DataBind Can Cause A Memory Leak, I talked about how I suffered from serious memory leaks in my current project, because of some user control not being disposed when doing manual databinding multiple times.

After being really happy about my tedious quest being finished to fix the problems, I today noticed the memory leak still occurred in exact the same location, but when using the functionalitity in another way.

The case is still the same:
Page -> UserControl -> Repeater -> UserControl

So the loaded page of the scenario, contained a user control of our own. This user control had a repeater, repeating over the rows of a generic list, creating another UserControl for each found row.

Every row, rendered through the ASP.NET Repeater, has 2 buttons available:

• A button to edit the given row
• A button to delete the given row

The button to edit the given row is not giving me any problems. When I click it, the user gets redirected to the edit page and everything gets cleaned up pretty well.

The delete button on the other hand, is performing a postback on the same page, calling an event to handle the delete:

Ofcourse this delete event is doing the following stuff:

• Getting the row’s id from the CommandArgument
• Calling our business layer to delete the record
• Performing a databind to reflect the changes

It is the last action, which performs the databind, which is now triggering the memory leak … again.

The problem

By doing a postback when clicking the button, a new instance of my user control with all it’s needed resources is created.
Because I’m doing a DataBind() at the end of my “Action”, a second instance of the same user control with all it’s resources is created, leaving the first one undisposed forever.

The second one though, is being disposed properly and does not give me any problems.

The only explanation I would have for this kind of behaviour, is that it all happens inside one postback cycle.
Probably if a control is then created multiple times, only the last one gets disposed at the end of the cycle, leaving the other ones up to the Garbage Collector.

In our case the Garbage Collector is not of any use, since the user control has references to other undiposed objects, causing all those objects to remain in memory and causing a memory leak in the end.

The solution

Well, to be honest I don’t know a proper solution right now and I will probably need some time to figure out a proper way to fix this.

If I add the databind, the application is suffering from a memory leak, which is totally unacceptable.
If I remove the databind, deleted records are still shown on my page, which is totally user unfriendly.

I’ll keep you posted if I find a proper solution, and I then hope ASP.NET is not going to disappoint me ever gain.

Kristof Rennen Development .NET, ASP.NET, Memory, Microsoft, Profiling

Currently I’m working on an ASP.NET application for one of our customers. The application is nearly finished, and is going to be put in acceptance and production in a few weeks.

To test the scalability of the application, stress and load tests are run using Visual Studio Team Test at our side and Load Runner at the customers side.

The stress tests are all running well on both sides, except for some weird OutOfMemory exceptions after a run of 9 or 10 hours with 100 concurrent users. The scenarios are built to simulate one month of work in 10 hours.

Setting up the environment

Since the application consists of a front end and backend, it was important for me to setup my IIS properly to enable me to find whether the problem was caused in the backend or frontend.

The backend application, is a WCF service host, running inside IIS. The frontend application is an ASP.NET application consuming the backend through WCF services.

The running both the service host and web application in a separate application pool, I was able to monitor both separate worker processes to narrow down the problem to one of the two.

Finding the cause of the OutOfMemory exceptions

Before one can start fixing the cause of the OutOfMemory exceptions, you first have to find the cause of course.

Using windbg, DebugDiag and DotTrace, we started investigating the memory usage and patterns of the applications finding that the allocated virtual memory was exceeding the IIS limit, causing OutOfMemory exceptions and unexpected pool recycling.

It was clear the application was suffering from a memory leak, which could still be a native or a managed leak.

A great source to find more information on tracing, Windbg, … is the blog of Tess Ferrandez, an ASP.NET Escalation Engineer of Microsoft. Her blog really helped me out during my search for the cause of memory leaks, and by analyzing all the memory dumps I created.

Reproducing the memory leak

Before one can start fixing the memory leak, its best to have a scenario to easily reproduce the problem.

We started by creating a scenario using a lot of functionalities from the application. By running this scenario we could easily reproduce the problem after a run of a few hours.

By narrowing the scenario down, up to a single functionality only, I was able to trigger the high memory usage faster and find the location of the bottleneck in the application.

I now had an easy way to reproduce the problem, and to verify the fix afterwards.

Tracing the memory usage

Since I now had a scenario which I could use to trigger the memory leak, I used JetBrains’ DotTrace profile to profile my IIS while running the scenario.

Everytime I ran the scenario, I noticed some ASP.NET UserControl not being released from memory. Since this user control, had references to other components as well, those components were “leaked” as well.

By using the trace output, I now had the name of the user control which was not being disposed properly.

The actual problem

The actual problem was a page containing a user control with the following structure.

Page -> UserControl -> Repeater -> UserControl

So the loaded page of the scenario, contained a user control of our own. This user control had a repeater, repeating over the rows of a generic list, creating another UserControl for each found row.

Since we are using ASP.NET with the Model-View-Presenter pattern (to be able to develop everything Test-Driven), we were calling the databind of the view manually as follows:

First we DataBind() the user control after its properties have been set. Then we call base.DataBind() which does a DataBind() of the complete page.
Ofcourse the DataBind() of the usercontrol is not really needed since it will be automatically called when you DataBind the complete page, but hey it should not cause any problems either.

Since the memory profile told us the user control was not properly cleaned up, we just put a breakpoint in its constructor and dispose method.
Analyzing this really blew us of our seats. The constructor of the user control was called twice, but the dispose method was only called once so we were stuck with one instance of the user control for ever. Hello Memory Leak!

The solution

Solving the memory leak was in the end easier than we should have imagined. By just removing the UserControl.DataBind() did the trick.

The DataBind() of the page now looks like this:

Conclusion

I ran my stress tests again together with my DotTrace profiler and everything seemed to be working ok. The memory leak was fixed, and the memory usage of the application was back to normal.

Currently a new run of the stress tests are both running on our environment and our customer’s, but I’m really confident the memory leak was fixed.

Now that I had this memory leak by just using functionalities from ASP.NET, I’m not sure if it is like this by design or that it is really a problem in ASP.NET itself? I hope to find an answer soon, but in the meanwhile I call it dirty behaviour of ASP.NET itself ;-).

Kristof Rennen Development .NET, ASP.NET, Memory, Microsoft, Profiling