Currently I’m working on an ASP.NET application for one of our customers. The application is nearly finished, and is going to be put in acceptance and production in a few weeks.
To test the scalability of the application, stress and load tests are run using Visual Studio Team Test at our side and Load Runner at the customers side.
The stress tests are all running well on both sides, except for some weird OutOfMemory exceptions after a run of 9 or 10 hours with 100 concurrent users. The scenarios are built to simulate one month of work in 10 hours.
Setting up the environment
Since the application consists of a front end and backend, it was important for me to setup my IIS properly to enable me to find whether the problem was caused in the backend or frontend.
The backend application, is a WCF service host, running inside IIS. The frontend application is an ASP.NET application consuming the backend through WCF services.
The running both the service host and web application in a separate application pool, I was able to monitor both separate worker processes to narrow down the problem to one of the two.
Finding the cause of the OutOfMemory exceptions
Before one can start fixing the cause of the OutOfMemory exceptions, you first have to find the cause of course.
Using windbg, DebugDiag and DotTrace, we started investigating the memory usage and patterns of the applications finding that the allocated virtual memory was exceeding the IIS limit, causing OutOfMemory exceptions and unexpected pool recycling.
It was clear the application was suffering from a memory leak, which could still be a native or a managed leak.
A great source to find more information on tracing, Windbg, … is the blog of Tess Ferrandez, an ASP.NET Escalation Engineer of Microsoft. Her blog really helped me out during my search for the cause of memory leaks, and by analyzing all the memory dumps I created.
Reproducing the memory leak
Before one can start fixing the memory leak, its best to have a scenario to easily reproduce the problem.
We started by creating a scenario using a lot of functionalities from the application. By running this scenario we could easily reproduce the problem after a run of a few hours.
By narrowing the scenario down, up to a single functionality only, I was able to trigger the high memory usage faster and find the location of the bottleneck in the application.
I now had an easy way to reproduce the problem, and to verify the fix afterwards.
Tracing the memory usage
Since I now had a scenario which I could use to trigger the memory leak, I used JetBrains’ DotTrace profile to profile my IIS while running the scenario.
Everytime I ran the scenario, I noticed some ASP.NET UserControl not being released from memory. Since this user control, had references to other components as well, those components were “leaked” as well.
By using the trace output, I now had the name of the user control which was not being disposed properly.
The actual problem
The actual problem was a page containing a user control with the following structure.
Page -> UserControl -> Repeater -> UserControl
So the loaded page of the scenario, contained a user control of our own. This user control had a repeater, repeating over the rows of a generic list, creating another UserControl for each found row.
Since we are using ASP.NET with the Model-View-Presenter pattern (to be able to develop everything Test-Driven), we were calling the databind of the view manually as follows:
1 public override void DataBind()
2 {
3 //Set some UserControl properties here
4 UserControl.DataBind();
5
6 base.DataBind();
7 }
First we DataBind() the user control after its properties have been set. Then we call base.DataBind() which does a DataBind() of the complete page.
Ofcourse the DataBind() of the usercontrol is not really needed since it will be automatically called when you DataBind the complete page, but hey it should not cause any problems either.
Since the memory profile told us the user control was not properly cleaned up, we just put a breakpoint in its constructor and dispose method.
Analyzing this really blew us of our seats. The constructor of the user control was called twice, but the dispose method was only called once so we were stuck with one instance of the user control for ever. Hello Memory Leak!
The solution
Solving the memory leak was in the end easier than we should have imagined. By just removing the UserControl.DataBind() did the trick.
The DataBind() of the page now looks like this:
1 public override void DataBind()
2 {
3 //Set some UserControl properties here
4
5 base.DataBind();
6 }
Conclusion
I ran my stress tests again together with my DotTrace profiler and everything seemed to be working ok. The memory leak was fixed, and the memory usage of the application was back to normal.
Currently a new run of the stress tests are both running on our environment and our customer’s, but I’m really confident the memory leak was fixed.
Now that I had this memory leak by just using functionalities from ASP.NET, I’m not sure if it is like this by design or that it is really a problem in ASP.NET itself? I hope to find an answer soon, but in the meanwhile I call it dirty behaviour of ASP.NET itself ;-).
Kristof Rennen Development .NET, ASP.NET, Memory, Microsoft, Profiling