Broken Reference to SharePoint 2010 Content Database

We got a nice surprise right before leaving for the Easter holiday. One of our web applications stopped working and it was sending all the users to the custom error page.

Checking the event viewer and the ULS logs we could identify this error message:

Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          4/5/2012 4:29:46 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SERVERNAME
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 4/5/2012 4:29:46 PM
Event time (UTC): 4/5/2012 9:29:46 PM
Event ID: e14426efea4a44bb9aa16d14ea767445
Event sequence: 57
Event occurrence: 21
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/1726127450/ROOT-1-129781347974963750
Trust level: WSS_Minimal
Application Virtual Path: /
Application Path:
E:\inetpub\wwwroot\wss\VirtualDirectories\WEBAPPNAME\
Machine name: SERVERNAME

Process information:
Process ID: 5064
Process name: w3wp.exe
Account name:DOMAIN\APPLICATIONPOOLACCOUNT

Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.

Request information:
Request URL: http://WEBAPPURL/Pages/Error.aspx
Request path: /Pages/Error.aspx
User host address: 10.0.0.2
User:
Is authenticated: False
Authentication Type:
Thread account name: DOMAIN\APPLICATIONPOOLACCOUNT

Thread information:
Thread ID: 8
Thread account name: DOMAIN\APPLICATIONPOOLACCOUNT
Is impersonating: True
Stack trace:    at
Microsoft.SharePoint.SPSite.PreinitializeServer(SPRequest request)
at Microsoft.SharePoint.SPWeb.InitializeSPRequest()
at Microsoft.SharePoint.WebControls.SPControl.EnsureSPWebRequest(SPWeb web)
at Microsoft.SharePoint.WebControls.SPControl.SPWebEnsureSPControl(HttpContext
context)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.GetContextWeb(HttpContext
context)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PostResolveRequestCacheHandler(Object
oSender, EventArgs ea)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step,
Boolean& completedSynchronously)

I’ve seen this same error message before when we restored a content database backup and got an error navigating on the site . I tried to resolve it applying the same steps but it didn’t work.

Checking further the issue we found out this error message on the Microsoft.SharePoint.SPSite.PreinitializeServer method has a relationship with the database connection. On Central Admin, under Application Management > Manage Content Databases I checked the content database for this web application and I couldn’t find it, there was no content database attached to the web application. It was a huge surprise. Then checking for the same content database on Upgrade and Migration > Review database status screen I was able to find. I tried to remove it from the web application but it was throwing an “object reference is null” exception.

We then went to the lower level attempts on solving and started trying Powrshell and stsadm scripts. We tried  to run the following commands:

  • remove-spcontentdatabase -identity “database name”
  • test-spcontentdatabase -name “database name” -webapplication “web application url”
  • stsadm -o deletecontentdb -databasename “database name”
  • stsadm -o addcontetdb -databasename “database name”
  • mount-spcontentdatabase -name “database name” -webapplication “web application url”
  • dismount-spcontentdatabase -identity “database name”

None of them worked.

In order to get more information about the content database we ran the following commands:
$db =  Get-SPDatabase | where {$_.Name -eq “database name”}
Then we printed out the $db object and got an output like this:

Id                                : 1c5db325-b77f-4f18-a7a1-664ca0d05249
Name                         : content database name
WebApplication    :
Server                       : database server name
CurrentSiteCount : 1

The WebApplication property was empty, and checking the other content databases we could see a value like “SPWebApplication name=web application name”.

Since this was a PROD issue we called Microsoft support and got a SharePoint Engineer to handle it. After a little research on the servers, lot of Powershell commands we got the confirmation of what was wrong. The content database was recognized by SharePoint as part of the farm but it wasn’t recognized as a content database. It was returned when we ran the get-spdatabase Powershell cmdlet,  but it wasn’t returned when we ran the get-spcontentdatabase Powershell cmdlet.

The solution for this issue was:

Run a Powershell script to get and deleter the content database:
$db =  Get-SPDatabase | where {$_.Name -eq “database name”}

$db.delete()

Actually, it doesn’t delete the database itself but removes all references SharePoint has to it.

Then we attached the content database back to the web application using Central Admin > Application Management > Manage Content Databases page.

It worked liked a charm and everybody could go have fun and enjoy their Easter holiday…..a few hours late of course. lol.

See you,

Amadeu.

Unexpected System.NullReferenceException After Restoring a SharePoint 2010 Content Database

After a user deleted lots of permissions on a SharePoint site we decided to ask for a restore of the content database in order to solve the issue.

Everything was fine, we had all the backups, we knew the time the user deleted the permission assignments and the had the downtime window to restore the backup.

The restore process went fast and fine, the database has been restored and I attached it back to SharePoint.

I tested the site on one of the WFE servers and it looked great, but when I went to the second load balanced server I got the following error message:

Unexpected System.NullReferenceException: Object reference not set to an instance of an object.

at Microsoft.SharePoint.SPSite.PreinitializeServer(SPRequest request)
at Microsoft.SharePoint.SPWeb.InitializeSPRequest()
at Microsoft.SharePoint.WebControls.SPControl.EnsureSPWebRequest(SPWeb web)
at Microsoft.SharePoint.WebControls.SPControl.SPWebEnsureSPControl(HttpContext context)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.GetContextWeb(HttpContext context)
at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PostResolveRequestCacheHandler(Object oSender, EventArgs ea)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& complet…

It didn’t make any sense. Why only one of the servers was not working?

I tried the regular debug approach but none of the actions worked:

  • Restart the application pool.
  • Reboot the server.
  • Attach and dettach the content database.

I also tried to connect to the SPWeb using Powershell and it worked on one server and didn’t work on the other one.

I went then to Google to research a little bit and after several false positive articles, I found this article explaining the issue (thank you very much Giles!!!):

http://ghamson.wordpress.com/2011/10/28/sharepoint-web-front-end-http-404-response-but-no-errors-in-the-log-sp2010-ps2010-msproject-projectserver-in/#comment-429

SharePoint has a configuration cache and if it is not cleared you can have this kind of issue.

Follow the steps to solve the issue:

  1. Stop the OWSTIMER service on the problem SharePoint WFE server in the farm.
  2. On the problem server, navigate to:

    Windows Server 2003 location: Drive:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\GUID and delete all the XML files from the directory.
    Windows Server 2008 location: Drive:\ProgramData\Microsoft\SharePoint\Config\GUID and delete all the XML files from the directory.

  3. Delete all the XML files in the directory. NOTE: ONLY THE XML FILES, NOT THE .INI FILE.
  4. Open the cache.ini with Notepad and reset the number to 1. Save and close the file.
  5. Start the OWSTIMER service on the server and wait for XML files to begin to reappear in the directory.
  6. IIS Reset (optional) – you can just recycle the app pool and it will work.

 

Thanks @twlamb for the revision…..

See you,

Amadeu.