My Insights

Large list performance: SPMetal vs. SPQuery

Pranav Sharma — Fri, 16 Dec 2011 06:14:20 +0000

Key takeaway

Use SPQuery for large lists instead of SPMetal. While testing with a list of 45K items, SPQuery performed at 0.06s compared to SPMetal’s performance at 9.98s.

Background

I was recently faced with determining the storage mechanism for a custom application. The choices were ‘SharePoint lists vs. SQL tables’. The user interface of this application would be implemented as a set of webparts either way. Now SharePoint has the advantage on UI which allows end users to easily interact with the data. SQL on the other hand takes the trophy on performance. So the choice really boiled down to ‘End-User Access vs. Performance’. I really liked the idea of users being able to interact with the data so the obvious question arose: Could SharePoint perform reasonably well when dealing with ‘large’ data sets?

Throttle Levels

What is considered ‘large’ in the SharePoint world? For end users, 5000 items is where SharePoint starts to throttle queries. Using the object model however, queries of over 20,000 items leads to an Expensive Query Exception. This behavior can be overriden by setting the SPQuery.QueryThrottleMode to SPQueryThrottleOption.Override. Additionally, these limits can be increased from Central Admin. So in order to create a large enough data set, let’s populated a blank custom list with 45,000 rows of data by setting the title column to a unique number. Additionally, for roughly 100 of these list items, let’s set a second column’s value (Column=Campaign) to a predetermined text since that way we can pull these 100 rows as part of a query.

using (var context = new ClientDataContext("http://site"))
{
    for (int i = 0; i < 45000; i++)
    {
        var newItem = new PerfListItem {Title = i.ToString()};
            if (i >= 44900)
                newItem.Campaign = "ABC";
            context.PerfList.InsertOnSubmit(newItem);
    }
    context.SubmitChanges();
}

Column Indexing

Before diving into the test results, let me briefly point out that SharePoint list columns can be indexed for better performance. When dealing with large lists, throttling kicks in on sort, where and join operations if dealing with non-indexed columns. For example, a where clause on a non-indexed column is a resource-intensive operation and SharePoint will throttle the query leading to unexpected results. The correct way to deal with large lists is to index any columns that would be used for any sorting, filtering or lookups. For the purposes of our test, I will be filtering based upon the ‘Campaign’ column so let’s add an index for that. Additionally, let’s add an index for the ‘Content Type’ column since SPMetal queries add a ContentTypeId filter to the underlying CAML.

SPMetal Test

Let me begin by saying that I’m a huge fan of SPMetal. For those who don’t know, SPMetal is a command-line tool that generates entity classes, which are primarily used in LINQ to SharePoint queries. Let’s setup this test as a simple filter query which consequently displays the count of the result set.

using (var context = new ClientDataContext("http://site"))
{
    var res = from item in context.PerfList
              where item.Campaign == "ABC"
              select item;
    Console.WriteLine(res.Count());
}

The result displayed a whopping count of 100 as expected and the execution time was an average of 9.98s. Test was conducted 3 times and the elapsed time was measured using System.Diagnostics.Stopwatch.

Extracting SPMetal’s underlying CAML

SPMetal’s performance of 9.98s wasn’t exactly stellar so let’s try a native CAML SPQuery instead. Since SPMetal ultimately turns LINQ queries into CAML queries, a fair test would be to try the SPMetal generated CAML query directly against the list. In order to extract the underlying query used by SPMetal, we can assign a TextWriter to the Log property of the data context.

var sb = new StringBuilder();
context.Log = new StringWriter(sb);
//LINQ Query
Console.WriteLine(sb.ToString());

This results in the following CAML-

<View>
  <Query>
    <Where>
      <And>
        <BeginsWith>
          <FieldRef Name="ContentTypeId" />
          <Value Type="ContentTypeId">0x0100Value>
        BeginsWith>
        <Eq>
          <FieldRef Name="Campaign" />
          <Value Type="Text">ABCValue>
        Eq>
      And>
    Where>
  Query>
  <ViewFields>
    <FieldRef Name="Campaign" />
    <FieldRef Name="ID" />
    <FieldRef Name="owshiddenversion" />
    <FieldRef Name="FileDirRef" />
    <FieldRef Name="Title" />
  ViewFields>
  <RowLimit Paged="TRUE">2147483647RowLimit>
View>

SPQuery Test

Finally, let’s setup the test for a native CAML SPQuery against the same list.

var query = new SPQuery
                {
                    ViewXml = "" +
                    "0x0100" +
                    "ABC" +
                    "" +
                    "" +
                    "" +
                    "2147483647"
                };
using (var site = new SPSite("http://site"))
{
    using (SPWeb web = site.OpenWeb())
    {
        SPList list = web.Lists["PerfList"];
        SPListItemCollection items =list.GetItems(query);
        Console.WriteLine(items.Count);
    }
}

The result? A surprising 0.06s on average! Again the test was conducted 3 times and elapsed the time was measured using System.Diagnostics.Stopwatch.

Machine Specs

For anyone that is interested, these tests were performed on a Windows 7 Client machine with Intel Core 2 Duo 2.66GHz & 8GB RAM.

Conclusion

I finally got the answer to my initial question: It IS possible to both leverage the UI capabilities of SharePoint while working with large data sets. It is also painfully obvious that while SPMetal can save hours upon hours of development time, it clearly comes at a price. A price that gets prohibitively expensive as the data set grows in size. The lesson here is to be cognizant of the performance implications of SPMetal while balancing the need for faster development.

Blank Open/Save Dialog when browsing Document Library from Office clients

Pranav Sharma — Sun, 28 Aug 2011 21:05:52 +0000

Problem

When you browse to a document library from an office client dialog such as ‘MS Word > Open’, the dialog shows a blank space instead of the document library contents-

Root Cause

This issue is caused due to the lack of an appropriate ‘FormDialog’ view available in the document library schema. Any document library definitions created by Visual Studio 2010 suffer from this issue. If you notice, these DocLib schemas provisioned by VS do actually contain a stub for a file dialog view but this stub is not sufficient in and of itself for proper rendering within an office client dialog-

<View BaseViewID="2" Type="HTML" FileDialog="TRUE" TabularView="FALSE" DisplayName="$Resources:core,File_Dialog_View;" Hidden="TRUE" Path="filedlg.htm" ModerationType="Moderator">

Solution

If you are in fact working with a visual studio provisioned schema for a document library (most common scenario for this issue) then simply find the FileDialog view (BaseViewId=”2” as shown above) and make the following modifications to the view-

Remove the toolbar
```
<Toolbar Type="Standard" />
```
Add ViewHeader
```
<ViewHeader>
  <SetVar Name="FileDialog">1SetVar>
  <HTML>
    [CDATA[
    <table id="FileDialogViewTable" width="100%" style="cursor: default;" border="0" rules="rows" cellspacing="0" cellpadding="2">
<tr>
]]>
  HTML>
  <Fields>
    <HTML>[CDATA[<th class="ms-vh2-nofilter">]]>HTML>
    <Field />
    <HTML>[CDATA[th>]]>HTML>
  Fields>
  <HTML>[CDATA[tr>]]>HTML>
ViewHeader>
```

Add ViewBody

<ViewBody>
  <SetVar Name="FileDialog">1SetVar>
  <IfEqual>
    <Expr1>
      <GetVar Name="AlternateStyle" />
    Expr1>
    <Expr2>ms-alternatingExpr2>
    <Then>
      <SetVar Scope="Request" Name="AlternateStyle">
      SetVar>
    Then>
    <Else>
      <SetVar Scope="Request" Name="AlternateStyle">ms-alternatingSetVar>
    Else>
  IfEqual>
  <Switch>
    <Expr>
      <LookupColumn Name="FSObjType" />
    Expr>
    <Case Value="1">
      <HTML><TR fileattribute=folder ID="HTML>
    Case>
    <Default>
      <HTML><TR fileattribute=file ID="HTML>
    Default>
  Switch>
  <Field Name="EncodedAbsUrl" />
  <HTML>[CDATA[" class="]]>HTML>
  <GetVar Name="AlternateStyle" />
  <HTML>[CDATA[" onmousedown="selectrow()" onclick="selectrow()">]]>HTML>
  <Fields>
    <HTML>[CDATA[<td class="ms-vb" style="padding-left: 4px">]]>HTML>
    <FieldSwitch>
      <Expr>
<Property Select="Type" />
      Expr>
      <Case Value="User">
<LookupColumn HTMLEncode="TRUE" />
      Case>
      <Default>
<FieldSwitch>
  <Expr>
    <Property Select="Name" />
  Expr>
  <Case Value="CheckoutUser">
    <Field HTMLEncode="TRUE" />
  Case>
  <Default>
    <Field />
  Default>
FieldSwitch>
      Default>
    FieldSwitch>
    <HTML>[CDATA[td>]]>HTML>
  Fields>
  <HTML>[CDATA[tr>]]>HTML>
ViewBody>

Add ViewFooter
```
<ViewFooter>
  <HTML>[CDATA[table>]]>HTML>
ViewFooter>
```

Add ViewEmpty

<ViewEmpty>
  <SetVar Name="FileDialog">1SetVar>
  <HTML>
    [CDATA[
    <table id="FileDialogViewTable" width="100%" style="cursor: default;" border="0" rules="rows" cellspacing="0" cellpadding="2">
<tr>
]]>
  HTML>
  <Fields>
    <Switch>
      <Expr>
<Property Select="Name" />
      Expr>
      <Case Value="FileLeafRef">
      Case>
      <Default>
<HTML>[CDATA[<th class="ms-vh2-nofilter">]]>HTML>
<Field />
<HTML>[CDATA[th>]]>HTML>
      Default>
    Switch>
  Fields>
  <HTML>[CDATA[tr>table>]]>HTML>
  <HTML>[CDATA[<table width="100%" border="0" rules="rows"><tr>]]>HTML>
  <HTML>[CDATA[<td class="ms-vb">]]>HTML>
  <HTML>$Resources:core,noDocOfSpecType;HTML>
  <HTML>[CDATA[td>tr>table>]]>HTML>
ViewEmpty>

Things to note

If you’re creating your own list schema by hand, simply ensure that there is a view definition with the following attributes-
```
<View BaseViewID="" Type="HTML" FileDialog="TRUE" TabularView="FALSE" DisplayName="" Hidden="TRUE" Path="filedlg.htm" ModerationType="Moderator">
```
For lists that have already been provisioned using a custom schema, simply update the schema definition and redeploy to enact the change in existing list instances. Be sure to re-activate the feature so that an updated schema is read from the disk.

Rendering User Control ASCX via XSL in SharePoint

Pranav Sharma — Fri, 12 Aug 2011 17:19:50 +0000

Background & The Problem

Last time we discussed how to integrate a custom control into an XSL transformation. The key point to note about the previously discussed approach is that all the markup is output by the overridden Render method. This can easily get cumbersome and ineffective for complex server side controls. The ASP.net stack offers us a solution to this common problem with Control definitions. This solution is a UserControl which adds a special ASCX file containing declarative markup which makes the development task much easier, simpler & cleaner.

In this post, we will show how to incorporate a server side UserControl with markup in it’s ASCX file to render within an XSLT instead of a simple Control definition.

How to do this

In order to render a UserControl ASCX within an XSLT, we need to follow a slightly different method that the one discussed in the previous post. Here are the steps-

First thing we need to do is to define our user control. Here is a sample ASCX definition. Notice that this control renders a variable called ‘Input’ which we will also define in the code-behind class in the next step-
```
<%@ Assembly Name="$SharePoint.Project.AssemblyFullName$" %>
<%@ Assembly Name="Microsoft.Web.CommandUI, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Control Language="C#" AutoEventWireup="true" CodeBehind="DemoControl.ascx.cs" Inherits="Demo.ControlTemplates.DemoControl" %>

<div>(<%= Input %>)div>
```
Next, we define the code-behind that our ASCX inherits from. Notice that we are declaring a public string property called ‘Input’ whose value is being directly rendered within the ASCX as an example. This variable is used to accept contextual information about the XML node being processed by the XSL transform at the time of evaluating this control during the runtime-
```
using System.Web.UI;

namespace Demo.ControlTemplates
{
    public partial class DemoControl : UserControl
    {
        public string Input;
    }
}
```
Next, we need to globally register our user control so that our XSLT can find it. This is done because we don’t have the ability to register custom user controls from within an XSLT webpart. The following registration happens in the web.config in this node:
…
…
…
. Notice that we assign a tagName for this user control. This tag name will be used when we reference the user control in our XSLT-

<add tagPrefix="PSControl" src="/_controltemplates/demo/democontrol.ascx" tagName="PSC"/>
Now that SharePoint globally knows about this user control, we need to modify our xsl declaration to assign a tag prefix for this control so that we can reference it wherever we want to use it. In this case, our tag prefix is ‘PSControl’-
```
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:PSControl="Demo.ControlTemplates" >
```
Now that everything has been setup, we can simply reference our custom user control using the tag prefix defined in step 4 (PSControl) and the tag name defined in step 3 (PSC). Combined this tag declaration allows us to render our custom user control ASCX which was written declaratively to save development time associated with defining a custom server side control for our XSLT. Notice that we are using the Input property defined in the UserControl and passing it contextual value about the current node being processed in this case-
```
<PSControl:PSC Input="{title}" runat="server" />
```

That’s it! You can now write declarative markup in a UserControl ASCX file and have that markup process XML nodes using a code-behind, all within the context of an XSL transform.

Custom Controls in XSLT

Pranav Sharma — Wed, 08 Jun 2011 19:04:31 +0000

Background & The Dream

SharePoint relies heavily on XML data and XSLT processing of such data. Whether we are dealing with Search results, Data view web parts or Content Query web parts, XSLT processing is something all SharePoint developers have to deal with on regular basis. SharePoint provides an XSLT extension object called ddwrt to perform tasks such as accessing properties of a SharePoint list or firing events to connected Web Parts.

Even though XSL transforms are incredibly powerful, especially with the availability of extension objects like ddwrt, sometimes it is much quicker and easier to perform advanced processing using server side controls. In other cases, it’s almost a necessity. Imagine being able to take xml data and query external web services during the transformation process to render customized contextual information about the data being displayed to the user.

For example, let’s say we were to implement a staff directory using SharePoint people search. The underlying data is stored in the User Profile service application and the results are rendered using the people core results web part. However what if instead of simply displaying a list of people, we additionally wanted to display a list of projects that they are participating in? In this case, imagine the power of being able to insert a server side control in the XSLT to query a 3rd party project management system to retrieve the list of a particular user’s projects and then rendering that nested within the rest of the XSL transform

How to do it

There are 4 steps needed in order to be able to insert server side controls within a SharePoint XSLT-

First thing we need to do is to write the control that we will be inserting into our XSLT. Notice that we are declaring two public properties called ‘itemUrl’ and ‘print’. These properties are used to pass xml data at run-time into our control. We are also overriding the Render method which is used to spit out the code that will be output to the end user-

namespace PortalSolutions.WebUI.Controls
{
    public class SampleFields : Control
    {
        public string itemUrl;
        public string print;

        protected override void Render(HtmlTextWriter writer)
        {
            writer.Write("Hello World
");
        }
    }
}

The next thing for us to accomplish is to globally register our assembly and a tagPrefix for this assembly so that SharePoint can find our control. This is done because we don’t have access to register our assembly from within the XSL editor on SharePoint web parts. This registration is added in the web.config under the following node:
…
…
…
-

<add tagPrefix="PS" namespace="PortalSolutions.WebUI.Controls" assembly="PortalSolutions.WebUI, Version=1.0.0.0, Culture=neutral, PublicKeyToken=00aaa666444222cc" />
Now that SharePoint knows about our assembly and can find the control, we have add a declaration on top of XSL to attach a prefix to our namespace so that we can reference our control. In this case, ‘PS’ is the tag prefix we will be using-
```
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:PS="PortalSolutions.WebUI.Controls" >
```
Finally, now that we have everything set, we simply have to reference our control where we want to render it by using our tag prefix we registered in step 3 and the name of our control class which in this case is ‘SampleFields’. Additionally we use the public properties declared in the control definition as attributes in our XSL to pass in any information that the control may need to process the current XML node-
```
<PS:SampleFields itemUrl="{$url}" print="HelloWorld" runat="server" />
```

And there we have it! A custom server side control puts a lot of power in the hands of developers looking to deliver some really powerful and amazing solutions.

Missing User Profile values after migration

Pranav Sharma — Thu, 14 Apr 2011 15:08:41 +0000

Problem

Are you missing user profile values for certain fields after a migration of the SSP database to SharePoint 2010 User Profile Service Application? If you look closely, the properties that are missing are managed metadata properties and this is a common oversight during a migration.

Solution

Run the “Move-SPProfileManagedMetadataProperty” commandlet on each of the missing properties-

Move-SPProfileManagedMetadataProperty
-ProfileServiceApplicationProxy ee475d63-fa17-4439-ac5c-656c164a86c3 -Identity SPS-Skills -AvailableForTagging
-TermSetName myTermSet

SPS-School
LanguagesSpoken
Areasofexpertise
Areasofstudy
Degrees
SPS-Responsibility
SPS-Skills
SPS-PastProjects
AreasofInterest

Search Results Hidden Object

Pranav Sharma — Thu, 14 Apr 2011 01:54:50 +0000

SharePoint search webparts leverage an internal private class called SearchResultsHiddenObject which does a lot of heavy lifting around query execution. This object is also responsible for ‘connecting’ results between webparts such as the Paging, Core Results and Search Actions webparts. Now what if you want to do a custom keyword query or perhaps even a full text query?

The typical answer is to write a custom core results webpart, but what about the other related webparts? You could in theory write a new paging webpart and a new search actions webpart and so on but wouldn’t it be nice if all these related webparts just worked with your new core results webpart? The answer is that they can but you have to get a little creative about modifying the SearchResultsHiddenObject. Here is an example where we disable duplicate trimming for correct paging behavior-

object srho = HttpContext.Current.Items["OSSSRHDC_0"];
if (srho != null)
{
    Type srhoType = srho.GetType();
    FieldInfo startAt = srhoType.GetField("_DuplicatesRemoved", BindingFlags.Instance | BindingFlags.NonPublic);
    startAt.SetValue(srho, false);
}

Here is a great walkthrough by Corey Roth on this topic: http://www.dotnetmafia.com/blogs/dotnettipoftheday/archive/2008/06/09/new-web-part-for-wildcard-search-in-enterprise-search.aspx

Change display name on CBA dropdown items

Pranav Sharma — Tue, 01 Mar 2011 07:00:35 +0000

Here is an easy way to change the display names for the authentication providers on the claims drop down in sharepoint 2010. Simply create a new signin page with a LogonSelector control and add this to the code-behind:

protected override void OnPreRenderComplete(EventArgs e)
{
    base.OnPreRenderComplete(e);

    var ntlm = ClaimsLogonSelector.Items.FindByText("Windows Authentication");
    if (ntlm != null)
    {
        ClaimsLogonSelector.Items.Remove(ntlm);
        ntlm.Text = "Internal";
        ClaimsLogonSelector.Items.Add(ntlm);
    }

    var fba = ClaimsLogonSelector.Items.FindByText("Forms Authentication");
    if (fba != null)
    {
        ClaimsLogonSelector.Items.Remove(fba);
        fba.Text = "Guest";
        ClaimsLogonSelector.Items.Add(fba);
    }
}

Implement this in addition to the automatic signin solution to automatically forward internal users to windows authentication for a seamless intranet experience: http://spautomaticsignin.codeplex.com/

SharePoint 2010 Notifications & Statuses

Pranav Sharma — Mon, 13 Sep 2010 22:11:22 +0000

While the new SharePoint 2010 Dialog framework is great for modal interaction with the end user, there are times when a quick non-modal notification or a semi-permanent status message is more appropriate. The great news is that SP 2010 already ships with an OOTB Notification and Status framework in addition to the dialogs.

Notification

Notifications show up on the top-right of a page and by default show for 5 seconds.

SP.UI.Notify.addNotification("Custom Notification (5 sec)", false);

Status

Status messages are semi-permanent and stay on the screen until cleared

var statusId = SP.UI.Status.addStatus("Your email address has not yet been confirmed. Click Here to resend a confirmation email");
SP.UI.Status.setStatusPriColor(statusId, 'red');

Check out this MSDN article for more information: http://msdn.microsoft.com/en-us/library/ee658473.aspx

How to leverage SharePoint 2010 dialogs?

Pranav Sharma — Wed, 01 Sep 2010 02:39:53 +0000

SharePoint 2010 has a robust modal dialog system that can be easily leveraged to build highly interactive solutions. Imaging being able to pop-up item edit forms or displaying notifications all without redirecting the user back and forth!

Here is the client side JavaScript that will allow you to open any page within a dialog box-

//Handle the DialogCallback callback
function DialogCallback(dialogResult, returnValue){
    window.location = window.location;
}

//Open the Dialog
function OpenEditDialog(link){
    var options = {
        url:link + '&IsDlg=1',
        width: 700,
        height: 700,
        dialogReturnValueCallback: DialogCallback};
    SP.UI.ModalDialog.showModalDialog(options);
}

To invoke the dialog, simply call the OpenEditDialog function with the target URL.

Notice that the IsDlg=1 query string parameter is appended within the OpenEditDialog function. The presence of “IsDlg” dynamically loads the "/_layouts/styles/dlgframe.css” file which applies “display:none” to all items using the “s4-notdlg” css class. This effectively hides items of this class in a dialog box. For example, v4.master uses the “s4-notdlg” class for the Ribbon control to hide the ribbon within dialogs.

UPDATE (09/13/2010): There are times when a quick non-modal notification or a semi-permanent status message is more appropriate. Check out this post on how to leverage these new UI frameworks: http://www.pranavsharma.com/blog/2010/09/13/sharepoint-2010-notifications-statuses/

Inherit Site Theme programmatically

Pranav Sharma — Wed, 21 Jul 2010 01:36:29 +0000

The Microsoft.SharePoint.Utilities namespace contains a class called ThmxTheme which gives us some nice utility methods to interact with site themes in SharePoint 2010. Here is an example of how to automatically set the theme for a newly created SPWeb to be the same as the root web-

ThmxTheme.SetThemeUrlForWeb(newWeb, ThmxTheme.GetThemeUrlForWeb(newWeb.Site.RootWeb));

Do you know of a nicer way to inherit site themes? Please post a comment below.

UPDATE (09/09/10): An alternative way to do this-

ThmxTheme theme = ThmxTheme.Open(web.Site, "_catalogs/theme/MyTheme.thmx");
theme.ApplyTo(web, true);