How to rewrite a MongoDB C# LINQ with a Projection Requirement using a MongoCursor

The LINQ Provider for MongoDB does not currently take into account data projections efficiently when returning data. This could mean that you’re unnecessarily returning more data from the database than is needed.

So, I’m going to show you the pattern I applied as a replacement for the LINQ queries when I need to use a projection.

Given the following simple LINQ statement:

var query = 
    (from r in DataLayer.Database
         .GetCollection<Research>()
         .AsQueryable<Research>()
        where !r.Deleted
        select new
        {
            Id = r.Id,
            Title = r.Title,
            Created = r.Created
        }).Skip(PageSize * page).Take(PageSize);

it can be converted to a MongoCursor style search like this:

var cursor = DataLayer.Database.GetCollection<Research>()
    .FindAs<Research>(Query<Research>.NE(r => r.Deleted, true))
        .SetFields(
            Fields<Research>.Include(
                r => r.Id, 
                r => r.Title, 
                r => r.Created))
        .SetLimit(PageSize)
        .SetSkip(PageSize * page);           

I’ve attempted to format it in a similar way mostly for my own sanity. As you see, both queries first get access to the database collection. But, instead of using AsQueryable<T>, you’ll use the FindAs<T> method. The query is more verbose in the second example, although not overly so. I chose to keep it strongly typed by using the generic version Query<Research>. By doing so, it meant that I could use an Expression to set the field/property that was being queried, rather than rely on a string (I could have just passed “Deleted” as a string in the code).

By strongly typing the parameters in this way, it meant that the compile process can catch things like type mismatches (verifying that the value being compared is a Boolean for example as is the Deleted property).

Secondly, and this addresses the requirement of a result projection, I’ve included just those fields that are required by the UI rather than all of the fields in the document. One of the fields is potentially quite long (up to 1MB of text), and in this case, unnecessary in a summary list display in my web application. Here, I used the SetFields method of the MongoCursor.

The C# driver includes a static class called Fields (in a generic and non-generic form) which can be used to express the set of fields to be included/excluded. I’ll point out that there is an option to just pass in a list of strings to SetFields, but it’s not strongly typed. So again, no compile-time checking that I’ve got the property names correct. I’m going for safety here, so I chose the strongly-typed generic implementation of Fields<Research>. Then, using the Expression syntax again, I’ve selected the fields I needed as a parameter list.

Finally, I added some code to limit the result size and set the skip equivalent to the original LINQ query.

There are a number of other Query methods that you can use to build more complex operations.

For example:

var q = Query.And(
    Query<Research>.Exists(r => r.Title), 
    Query<Research>.Matches(
        r => r.Title, BsonRegularExpression.Create(new Regex("^R"))));

The above maps to the following MongoDB query:

{ "$and" : [{ "Title" : { "$exists" : true } }, { "Title" : /^R/ }] }

Title field exists and the Title field starts with an uppercase “R”.

While the Query style syntax is more verbose than the equivalent LINQ statement, the result still is expressive and very readable and maintainable.

FYI: If there’s an index on Title, then the /^R/ syntax returns the results the most efficiently in MongoDB (as it stops searching after the first character).

How to view the MongoDB Query when using the C# LINQ Provider

If you’re using the Official MongoDB C# Driver from 10gen, you may want to occasionally verify that the generated query matches your LINQ query (or at least that it’s building something efficient).

Take for example this query:

var query = 
    (from r in DataLayer.Database.GetCollection<Research>().AsQueryable<Research>()
        where !r.Deleted
        select new
        {
            Id = r.Id,
            Title = r.Title,
            Created = r.Created
        }).Skip(PageSize * page).Take(PageSize);

query.DebugWriteMongoQueryText();

I wanted to verify that the query was checking the Deleted property, and see if the projection was considered as part of the query, so a few lines of code later …

public static string ToMongoQueryText<TQueryable>(this IQueryable<TQueryable> query)
{
    return (query as MongoQueryable<TQueryable>).GetMongoQuery().ToString();            
}

[Conditional("DEBUG")]
public static void DebugWriteMongoQueryText<TQuerable>(this IQueryable<TQuerable> query)
{
    Debug.WriteLine("QUERY: " + query.ToMongoQueryText());
}

I added two extension methods to IQueryable. As the return type for the query in my sample is actually an anonymous type, it wasn’t something I could easily type in the Immediate Window in Visual Studio for example.

<>f__AnonymousType2<string,string,System.DateTime>

No thanks  <>f__AnonymousType2<string,string,System.DateTime>.

I added the Conditional Attribute so that the code would only execute in a Debug build of my application.

query.DebugWriteMongoQueryText();

The output of the query above was:

QUERY: { "Deleted" : { "$ne" : true } }

As you’ll see, the anonymous projection wasn’t considered. Unfortunately, that’s a known issue with the current driver. This mattered in my above example, as some of the fields could contain a large amount of data that wasn’t needed by the current display.

Knockout binding for JavaScript route fixup

Part one.

After the first round, I felt compelled to KnockOut the code a bit more. I’d mentioned I wasn’t pleased with the code exactly. It needed some refactoring.

So, I’ve created a new Knockout binding handler. This binding handler replaces  named parameters with a model’s properties in a path.

For example, given this object:

Property Name Value
id A123
first_name Aaron
state WI

The following paths would be converted thusly:

Original Replaced
/person/{id} /person/A123
/person/{state}/{id} /person/WI/A123

You get the idea. Here’s the JavaScript code:

ko.bindingHandlers['route'] = {
    // Examples: 
    //      <a data-bind="route: {model: $data, url: 'person_details', attr: 'href' }" >
    // or, you can shortcut the syntax to default to the currently bound object and just pass 
    // the url or the route name as a string directly
    //      <a data-bind="route: 'person_details' }" >
    update: function (element, valueAccessor, allBindingsAccessor, $data) {
        var valueUnwrapped = ko.utils.unwrapObservable(valueAccessor());
        var options = ko.bindingHandlers.route.options || {};

        // look for the model property 
        var model = ko.utils.unwrapObservable(valueUnwrapped['model']);
        if (typeof model !== 'object') {

            model = ko.utils.unwrapObservable($data);
            if (typeof model === 'undefined' || model === null) {
                throw new Error('set route model to object (or nothing bound?)');
            }
        }

        // look for the url property first
        var url = ko.utils.unwrapObservable(valueUnwrapped['url']);
        // validate we've got something as a url (might be a name, might be a full url)        
        if (typeof url !== 'string' || url == "") {
            url = valueUnwrapped;
            if (typeof url !== 'string' || url == "") {
                throw new Error("set route url property to route name or url directly");
            }
        }

        // is it on the keyed collection?
        var map = options.map;
        if (typeof map !== 'undefined' && map !== null) {
            if (map.hasOwnProperty(url)) {
                url = map[url];
            }
        }
        // check for a routing function as well
        var fn = options.routeNameToUrl;
        if (typeof fn === 'function') { url = fn.call(null, url); }
        // did we get something meaningful?
        if (url !== null && url !== '' && url.length > 0) {
            url = ko.bindingHandlers.route.buildUrl(url, model);            
        }
        // the url might need some fixin after a routing, anything goes here (might just be a default)
        fn = options.fixUrl;
        if (typeof fn === 'function') { url = fn.call(null, url); }
        element.setAttribute(ko.utils.unwrapObservable(valueUnwrapped['attr']) || 'href', url);
    },    

    // given a model, this function replaces named parameters in a simple string 
    // with values from the model
    //     /path/to/some/{id}/{category}
    // with object { 'id' : 'abc', 'category' : 'cars' }
    // becomes
    //     /path/to/some/abc/cars
    buildUrl : function(url, model) {
        // unfixed if there's not a thing
        if (typeof model === 'undefined' || model === null) { return url; }

        var propValue;
        for (var propName in model) {                
            if (model.hasOwnProperty(propName)) {
                propValue = model[propName];
                if (ko) { propValue = ko.utils.unwrapObservable(propValue); }

                if (typeof propValue === 'undefined' || propValue === null) {
                    propValue = "";
                } else {
                    propValue = propValue.toString();
                }
                url = url.replace('{' + propName.toLowerCase() + '}', propValue);
            }
        }
        return url;
    },

    options: {
        // ** convert a route name to a url through whatever means you'd like
        // routeNameToUrl : function(routeName) { return url; } 

        // ** anything you want, called after routeNameToUrl, might add a virtual directory
        // ** for example
        // fixUrl: function(url) { return url;  }  

        // ** A map route names to URLs **
        // all other functions are called if set (to possibly override this)
        // this is not required if you use one of the other functions
        // map : { 'a_route_name' : '/path/to/something/{id}/{action}' }        
    }
};

In another JavaScript file, I did initialize some of the options:

ko.bindingHandlers.route.options.routeNameToUrl = getRoute;
ko.bindingHandlers.route.options.fixUrl = app_url;

The getRoute function just maps a route name to a path, and the app_url prepends the virtual directory to the path as needed.

Here it is in use:

<div data-bind="foreach: data.persons">
    <h3 class="title" data-bind="text: Title"></h3>
    <div>
        <a data-bind="route: { model: $data, url: 'person_details', attr: 'href' } ">Details2</a>
        <a data-bind="route: '/data/details/{id}/{title}' ">Details</a>
        <a data-bind="route: 'person_details' ">Details</a>
    </div>
</div>

You’ll probably like the new way better syntactically at least compared to the old way. A route binding requires one input when used in it’s most basic form:

  • url = the URL or route name to use as the template for the replacement. It should contain (or later resolve to) curly-braced enclosed property name keys which will be substituted by values from the model. The value of the property could be either a route name (see options below) or a path.

When using just the basic form you can use the shortened syntax:

<a data-bind="route: 'research_details' ">Details</a>

This handily binds to the current object (via the bindingContext.$data property of the bindingHandler update function call, which is also the fourth parameter, which I’ve renamed to $data rather than the typical viewModel). So, you won’t need to necessarily (explicitly) set the model for the binding.

If you need a a bit more control, you can change the syntax and get access to a few other options (including direct access to the model you want to bind to if the current item isn’t directly what you want to use).

  • model = this is the object that contains the properties and values to be used as replacements within the url
  • attr (optional) = the name of the attribute to set the generated url into. Defaults to href if not set.

There are a few global options you can control as well:

  • routeNameToUrl = (function)(routeName) optionally, given a route name, should return the path (or the original value). Here you can do a lookup of routeName to path.
  • map = {object} the object properties should be route names and set equal to the path. this optional lookup is performed before the routeNameToUrl function is called.
  • fixUrl = (function)(url) do anything here. this is called after the mapping and routeNameToUrl is optionally called. I use this to correct javascript Ajax request paths by appending the application virtual directory

(Thanks to Ryan for the suggestion to use the $data on the bindingContext, and then again for the nudge to just use the 4th parameter. Smile )

Named routing with Knockout, ASP.NET MVC, and AttributeRouting, from JavaScript

It’s hard to describe exactly what I’ve built here, but I’m just throwing these pieces out on the Internet in case someone: a) finds them useful, or b) has a better solution.

Goals:

  1. Named routes, with a wee bit of a Rails feel
  2. A Knockout friendly syntax for binding to the values of a model to build an href
  3. Automatic replacement of values in the url with values from the model
  4. JavaScript generation (OK, not really a goal, but a necessary evil I suspected)

Solution:

I started by using AttributeRouting (from AttributeRouting.net). It offers a simple syntax for creating routes (and naming them):

[sourcecode language=”csharp”]
[GET("research/{id}", RouteName = "research_details")]
public ActionResult Details(string id)
{
var model = new ResearchDetailsViewModel();
[/sourcecode]

In RouteConfig.cs, code gathers the list of named routes, and builds a snippet of JavaScript which will be inserted on every page (yes, it would be nice if this were cached, see refactoring below):

[sourcecode language=”csharp”]

var named = new List();
foreach (var route in RouteTable.Routes)
{
AttributeRoute r = route as AttributeRoute;
   if (r != null)
   {
      if (!string.IsNullOrWhiteSpace(r.RouteName))
      {
         named.Add(r);
      }
   }
}
if (named.Count > 0)
{
   StringBuilder js = new StringBuilder();
   foreach (var namedRoute in named)
   {
      js.AppendFormat(@"function {0}_url(model){{ return buildUrl(""{1}"", model); }}",
      namedRoute.RouteName, namedRoute.Url);
      js.AppendLine();
   }
   RouteScript = js.ToString();
}
[/sourcecode]

The name of the function is based on the RouteName property of the attribute.

I added a function called buildUrl to a common JavaScript file:

[sourcecode language=”javascript”]

function buildUrl(url, model) {
// unfixed if there’s not a thing
  if (typeof model === ‘undefined’ || model === null) { return url; }

var propValue;
  for (var propName in model) {
       if (model.hasOwnProperty(propName)) {
propValue = model[propName];
        if (ko) { propValue = ko.utils.unwrapObservable(propValue); }

if (typeof propValue === ‘undefined’ || propValue === null) {
           propValue = "";
} else {
          propValue = propValue.toString();
       }
       url = url.replace(‘{‘ + propName.toLowerCase() + ‘}’, propValue);
  }
   }
   return url;
}
[/sourcecode]

The code above validates there is data and then proceeds to loop through every property of the model to see if there might be a replacement needed within the passed URL. If Knockout is available (if ko check), then it unwraps the value as the code needs the raw value from this point onward.

It lowercases each property name and tries a replacement.

So, if the string was /research/{id}/{category}, and the model has properties named, Id and Category, the values of each will be substituted and returned as a full string. For example, it might be /research/123/Coding.

Next, a simple example of using some JSON data:

[sourcecode language=”javascript”]
$(function () {
var vm = {
url: function ($data) {
return app_url(research_details_url($data));
},
data : ko.observable()
};

$.getJSON(app_url("api/research/")).success(function (data) {
vm.data = ko.mapping.fromJS(data);
ko.applyBindings(vm);
});
});
[/sourcecode]

Then, finally, the Knockout template:

[sourcecode language=”html”]
<div data-bind="foreach: data.research">
<h3 class="title" data-bind="text: Title"></h3>
<div>
<a data-bind="attr: { href: $parent.url($data) }">Details</a>
</div>
</div>
[/sourcecode]

By using $parent in the template, Knockout points at the view model in this case (vm). On the view model, I added a function called url. Right now, I’ve hard coded it to point to a specific dynamically generated function called research_details_url. It’s passing the array item (which simply has an Id & Title property). So, the data-bind for the anchor (A) element assigns the value of the computation to the href attribute.

app_url:

[sourcecode language=”javascript”]

function app_url(url) {
return "@Request.ApplicationPath" + url;
}

[/sourcecode]

As I said, it’s not super elegant, but it achieved my basic goals.

Elegance rating: 59%

Function rating: 100%

Need of some refactoring rating: 99%

Nest Thermostat Wifi Connectivity Issues?

I’ve been seeing more Wifi issues recently with my Nest thermostats. I don’t see a pattern to the problem, and it seems to randomly affect all of them (sometimes at the same time, but often just one).

image

I’ve got a supported Wifi router (Apple Airport Express), and it’s within reasonable range to all of the thermostats in our house (no more than 20 foot from any thermostat). While it’s unfortunately overlapping with a similar band/frequency in our house, the router is dedicated to only the Thermostats right now (a unique SSID).

It doesn’t have any settings that seem necessary to change:

SNAGHTMLe2a23de

image

Any ideas?