Category / Coding

by Aaron

Nest Thermostat API/Protocol

While Nest Labs hasn’t released a formal (documented & supported) API, I thought I’d do a bit of digging to see how they’re using the network and what might be achievable.

A few things are going on, the majority as you’d probably expect.

The web interface is using a long polling technique apparently to watch for updates to the schedule, temperature, etc.

image

I haven’t determined what the frequency is though, or the wait time. It’s very inconsistent, even when I wouldn’t expect much new “live” data to be available on the network, it frequently updates and polls again.

There are a few constants set in the HOME page script:

C.ABSENT_USER_THRESHOLD     = +('300') || 0;  // seconds
C.DEAD_DEVICE_THRESHOLD     = +('300') || 0;  // seconds
C.pollingInterval           = +('2500') || 0;       // ms
C.WEATHER_POLLING_INTERVAL  = +('120000') || 0; // ms

 

If the C.pollingInterval value were for the subscribe endpoint mentioned above, I’d see a LOT more calls than I do – so I’m still not clear how the polling interval is decided.

The API calls, for the most part are using JSONP syntax over an HTTPS connection.

The most frequent request is to “subscribe.” It sends as part of the GET request a large block of encoded JSON (using encodeURIComponent and then JSON.stringify).

I’m not familiar with the key/value system that they’re using (it may just be something they’ve constructed in-house – although given the number of open source JavaScript libraries they’re using, I thought someone might recognize it):

“key”, “{actualkey}.{value}”

I don’t understand why they’ve redundantly specified “key” in a list of keys when it’s evident that the actual key is contained within the value as a delimited string. It’s more data to send and more data to parse this way. So, again, maybe it’s based on some DB or model system I’m not familiar with. (Anyone recognize it?)

I’ve substituted the actual values (as they are serial numbers of my devices) with text representations of what the value represented below: 

{"keys":
    [{"key":"user.#USERID#",
        "version":209478897,"timestamp":1324159145000},
    {"key":"user_alert_dialog.#USERID#",
        "version":-1320296685,"timestamp":1325967612000},
    {"key":"structure.#STRUCTURE-GUID#",
        "version":656192675,"timestamp":1325967612000},
    {"key":"device.#DEVICE 1 SERIAL NUMBER#",
        "version":1485027516,"timestamp":1326034984000},
    {"key":"shared.#DEVICE 1 SERIAL NUMBER#",
        "version":588844038,"timestamp":1326034818000},
    {"key":"schedule.#DEVICE 1 SERIAL NUMBER#",
        "version":1187107985,"timestamp":1326005677000},
    {"key":"track.#DEVICE 1 SERIAL NUMBER#",
        "timestamp":1326035650601,"version":1041047847},
    {"key":"device.#DEVICE 2 SERIAL NUMBER#",
        "version":149169270,"timestamp":1326034820000},
    {"key":"shared.#DEVICE 2 SERIAL NUMBER#",
        "version":659841570,"timestamp":1326034820000},
    {"key":"schedule.#DEVICE 2 SERIAL NUMBER#",
        "version":-2016290692,"timestamp":1326005625000},
    {"key":"track.#DEVICE 2 SERIAL NUMBER#",
        "timestamp":1326035650862,"version":528978433},
    {"key":"device.#DEVICE 3 SERIAL NUMBER#",
        "version":1637112547,"timestamp":1326035399000},
    {"key":"shared.#DEVICE 3 SERIAL NUMBER#",
        "version":760504326,"timestamp":1326035397000},
    {"key":"schedule.#DEVICE 3 SERIAL NUMBER#",
        "version":-314552357,"timestamp":1326003402000},
    {"key":"track.#DEVICE 3 SERIAL NUMBER#",
        "version":-645931164,"timestamp":1326035531802}]}"

We’ve got three thermostats, so there are always three sets of subscription requests for each call to subscribe.

Using my iPad, I adjusted the set point for our second story (#DEVICE 2#) down one degree Fahrenheit (to 67°).

Within approximately a second, the most recent pending subscribe request returned with a far more interesting payload: 

jQuery17108417355176061392_1326035646750(
    { "status": 200,
        "headers": {
            "X-nl-skv-key": "shared.#DEVICE 2 SERIAL NUMBER#",
            "X-nl-skv-version": 869022424,
            "X-nl-skv-timestamp": 1326038279000,
            "X-nl-service-timestamp": 1326038279825
        },
        "payload": {
            "current_temperature": 19.98,
            "hvac_fan_state": false,
            "name": "TWO", "hvac_heat_x2_state": false,
            "hvac_ac_state": false,
            "can_cool": true,
            "auto_away": 0,
            "compressor_lockout_enabled": false,
            "target_temperature_low": 16.66667,
            "target_temperature_high": 26.66667,
            "compressor_lockout_timeout": 0,
            "hvac_heater_state": false,
            "hvac_aux_heater_state": false,
            "target_temperature": 19.44444,
            "can_heat": true,
            "target_temperature_type": "heat",
            "target_change_pending": true
        }
    });

Everything above is needed to update the current state of the UI. As you can see, the current temperature (returned as Celsius apparently) is 19.98 (67.964°F). The current temperature as displayed on the thermostat and the web UI was 68.

Seeing these return values makes me think that they may be using Ruby and Rails (as the naming convention tends to follow Rails naming using underscores between words). I know for example, I wouldn’t name variables/columns that way when building a C#/JavaScript MVC project.

Rather than just a delta payload of what’s changed, they’ve currently opted for a full update of all information related to the thermostat state.

Several seconds later, a much larger payload was returned to a subscribe request: 

"status": 200,
"headers": {
    "X-nl-skv-key": "device.#DEVICE 2 SERIAL NUMBER#",
    "X-nl-skv-version": -2086438581,
    "X-nl-skv-timestamp": 1326038378000,
    "X-nl-service-timestamp": 1326038379023
},
"payload": {
    "ob_orientation": "O",
    "upper_safety_temp": 1000.0,
    "forced_air": true,
    "creation_time": 1324142042019,
    "switch_preconditioning_control": false,
    "click_sound": "on",
    "leaf": false, "user_brightness": "auto",
    "learning_state": "steady",
    "heat_pump_comp_threshold": -1000.0,
    "local_ip": "10.0.0.205",
    "backplate_serial_number": "#SHOULD BE DEVICE 2 SERIAL NUMBER, BUT ISN'T?#",
    "capability_level": 1.03,
    "postal_code": "#POSTALCODE#",
    "upper_safety_temp_enabled": false,
    "heat_pump_aux_threshold": 10.0,
    "lower_safety_temp_enabled": true,
    "serial_number": "#DEVICE 2 SERIAL NUMBER#",
    "temperature_lock": false,
    "learning_time": 1002,
    "current_version": "1.0.4",
    "model_version": "Diamond-1.10",
    "backplate_bsl_info": "BSL",
    "auto_away_enable": true,
    "heat_pump_comp_threshold_enabled": false,
    "fan_mode": "auto",
    "range_enable": false,
    "temperature_scale": "F",
    "backplate_mono_info": "TFE (BP_DVT) 3.5.2 (ehs@ubuntu) 2011-11-05 12:00:00",
    "backplate_bsl_version": "1.1",
    "equipment_type": "gas",
    "range_mode": false,
    "lower_safety_temp": 7.0,
    "has_fan": true,
    "hvac_wires": "Heat,Cool,Fan,Common Wire,Rc",
    "learning_mode": true,
    "away_temperature_high": 32.0,
    "switch_system_off": false,
    "time_to_target": 1326039444,
    "away_temperature_low": 14.444444444444445,
    "current_humidity": 45,
    "mac_address": "#MACADDR#",
    "backplate_mono_version": "3.5.2",
    "has_aux_heat": false,
    "type": "TBD",
    "hvac_pins": "W1,Y1,C,Rc,G",
    "has_heat_pump": false,
    "heat_pump_aux_threshold_enabled": true,
    "battery_level": 3.945,
    "target_time_confidence": 1.0
}

 

A few things to note:

  • Upper_safety_temperature is just a bit beyond my comfort zone at 1832°F. I don’t know why it’s sending a value like that to the client, and why it’s stupidly high.
  • The backplate serial number doesn’t match with the thermostat according to the payload response. I don’t know why this might be as I confirmed that the numbers matched through visual inspection of the device just now.
  • The majority of these details are exposed in one way or another in the details area of the web UI.
  • Time to target (payload.time_to_target) is unusual in that it’s a JavaScript Date value, divided by 1000. So, in the example above, the time to target is: new Date(1326039444 * 1000).toString() = >”Sun Jan 08 2012 10:17:24 GMT-0600 (Central Standard Time)”Next, a payload is returned with the new status: 
    "status": 200,
"headers": {
    "X-nl-skv-key": "shared.#DEVICE 2 SERIAL NUMBER#",
    "X-nl-skv-version": 1689916148,
    "X-nl-skv-timestamp": 1326038378000,
    "X-nl-service-timestamp": 1326038379151
},
"payload": {
    "hvac_fan_state": false,
    "name": "TWO",
    "hvac_heat_x2_state": false,
    "hvac_ac_state": false,
    "can_cool": true,
    "auto_away": 0,
    "compressor_lockout_enabled": false,
    "target_temperature_low": 16.66667,
    "current_temperature": 19.53,
    "target_temperature_high": 26.66667,
    "compressor_lockout_timeout": 0,
    "target_change_pending": false,
    "hvac_aux_heater_state": false,
    "target_temperature": 20.55556,
    "can_heat": true,
    "target_temperature_type": "heat",
    "hvac_heater_state": true
}

     

    Here, the hvac_heater_state is set to true. The furnace is on.

    A little while later, that value is set to false.

    Occasionally, the payload includes the complete schedule for the thermostat. I’m not going to reproduce the entire payload here as it’s too large, and quite boring. Here’s a snippet of what it returns: 

    "schedule": {
    "#DEVICE 2 SERIAL NUMBER#": {
        "$version": 1187107985,
        "$timestamp": 1326005677000,
        "name": "One Current Schedule",
        "days": {
            "0": {
                "0": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 0,
                    "entry_type": "continuation"
                },
                "1": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 27900,
                    "entry_type": "setpoint"
                },
                "2": {
                    "type": "HEAT",
                    "temp": 20.556,
                    "time": 63000,
                    "entry_type": "setpoint"
                },
                "3": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 70200,
                    "entry_type": "setpoint"
                }
            },
            "1": {
                "0": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 0,
                    "entry_type": "continuation"
                },
                "1": {
                    "type": "HEAT",
                    "temp": 18.889,
                    "time": 20700,
                    "entry_type": "setpoint"
                },

     

    It’s a basic table structure. The first set point of the day is at 0, and is a “continuation.” These don’t show up in the UI.

    Here’s what the day 1 looks like on the Nest thermostat UI:

    image

    When changing a temperature setpoint, I’m a bit disappointed to see that the entire schedule is sent with every request apparently. I just wouldn’t have expected that given that the more setpoints that there are, the bigger the payload must be. The UI is often sluggish when rapidly making adjustments in the schedule, and this could be one of the factors.

    In the example below (which I’ve snipped most of the payload sent again as a JSONP request), I’ve set the first set point to 57F. 

        "payload": {
        "days": {
            "0": {
                "0": {
                    "type": "HEAT",
                    "temp": 14.685,
                    "time": 0,
                    "entry_type": "continuation"
                },
                "1": {
                    "type": "HEAT",
                    "temp": 15.000444444444444,
                    "time": 24300,
                    "entry_type": "setpoint"
                },

    For the JSONP requests sent as “MAKE CHANGE” (easily could have been PUT), they each contained the following attributes as shown below. All JSONP requests are apparently routed on the web server using “headers” rather than a RESTful URL based system: 

        },
    "headers": {
        "X-nl-client-timestamp": 1326041210566,
        "X-nl-session-id": "#SESSION ID#",
        "X-nl-protocol-version": 1,
        "Authorization": "Basic #BASIC AUTH#"
    },
    "path": "/v1/put/schedule.#DEVICE 2 SERIAL NUMBER#",
    "redir": "https://home.nest.com/post_jsonp",
    "jsonp": "4_"
}

    It’s RESTful in spirit as there is a route (“path”), but it’s managed by some internal routing engine. (Now, I think that they’re not using Ruby and Rails).

    For something simple, like changing the current temperature of a thermostat, the request is thankfully simple: 

    {
    "payload": {
        "shared": {
            "#DEVICE 2 SERIAL NUMBER#": {
                "target_temperature": 18.333333333333336
            }
        }
    },
    "headers": {
        "X-nl-client-timestamp": 1326041744556,
        "X-nl-session-id": "#SESSION ID#",
        "X-nl-protocol-version": 1,
        "Authorization": "Basic #BASIC AUTH#"
    },
    "path": "/v1/put",
    "redir": "https://home.nest.com/post_jsonp",
    "jsonp": "14_"
}

    While, I haven’t taken the time to try to write a custom UI for this undocumented API yet, it looks like it should be relatively easy to do, especially as it relates to the schedule and current temperature settings. I know there’s been some Siri proxy stuff that’s been written – but I don’t have any interest in trying to get that to work.

    As with most APIs like this, the trick is often getting authorization correct. For Nest, it appears that making a POST request to https://home.nest.com/accounts/login/ with username and password as form data, that the server responds with 2 cookies:

    1. sessionid == used in X-nl-session-id in headers
    2. cztoken == used as the Authorization in headers (prepended with the text “Basic “

    FYI: I also have a Node version of the API that is more up to date than this.

by Aaron

Ember.JS and EZdata, and Rails

I’ve been trying to do some additional work on my ember.js extension for data management. At the same time though, I’ve been trying (to learn and) build a simple Ruby on Rails web demo application using the new JavaScript library. There have been more than a few things that have mystified me about the framework and the structuring of an application. One aspect in particular was how to best manage foreign keys and join tables with the ActiveRecord class (and the corresponding SQL tables). So many tutorials have the same lame example of: a CART, an ORDER, a CUSTOMER …, that it’s often difficult to apply the same patterns to a more interesting system.

I started simple this time.

I wanted a PERSON class and a GIFT class.

image

A Person has been given gifts and may give gifts (and a few other common attributes). 

class CreatePeople < ActiveRecord::Migration
  def change
    create_table :people do |t|
      t.string :first_name
      t.string :last_name
      t.date :date_of_birth
      t.string :email_address

      t.timestamps
    end
  end
end

One of the things that I can’t decide if I like is the automatic pluralization of words, especially People/Person. I would have been content with a Persons table, but when creating a model, by default (as I’m aware it can be overridden), a Person is mapped to a table called “People.”

The second table, Gifts is very simple:

class CreateGifts < ActiveRecord::Migration
  def change
    create_table :gifts do |t|
      t.string :description
      t.integer :from_person_id
      t.integer :to_person_id
      t.timestamps
    end
  end
end

As I thought I might want a richer structure for the Gift class in the future, I did not use the more standard “person_id” name for the foreign key column that maps a gift to a Person. I wanted the column name to be more obvious what it was. Additionally, I needed two columns that both mapped to a “Person", so I couldn’t have both be called “person_id” anyway.

By deviating from the normal pattern, there are a few expectations when defining the ActiveRecord class. It was these expectations that weren’t clear to me (especially with examples).

The Ruby class for Gift is defined like so:

class Gift < ActiveRecord::Base
  belongs_to :from_person, :class_name => "Person", :foreign_key => "from_person_id"
  belongs_to :to_person, :class_name => "Person", :foreign_key => "to_person_id"
end

and the Person:

class Person < ActiveRecord::Base
  has_many  :gifts_given , :class_name => "Gift", :foreign_key => "from_person_id"
  has_many  :gifts, :foreign_key => "to_person_id"
end

The key (and the ‘ah ha’ moment for me) was the use of the foreign_key parameter to the on the has_many and belongs_to associations.

In the Gift class, I included the belongs_to association macro. In this case, :from_person is the name of the rich accessor method (which looks like a property in other languages) that will be added to the Gift class. Using the symbol :class_name is like a class finding assistant. Without it, the Rails framework assumes that there would be a class named “FromPerson.” Of course, that would fail. By specifying “Person,” I’ve indicated to Rails that the class it should map to is called “Person” which I defined earlier. The :foreign_key symbol and value indicates which column in the backing table has the value which will map to an instance of a Person. In the SQL table, I added a “from_person_id” column and this points at that as the “from_person_id” column is the foreign key to the People table. (The same is true for :to_person.)

Looking at the Person class, it is using another common association macro, :has_many. :Has_many when used here, is indicating that a Person may have zero or more “gifts.” The new accessor method is named gifts (by using :gifts). Here, too, you’ll specify the name of the foreign_key. Again, repeat this for the :gifts_given automatically added accessor method. One interesting thing is that only :gifts_given requires the :class_name to be specified. The reason is that Rails automatically maps :gifts to the Gifts class (by way of naming). The :gifts_given cannot be automatically mapped, so you need to give (sigh) it a little help.

Here’s a little test:

>> jason = Person.find(1)
  Person Load (18.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = ? LIMIT 1  [["id", 1]]
#<Person id: 1, first_name: "Jason", last_name: "Bourne", date_of_birth: nil, email_address: nil, created_at: "2012-01-06 13:47:40", updated_at: "2012-01-07 03:10:29">
>> jason.gifts_given
  Gift Load (0.0ms)  SELECT "gifts".* FROM "gifts" WHERE "gifts"."from_person_id" = 1
[#<Gift id: 1, description: "Machine Gun", from_person_id: 1, to_person_id: 4, created_at: "2012-01-07 14:39:17", updated_at: "2012-01-07 14:39:17">]
>> jason.gifts_given[0].to_person.first_name
"Magnum"
  Person Load (0.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = 4 LIMIT 1
>> jason.gifts_given[0].to_person.gifts
  [#<Gift id: Gift Load (1.0ms)1  SELECT "gifts".* FROM "gifts" WHERE "gifts"."to_person_id" = 4
, description: "Machine Gun", from_person_id: 1, to_person_id: 4, created_at: "2012-01-07 14:39:17", updated_at: "2012-01-07 14:39:17">]
>> jason.gifts_given[0].to_person.gifts[0].from_person.first_name
  Person Load (0.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = 1 LIMIT 1
"Jason"

I added two people: Jason, and Magnum, and one gift before executing the code above. Jason, as you should be able to follow, gave a wonderful gift to Magnum. As you can see, by using the automatically added accessor methods by way of the association macros described above, I was able to traverse the database structure very easily when mapped to a few simple objects.

One plus of experimenting and testing with the console while using Rails/Ruby in this case is that the output includes the SQL commands that are executed when the various method calls are made. Here’s an example where I rolled multiple calls into one chained call: 

>> jason = Person.find(1).gifts_given[0].to_person.first_name
  Person Load (19.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = ? LIMIT 1  [["id", 1]]
  Gift Load (0.0ms)  SELECT "gifts".* FROM "gifts" WHERE "gifts"."from_person_id" = 1
"Magnum"
  Person Load (0.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = 4 LIMIT 1
by Aaron

A post not related to Nest thermostat hardware….

I typed in https://www.nest.com this evening and instead of the nice looking Nest.com web site, I got this:

SNAGHTML69c9dbf8

What?! I hadn’t actually noticed that I’d typed https at first, so I was a bit baffled for a moment. After closer investigation, Nest doesn’t have HTTPS apparently for their marketing/support web site setup on their Amazon CloudFront account (which is what cloudfront.net is).

They really should fix that. Smile

I was having some serious performance issues with their remote access web site earlier. It was barely usable.

That inspired me to dig a little to see what JavaScript libraries they might be using, the chatter on the network, etc. Here’s what I dug up.

JavaScript libraries:

  1. Jquery 1.7
  2. Raphael (for doing vector graphics using SVG or VML)
  3. Jquery UI
  4. eve.js (a “javascript events library”)
  5. jquery.ui.spinner
  6. iphone style checkboxes (or maybe a variation on this…it’s very similar)
  7. backbone.js
  8. underscore.js
  9. modernizer (1.6)
  10. jquery-animate-css-rotate
  11. jquery double tap plugin
  12. jquery form plugin
  13. jquery right click plugin
  14. jquery mouse wheel support

An interesting DIV and solution to a problem with web fonts (found on their main page): 

<div id="digits">
  <!--  Some browsers suffer unfortunate lag the first time they
        draw text in fonts loaded over the network. We pre-render
        these digits to avoid this lag, and this gets removed at
        the same time the loading message is removed. -->
  1234567890
</div>

And a method for activating DEBUG mode:

N.DEBUG = ('False' == 'True') || (document.location.search.indexOf('debug') > -1) || (C.USERNAME.indexOf('@nestlabs.') > -1);

Thus, to activate DEBUG mode on your nest, simply navigate to here:

https://home.nest.com/home?debug

Brings up a slightly scary option for debugging. Be careful with this – I didn’t try ANY of the options, so it’s at your own risk.

image

Also, interestingly it adds a WEATHER tab:

image

by Aaron

EzData for Ember.js

Previous post: Entity reference

I’ve been extending my original post (above) in an effort to create a simple entity system using Ember.js (SproutCore 2.0). While it’s grown in sophistication, it is far from complete. Smile

The project is now located on GitHub.

This is the introduction, from the readme on GitHub.

EmberJS – EZData

EZData is a library intended to a simple way of accessing data from a relational database when using Ember.js. It will never try to be everything to everyone. :) Instead, it’s intended to be small, efficient, and easy to learn and use javascript library.

The basic functionality is that it’s designed to manipulate individual entities (or Records), not a complex object structure that is often stored as JavaScript objects. This mirrors what’s found in many web systems today in the data model.

Right now, the project is brand new and quite in flux.

Essentials

Create types by calling Entity.define (instead of the typical Ember.Object.extend). 

DemoApp.Person = Entity.define(Entity, "Person", {
    firstName:String,
    lastName:String,
    DOB:Date,

    fullName:function () {
        return this.get('lastName') + ", " + this.get('firstName');
    }.property('firstName', 'lastName')
});

This will create a new Class internally by using Ember.extend but it also does some other magic (e.g., creating a data store for all entities of a given type). First and foremost, by calling define, it’s expected that you’re mirroring a relational data structure of some sort which may have foreign key relationships to other tables.

In the example above, the Person entity is expected to have the following columns in a database table:

  • id => Number (from the basic Entity type)
  • firstName => String
  • lastName => String
  • DOB => DOB

Each property is assigned the basic data type that it will contain. The data type is used for linking and serialization.

Linking

The second example is a Gift:

DemoApp.Gift = Entity.define(Entity, "Gift", {
    from:DemoApp.Person
});

The Gift class is more interesting as one of the properties links to another type/Class (Person). As a direct connection to a Person as an object instance isn’t possible in traditional relational tables, some automatic linking occurs by declaring this linkage.

The from property is mapped to a second, hidden property that is only used for the foreign key relationship. By default, the hidden property will be calledfromPersonId. This can be overridden by creating a custom naming function, replacing the default stored atEntity.Settings.FOREIGN_KEY_GENERATOR_FUNCTION.

By making the linkage between the two types in the way demonstrated above, the standard Ember.js handlebar templating engine (and automatic value updating) features just work (see from.fullName below). 

{{#each gifts}}
<tr>
    <td>
        {{ name }}
    </td>
    <td>
        {{ excitement }}
    </td>
    <td>
        {{ from.fullName }}
    </td>
</tr>

The fullName property of Person is a computed property, and continues to work as expected.

Getting the Data

If you want to retrieve a list from one of the data stores (or data tables), it’s a simple as calling the live function for one of the stores.

In the example below, there are two live collections. The first returns all of the gifts, and the second only returns gifts that have a lowercase letter o in the name. 

// create some "live" connections to the data store
DemoApp.mainController.set('gifts', Entity.Stores.get(DemoApp.Gift).live());
// create a live connection with a filter (returns true if it contains the letter 'o')
DemoApp.mainController.set('giftsFiltered', Entity.Stores.get(DemoApp.Gift).live(
    function () {
        return this.get('name').indexOf('o') > -1;
    }));

To get access to one of the automatically created data stores, call Entity.Stores.get({Class}) where {Class} is the type that was created using thedefine method.

Notes / Cautions

Right now, all entities are required to have a property called id and be of type Number.

by Aaron

Creating a simple Entity Reference system for Ember.js

I had some data stored in a structure similar to this:

image

Nothing too fancy. A table of Gift(s) and a table of Person(s). Each gift had a foreign key relationship to a Person (the person who “gave” the gift).

Since some people may give several gifts, I didn’t want to load the same data multiple times (each gift shouldn’t have the person’s name for example spelled out). I wanted to build a RESTful web service to return the gifts and persons as independent queries, with the resulting JavaScript objects being very similar to the original data storage in the database.

Using Ember.JS I built a small demonstration of one way that this could be done. I am aware of the in-progress data library for ember.js on github, but it was way more than I wanted (or needed). Furthermore, I found it more far complex than I wanted (and I’m not particularly fond of the syntax it requires). So, I created a simple system that I’ll likely expand upon over time and post here.

Here’s the very basic demo using handlebars templates (a core feature of Ember.js). 

<!DOCTYPE html>

<html>
<head>
    <title>Demo2-EmberJS</title>
</head>
<body>

    <script type="text/x-handlebars" data-template-name="gifts">
      <h1>{{ giftReason }}</h1>
      <table>
          <thead>
          <tr>
              <td>Description</td>
              <td>Thrill</td>
              <td>From</td>
          </tr>
          </thead>
          {{#each gifts}}
          <tr>
              <td>
                  {{ name }} 
              </td>
              <td>
                  {{ excitement }}
              </td>
              <td>
                  {{ person.fullName }}
              </td>
          </tr>
          {{/each}}
      </table>
    </script>

    <script src="demo2/jquery-1.7.1.js" type="text/javascript"></script>
    <script src="demo2/ember-0.9.3.js" type="text/javascript"></script>
    <script src="demo2/app.js" type="text/javascript"></script>
</body>
</html>

And here is the content of app.js file referenced above (the other files can be downloaded from the web): 

var Demo2App;
Demo2App = Ember.Application.create();
window.Demo2App = Demo2App;

// standard class that has an 'id' property
var Entity = Ember.Object.extend({
    id:null
});

(function () {
    Ember.entityRef = function (property, type) {
        // auto build connection if property is in the form
        // of 'typenameID' if type is not specified
        if (arguments.length === 1 && property) {
            var l = property.length;
            if (l > 2 && property.substr(l - 2).toLowerCase() === 'id') {
                type = property.substr(0, l - 2);
            } else {
                throw new Error("Type not specified, and cannot automatically determine store name. Referenced property name must end with Id.");
            }
        }
        var fn = new Function("return Demo2App.Stores.find('" + type + "', this.get('" + property + "')); ");
        return Ember.computed(fn).property(property);
    };
})();

/*
 Class definitions
 */

Demo2App.Person = Entity.extend({
    firstName:'',
    lastName:'',

    fullName:function () {
        return this.get('lastName') + ", " + this.get('firstName');
    }.property('firstName', 'lastName')
});


Demo2App.Gift = Ember.Object.extend({
    personId:null,
    person:Ember.entityRef('personId')
});


// build in a data store API
(function () {
    // within a closure, we'll store the current stores

    var stores = {};

    var Store = Ember.Object.extend({
        name:null,
        _data:null,
        add:function (obj) {
            if (!obj) {
                return;
            }
            if (!obj.id) {
                return;
            }
            this.get('_data')[obj.id] = obj;
        },
        remove:function (obj) {
            if (!obj) {
                return;
            }
            if (!obj.id) {
                return;
            }
            var data = this.get('_data');
            delete data[obj.id];
        },

        find:function (id) {
            if (!id) {
                return;
            }
            return this.get('_data')[id];
        }
    });

    Demo2App.Stores = Demo2App.Stores || {};

    Demo2App.Stores.create = function (name, options) {
        name = name.toLowerCase();
        var s = Store.create({
            name:name
        });
        s.set('_data', {});
        stores[name] = s;
        return s;
    };

    Demo2App.Stores.get = function (name) {
        name = name.toLowerCase();
        return stores[name];
    };

    Demo2App.Stores.remove = function (name) {
        name = name.toLowerCase();
        delete stores[name];
    };

    Demo2App.Stores.clear = function (name) {
        name = name.toLowerCase();
        stores[name].set('_data', {});
    };

    Demo2App.Stores.find = function (name, id) {
        var ds = Demo2App.Stores.get(name);
        var entity = ds.find(id);
        if (entity) {
            return entity;
        }
        return null;
    }

})();


var personsDS = Demo2App.Stores.create('person');
personsDS.add(Demo2App.Person.create({
    id:"123",
    firstName:"Aaron",
    lastName:"Bourne"
})
);

personsDS.add(Demo2App.Person.create({
    id:"234",
    firstName:"Bonnie",
    lastName:"Highways"
})
);

personsDS.add(Demo2App.Person.create({
    id:"345",
    firstName:"Daddy",
    lastName:"Peacebucks"
})
);

personsDS.add(Demo2App.Person.create({
    id:"456",
    firstName:"Cotton",
    lastName:"Kandi"
})
);


Demo2App.mainController = Ember.Object.create({

    gifts:Ember.ArrayController.create({
        content:[],

        newGift:function (details) {
            var gift = Demo2App.Gift.create(details);
            this.addObject(gift);
            return gift;
        }
    })
});

var giftsView = Ember.View.create({
    templateName:'gifts',
    giftsBinding:'Demo2App.mainController.gifts',
    giftReason:'Birthday 2011'
});

var moreGifts = [
    { name:'Book', excitement:'3', personId:'123' },
    { name:'Shirt', excitement:'1', personId:'234'},
    { name:'Game System', excitement:'5', personId:'123'},
    { name:'Movie', excitement:'4', personId:'345'},
    { name:'Gift Card', excitement:'3', personId:'123'},
    { name:'MP3 Player', excitement:'3', personId:'456'},
    { name:'Tie', excitement:'1', personId:'456'},
    { name:'Candy', excitement:'3', personId:'234'},
    { name:'Coffee', excitement:'3', personId:'123'},
    { name:'Blanket', excitement:'2', personId:'456'},
    { name:'Camera', excitement:'4', personId:'234'},
    { name:'Phone', excitement:'5', personId:'234'},
    { name:'Socks', excitement:'1', personId:'123'},
    { name:'Game', excitement:'5', personId:'456'}
];

var moreGiftsIndex = moreGifts.length;

$(function () {

    function addMoreGifts() {
        moreGiftsIndex--;
        if (moreGiftsIndex >= 0) {
            Demo2App.mainController.gifts.newGift(moreGifts[moreGiftsIndex]);
        }
        setTimeout(addMoreGifts, 2000);
    }

    giftsView.append();
    addMoreGifts();
});

Here’s a few details about the JavaScript code. #1, it’s got very little in the way of error checking. I’ll add that later. And if you use it, you should add it. :)

Recall the simple data model with each gift having a foreign key reference to a person. If you look at each Gift, it contains keys that mirror the DB: name, excitement, and personId. All 3 are handled as strings. The Person class has an Id, lastName, and firstName.

When creating the Ember Gift class, I made a connection to the Person class using a new function I added called entityRef. 

Demo2App.Gift = Ember.Object.extend({
    personId:null,
    person:Ember.entityRef('personId') // [3] 
});

As you can see on line [3] above, a person property is declared not with a value, but as a function. Ember.js includes support for computed properties which, when properly used, return a computed value and can be automatically updated when a dependency is established between the function and the values the computed property requires. Following is the code for the entityRef function. 

(function () {
    Ember.entityRef = function (property, type) {
        // auto build connection if property is in the form
        // of 'typenameID' if type is not specified
        if (arguments.length === 1 && property) {   // [1]
            var l = property.length;
            if (l > 2 && property.substr(l - 2).toLowerCase() === 'id') {
                type = property.substr(0, l - 2);
            } else {
                throw new Error("Type not specified, and cannot automatically determine store name. Referenced property name must end with Id.");
            }
        }
        var fn = new Function("return Demo2App.Stores.find('" + type + "', this.get('" + property + "')); ");
        return Ember.computed(fn).property(property);
    };
})();

Thankfully, the entityRef function doesn’t need to do much, and much of what it does is make it easy to make a connection automatically between the local data store and the source property by using a simple naming convention.

Starting at [1], the code makes a simple attempt at doing an “auto-map” when a type isn’t specified specifically. In this case, when “personId” is passed to the function, it chops the “Id” off and uses “person” as the name of the data store automatically. It doesn’t validate the name or anything sophisticated (as I said, it’s light on error checking), but it works when everything is specified correctly. In this case, there’s a “person” data store created in the JavaScript code created a little bit later. 

var personsDS = Demo2App.Stores.create('person');

Once the data store type name is located, it creates a new function which returns the object by Id stored in the named data store. That function is wrapped in an Ember computed function, and then the dependency on the “personId” is established. Both of these features are core to the Ember JavaScript library (check out the Ember JavaScript library web site for more information). While they may look a bit strange if you’re not familiar with Ember, trust me, they’re perfectly normal. :)

The remaining code is standard Ember JavaScript code with the addition of a simple data store object that manages objects by Id.

Have fun!

(And if you haven’t transitioned from the earlier SproutCore 2.0 betas to Ember, this code should work with a tiny bit of namespace fixing).