Month / January 2012

by Aaron

.NET API for Nest Thermostat

I just finished a preliminary read-only (think version 0.1) wrapper around the Nest Thermostat API that is used by their mobile phone and web applications. As Nest doesn’t have a formal API yet, the code could break at any time and may not be suitable for any use. However, it is working today. Smile

The project is hosted on GitHub. It uses JSON.NET for parsing the return values from the Nest servers.

There are three projects, with the lib containing the assembly that is used by the two test applications. One is a console app and the other a simple WPF application:

SNAGHTML88bff0b3

(My thermostats are named Zero, One, and Two).

If there’s interest, I plan on adding some methods to the library which allow modification of data (such as the current temperature), and ideally, support for live updates from the devices if I can make sense of the data that is returned.

FYI: I’ve now written a Node version of the API, detailed here.

by Aaron

Nest Thermostat API/Protocol

While Nest Labs hasn’t released a formal (documented & supported) API, I thought I’d do a bit of digging to see how they’re using the network and what might be achievable.

A few things are going on, the majority as you’d probably expect.

The web interface is using a long polling technique apparently to watch for updates to the schedule, temperature, etc.

image

I haven’t determined what the frequency is though, or the wait time. It’s very inconsistent, even when I wouldn’t expect much new “live” data to be available on the network, it frequently updates and polls again.

There are a few constants set in the HOME page script:

C.ABSENT_USER_THRESHOLD     = +('300') || 0;  // seconds
C.DEAD_DEVICE_THRESHOLD     = +('300') || 0;  // seconds
C.pollingInterval           = +('2500') || 0;       // ms
C.WEATHER_POLLING_INTERVAL  = +('120000') || 0; // ms

 

If the C.pollingInterval value were for the subscribe endpoint mentioned above, I’d see a LOT more calls than I do – so I’m still not clear how the polling interval is decided.

The API calls, for the most part are using JSONP syntax over an HTTPS connection.

The most frequent request is to “subscribe.” It sends as part of the GET request a large block of encoded JSON (using encodeURIComponent and then JSON.stringify).

I’m not familiar with the key/value system that they’re using (it may just be something they’ve constructed in-house – although given the number of open source JavaScript libraries they’re using, I thought someone might recognize it):

“key”, “{actualkey}.{value}”

I don’t understand why they’ve redundantly specified “key” in a list of keys when it’s evident that the actual key is contained within the value as a delimited string. It’s more data to send and more data to parse this way. So, again, maybe it’s based on some DB or model system I’m not familiar with. (Anyone recognize it?)

I’ve substituted the actual values (as they are serial numbers of my devices) with text representations of what the value represented below: 

{"keys":
    [{"key":"user.#USERID#",
        "version":209478897,"timestamp":1324159145000},
    {"key":"user_alert_dialog.#USERID#",
        "version":-1320296685,"timestamp":1325967612000},
    {"key":"structure.#STRUCTURE-GUID#",
        "version":656192675,"timestamp":1325967612000},
    {"key":"device.#DEVICE 1 SERIAL NUMBER#",
        "version":1485027516,"timestamp":1326034984000},
    {"key":"shared.#DEVICE 1 SERIAL NUMBER#",
        "version":588844038,"timestamp":1326034818000},
    {"key":"schedule.#DEVICE 1 SERIAL NUMBER#",
        "version":1187107985,"timestamp":1326005677000},
    {"key":"track.#DEVICE 1 SERIAL NUMBER#",
        "timestamp":1326035650601,"version":1041047847},
    {"key":"device.#DEVICE 2 SERIAL NUMBER#",
        "version":149169270,"timestamp":1326034820000},
    {"key":"shared.#DEVICE 2 SERIAL NUMBER#",
        "version":659841570,"timestamp":1326034820000},
    {"key":"schedule.#DEVICE 2 SERIAL NUMBER#",
        "version":-2016290692,"timestamp":1326005625000},
    {"key":"track.#DEVICE 2 SERIAL NUMBER#",
        "timestamp":1326035650862,"version":528978433},
    {"key":"device.#DEVICE 3 SERIAL NUMBER#",
        "version":1637112547,"timestamp":1326035399000},
    {"key":"shared.#DEVICE 3 SERIAL NUMBER#",
        "version":760504326,"timestamp":1326035397000},
    {"key":"schedule.#DEVICE 3 SERIAL NUMBER#",
        "version":-314552357,"timestamp":1326003402000},
    {"key":"track.#DEVICE 3 SERIAL NUMBER#",
        "version":-645931164,"timestamp":1326035531802}]}"

We’ve got three thermostats, so there are always three sets of subscription requests for each call to subscribe.

Using my iPad, I adjusted the set point for our second story (#DEVICE 2#) down one degree Fahrenheit (to 67°).

Within approximately a second, the most recent pending subscribe request returned with a far more interesting payload: 

jQuery17108417355176061392_1326035646750(
    { "status": 200,
        "headers": {
            "X-nl-skv-key": "shared.#DEVICE 2 SERIAL NUMBER#",
            "X-nl-skv-version": 869022424,
            "X-nl-skv-timestamp": 1326038279000,
            "X-nl-service-timestamp": 1326038279825
        },
        "payload": {
            "current_temperature": 19.98,
            "hvac_fan_state": false,
            "name": "TWO", "hvac_heat_x2_state": false,
            "hvac_ac_state": false,
            "can_cool": true,
            "auto_away": 0,
            "compressor_lockout_enabled": false,
            "target_temperature_low": 16.66667,
            "target_temperature_high": 26.66667,
            "compressor_lockout_timeout": 0,
            "hvac_heater_state": false,
            "hvac_aux_heater_state": false,
            "target_temperature": 19.44444,
            "can_heat": true,
            "target_temperature_type": "heat",
            "target_change_pending": true
        }
    });

Everything above is needed to update the current state of the UI. As you can see, the current temperature (returned as Celsius apparently) is 19.98 (67.964°F). The current temperature as displayed on the thermostat and the web UI was 68.

Seeing these return values makes me think that they may be using Ruby and Rails (as the naming convention tends to follow Rails naming using underscores between words). I know for example, I wouldn’t name variables/columns that way when building a C#/JavaScript MVC project.

Rather than just a delta payload of what’s changed, they’ve currently opted for a full update of all information related to the thermostat state.

Several seconds later, a much larger payload was returned to a subscribe request: 

"status": 200,
"headers": {
    "X-nl-skv-key": "device.#DEVICE 2 SERIAL NUMBER#",
    "X-nl-skv-version": -2086438581,
    "X-nl-skv-timestamp": 1326038378000,
    "X-nl-service-timestamp": 1326038379023
},
"payload": {
    "ob_orientation": "O",
    "upper_safety_temp": 1000.0,
    "forced_air": true,
    "creation_time": 1324142042019,
    "switch_preconditioning_control": false,
    "click_sound": "on",
    "leaf": false, "user_brightness": "auto",
    "learning_state": "steady",
    "heat_pump_comp_threshold": -1000.0,
    "local_ip": "10.0.0.205",
    "backplate_serial_number": "#SHOULD BE DEVICE 2 SERIAL NUMBER, BUT ISN'T?#",
    "capability_level": 1.03,
    "postal_code": "#POSTALCODE#",
    "upper_safety_temp_enabled": false,
    "heat_pump_aux_threshold": 10.0,
    "lower_safety_temp_enabled": true,
    "serial_number": "#DEVICE 2 SERIAL NUMBER#",
    "temperature_lock": false,
    "learning_time": 1002,
    "current_version": "1.0.4",
    "model_version": "Diamond-1.10",
    "backplate_bsl_info": "BSL",
    "auto_away_enable": true,
    "heat_pump_comp_threshold_enabled": false,
    "fan_mode": "auto",
    "range_enable": false,
    "temperature_scale": "F",
    "backplate_mono_info": "TFE (BP_DVT) 3.5.2 (ehs@ubuntu) 2011-11-05 12:00:00",
    "backplate_bsl_version": "1.1",
    "equipment_type": "gas",
    "range_mode": false,
    "lower_safety_temp": 7.0,
    "has_fan": true,
    "hvac_wires": "Heat,Cool,Fan,Common Wire,Rc",
    "learning_mode": true,
    "away_temperature_high": 32.0,
    "switch_system_off": false,
    "time_to_target": 1326039444,
    "away_temperature_low": 14.444444444444445,
    "current_humidity": 45,
    "mac_address": "#MACADDR#",
    "backplate_mono_version": "3.5.2",
    "has_aux_heat": false,
    "type": "TBD",
    "hvac_pins": "W1,Y1,C,Rc,G",
    "has_heat_pump": false,
    "heat_pump_aux_threshold_enabled": true,
    "battery_level": 3.945,
    "target_time_confidence": 1.0
}

 

A few things to note:

  • Upper_safety_temperature is just a bit beyond my comfort zone at 1832°F. I don’t know why it’s sending a value like that to the client, and why it’s stupidly high.
  • The backplate serial number doesn’t match with the thermostat according to the payload response. I don’t know why this might be as I confirmed that the numbers matched through visual inspection of the device just now.
  • The majority of these details are exposed in one way or another in the details area of the web UI.
  • Time to target (payload.time_to_target) is unusual in that it’s a JavaScript Date value, divided by 1000. So, in the example above, the time to target is: new Date(1326039444 * 1000).toString() = >”Sun Jan 08 2012 10:17:24 GMT-0600 (Central Standard Time)”Next, a payload is returned with the new status: 
    "status": 200,
"headers": {
    "X-nl-skv-key": "shared.#DEVICE 2 SERIAL NUMBER#",
    "X-nl-skv-version": 1689916148,
    "X-nl-skv-timestamp": 1326038378000,
    "X-nl-service-timestamp": 1326038379151
},
"payload": {
    "hvac_fan_state": false,
    "name": "TWO",
    "hvac_heat_x2_state": false,
    "hvac_ac_state": false,
    "can_cool": true,
    "auto_away": 0,
    "compressor_lockout_enabled": false,
    "target_temperature_low": 16.66667,
    "current_temperature": 19.53,
    "target_temperature_high": 26.66667,
    "compressor_lockout_timeout": 0,
    "target_change_pending": false,
    "hvac_aux_heater_state": false,
    "target_temperature": 20.55556,
    "can_heat": true,
    "target_temperature_type": "heat",
    "hvac_heater_state": true
}

     

    Here, the hvac_heater_state is set to true. The furnace is on.

    A little while later, that value is set to false.

    Occasionally, the payload includes the complete schedule for the thermostat. I’m not going to reproduce the entire payload here as it’s too large, and quite boring. Here’s a snippet of what it returns: 

    "schedule": {
    "#DEVICE 2 SERIAL NUMBER#": {
        "$version": 1187107985,
        "$timestamp": 1326005677000,
        "name": "One Current Schedule",
        "days": {
            "0": {
                "0": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 0,
                    "entry_type": "continuation"
                },
                "1": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 27900,
                    "entry_type": "setpoint"
                },
                "2": {
                    "type": "HEAT",
                    "temp": 20.556,
                    "time": 63000,
                    "entry_type": "setpoint"
                },
                "3": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 70200,
                    "entry_type": "setpoint"
                }
            },
            "1": {
                "0": {
                    "type": "HEAT",
                    "temp": 14.445,
                    "time": 0,
                    "entry_type": "continuation"
                },
                "1": {
                    "type": "HEAT",
                    "temp": 18.889,
                    "time": 20700,
                    "entry_type": "setpoint"
                },

     

    It’s a basic table structure. The first set point of the day is at 0, and is a “continuation.” These don’t show up in the UI.

    Here’s what the day 1 looks like on the Nest thermostat UI:

    image

    When changing a temperature setpoint, I’m a bit disappointed to see that the entire schedule is sent with every request apparently. I just wouldn’t have expected that given that the more setpoints that there are, the bigger the payload must be. The UI is often sluggish when rapidly making adjustments in the schedule, and this could be one of the factors.

    In the example below (which I’ve snipped most of the payload sent again as a JSONP request), I’ve set the first set point to 57F. 

        "payload": {
        "days": {
            "0": {
                "0": {
                    "type": "HEAT",
                    "temp": 14.685,
                    "time": 0,
                    "entry_type": "continuation"
                },
                "1": {
                    "type": "HEAT",
                    "temp": 15.000444444444444,
                    "time": 24300,
                    "entry_type": "setpoint"
                },

    For the JSONP requests sent as “MAKE CHANGE” (easily could have been PUT), they each contained the following attributes as shown below. All JSONP requests are apparently routed on the web server using “headers” rather than a RESTful URL based system: 

        },
    "headers": {
        "X-nl-client-timestamp": 1326041210566,
        "X-nl-session-id": "#SESSION ID#",
        "X-nl-protocol-version": 1,
        "Authorization": "Basic #BASIC AUTH#"
    },
    "path": "/v1/put/schedule.#DEVICE 2 SERIAL NUMBER#",
    "redir": "https://home.nest.com/post_jsonp",
    "jsonp": "4_"
}

    It’s RESTful in spirit as there is a route (“path”), but it’s managed by some internal routing engine. (Now, I think that they’re not using Ruby and Rails).

    For something simple, like changing the current temperature of a thermostat, the request is thankfully simple: 

    {
    "payload": {
        "shared": {
            "#DEVICE 2 SERIAL NUMBER#": {
                "target_temperature": 18.333333333333336
            }
        }
    },
    "headers": {
        "X-nl-client-timestamp": 1326041744556,
        "X-nl-session-id": "#SESSION ID#",
        "X-nl-protocol-version": 1,
        "Authorization": "Basic #BASIC AUTH#"
    },
    "path": "/v1/put",
    "redir": "https://home.nest.com/post_jsonp",
    "jsonp": "14_"
}

    While, I haven’t taken the time to try to write a custom UI for this undocumented API yet, it looks like it should be relatively easy to do, especially as it relates to the schedule and current temperature settings. I know there’s been some Siri proxy stuff that’s been written – but I don’t have any interest in trying to get that to work.

    As with most APIs like this, the trick is often getting authorization correct. For Nest, it appears that making a POST request to https://home.nest.com/accounts/login/ with username and password as form data, that the server responds with 2 cookies:

    1. sessionid == used in X-nl-session-id in headers
    2. cztoken == used as the Authorization in headers (prepended with the text “Basic “

    FYI: I also have a Node version of the API that is more up to date than this.

by Aaron

Nest Thermostat Review, Update #7

Update #6, Update #5, Update #4, Update #3, Update #2, Update #1, Install

I received a replacement thermostat earlier this week as promised by Nest Labs. I had time this afternoon to do a swap and reinstall. Nest had asked me to swap the thermostats between two floors experimentally to determine whether a temperature reading issue was related to the location or the thermostat. It was the thermostat.

As part of the swap, I had to reprogram the two thermostats.

I’d swapped the defective thermostat with one from the basement. Apparently, the base has the S/N on it of the thermostat and they’re intended to be “paired” so I decided to return the thermostat to the basement and install the new thermostat on the first floor (replacing the original defective unit).

I removed the old unit and replaced the wires. Depending on the type of wires you’re using, you may find that it’s far more difficult to do than you would expect. I ‘d forgotten how much I hated trying to stick the very stiff HVAC wires into the thermostat’s base!

I replaced everything and activated the unit. You can look at the installation experience post for more information about the general setup.

Past the wifi connection, rebooting, waiting, waiting, waiting, then “ERROR.” “No Rc or Rh” connection.

20111218-IMG_0138

Crud. So, I popped the thermostat off the base and looked at the wires. They all appeared to be fine. So, I reseated the Rc connection and replaced the thermostat. Success! I find that a tiny tug on the wire after you believe it has been seated does the trick. The rest of the install went without issue.

The thermostat switched to the normal temperature display after a few more alerts. The only thing was – the temperature read 76F. Whaaaa? Given the reboot cycles, etc., I really hadn’t handled it much, not enough to cause the temperature to be that high. I waited about 5 minutes for it to start dropping and when it did not, I called Nest to speak with the person who’d handled the replacement, Mark. He was out apparently, so I ended up speaking with someone who went by “DK” for about 30 minutes about a few topics.

We decided the best course of action was to wait and see.

Thankfully, the new thermostat is now reading a temperature that I would expect, so for some reason, this new thermostat took quite a while to acclimate to the room temperature (much longer than the original three thermostats – around 45 minutes).

by Aaron

Ember.JS and EZdata, and Rails

I’ve been trying to do some additional work on my ember.js extension for data management. At the same time though, I’ve been trying (to learn and) build a simple Ruby on Rails web demo application using the new JavaScript library. There have been more than a few things that have mystified me about the framework and the structuring of an application. One aspect in particular was how to best manage foreign keys and join tables with the ActiveRecord class (and the corresponding SQL tables). So many tutorials have the same lame example of: a CART, an ORDER, a CUSTOMER …, that it’s often difficult to apply the same patterns to a more interesting system.

I started simple this time.

I wanted a PERSON class and a GIFT class.

image

A Person has been given gifts and may give gifts (and a few other common attributes). 

class CreatePeople < ActiveRecord::Migration
  def change
    create_table :people do |t|
      t.string :first_name
      t.string :last_name
      t.date :date_of_birth
      t.string :email_address

      t.timestamps
    end
  end
end

One of the things that I can’t decide if I like is the automatic pluralization of words, especially People/Person. I would have been content with a Persons table, but when creating a model, by default (as I’m aware it can be overridden), a Person is mapped to a table called “People.”

The second table, Gifts is very simple:

class CreateGifts < ActiveRecord::Migration
  def change
    create_table :gifts do |t|
      t.string :description
      t.integer :from_person_id
      t.integer :to_person_id
      t.timestamps
    end
  end
end

As I thought I might want a richer structure for the Gift class in the future, I did not use the more standard “person_id” name for the foreign key column that maps a gift to a Person. I wanted the column name to be more obvious what it was. Additionally, I needed two columns that both mapped to a “Person", so I couldn’t have both be called “person_id” anyway.

By deviating from the normal pattern, there are a few expectations when defining the ActiveRecord class. It was these expectations that weren’t clear to me (especially with examples).

The Ruby class for Gift is defined like so:

class Gift < ActiveRecord::Base
  belongs_to :from_person, :class_name => "Person", :foreign_key => "from_person_id"
  belongs_to :to_person, :class_name => "Person", :foreign_key => "to_person_id"
end

and the Person:

class Person < ActiveRecord::Base
  has_many  :gifts_given , :class_name => "Gift", :foreign_key => "from_person_id"
  has_many  :gifts, :foreign_key => "to_person_id"
end

The key (and the ‘ah ha’ moment for me) was the use of the foreign_key parameter to the on the has_many and belongs_to associations.

In the Gift class, I included the belongs_to association macro. In this case, :from_person is the name of the rich accessor method (which looks like a property in other languages) that will be added to the Gift class. Using the symbol :class_name is like a class finding assistant. Without it, the Rails framework assumes that there would be a class named “FromPerson.” Of course, that would fail. By specifying “Person,” I’ve indicated to Rails that the class it should map to is called “Person” which I defined earlier. The :foreign_key symbol and value indicates which column in the backing table has the value which will map to an instance of a Person. In the SQL table, I added a “from_person_id” column and this points at that as the “from_person_id” column is the foreign key to the People table. (The same is true for :to_person.)

Looking at the Person class, it is using another common association macro, :has_many. :Has_many when used here, is indicating that a Person may have zero or more “gifts.” The new accessor method is named gifts (by using :gifts). Here, too, you’ll specify the name of the foreign_key. Again, repeat this for the :gifts_given automatically added accessor method. One interesting thing is that only :gifts_given requires the :class_name to be specified. The reason is that Rails automatically maps :gifts to the Gifts class (by way of naming). The :gifts_given cannot be automatically mapped, so you need to give (sigh) it a little help.

Here’s a little test:

>> jason = Person.find(1)
  Person Load (18.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = ? LIMIT 1  [["id", 1]]
#<Person id: 1, first_name: "Jason", last_name: "Bourne", date_of_birth: nil, email_address: nil, created_at: "2012-01-06 13:47:40", updated_at: "2012-01-07 03:10:29">
>> jason.gifts_given
  Gift Load (0.0ms)  SELECT "gifts".* FROM "gifts" WHERE "gifts"."from_person_id" = 1
[#<Gift id: 1, description: "Machine Gun", from_person_id: 1, to_person_id: 4, created_at: "2012-01-07 14:39:17", updated_at: "2012-01-07 14:39:17">]
>> jason.gifts_given[0].to_person.first_name
"Magnum"
  Person Load (0.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = 4 LIMIT 1
>> jason.gifts_given[0].to_person.gifts
  [#<Gift id: Gift Load (1.0ms)1  SELECT "gifts".* FROM "gifts" WHERE "gifts"."to_person_id" = 4
, description: "Machine Gun", from_person_id: 1, to_person_id: 4, created_at: "2012-01-07 14:39:17", updated_at: "2012-01-07 14:39:17">]
>> jason.gifts_given[0].to_person.gifts[0].from_person.first_name
  Person Load (0.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = 1 LIMIT 1
"Jason"

I added two people: Jason, and Magnum, and one gift before executing the code above. Jason, as you should be able to follow, gave a wonderful gift to Magnum. As you can see, by using the automatically added accessor methods by way of the association macros described above, I was able to traverse the database structure very easily when mapped to a few simple objects.

One plus of experimenting and testing with the console while using Rails/Ruby in this case is that the output includes the SQL commands that are executed when the various method calls are made. Here’s an example where I rolled multiple calls into one chained call: 

>> jason = Person.find(1).gifts_given[0].to_person.first_name
  Person Load (19.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = ? LIMIT 1  [["id", 1]]
  Gift Load (0.0ms)  SELECT "gifts".* FROM "gifts" WHERE "gifts"."from_person_id" = 1
"Magnum"
  Person Load (0.0ms)  SELECT "people".* FROM "people" WHERE "people"."id" = 4 LIMIT 1
by Aaron

Nest Thermostat Review, Update #6

Update #5, Update #4, Update #3, Update #2, Update #1, Install

I didn’t expect to have another post so soon. But, the Nest experience continues to frustrate and baffle.

On the 3rd of January, I took the following screen shot of our basement schedule for heating:

image

Tonight (one day later on the 4th), we headed down to the basement to watch a recorded episode of AMCs Hell on Wheels and it was cold. Frak! What the…

The thermostat reported the temperature in the room was 62F. Seriously? It should have been about 66F at the time I looked.

I brought up the schedule and was disappointed (yet, not shocked given the other problems) to see:

image

All of the evening settings (except Monday?) had disappeared completely. I had not made the change. I don’t understand. I’m no artificial intelligence expert (my wife though has a CS Master’s degree with a specialty in it Smile), but I’m confident any learning algorithm I would write wouldn’t be this stupid and this broken.

If I were a competitor reading this, I’d be laughing just a little. Understand though – Nest still can make this work – don’t rest or expect them to just disappear so easily.

As Nest continues to be silent on these matters, I strongly recommend you not buy a Nest thermostat. It’s an undone expensive piece of hardware, that while shiny and new, isn’t ready for the duties it claims to have mastered.

Update: Janurary 13, 2012 – This happened again. The schedule for Saturday and Sunday was modified to entirely remove the evening set points.  Nest support recommended that I turn off the “learning” feature of the thermostat yesterday and I hadn’t done that yet. But now I will and see if it happens regardless.