Flexible Queries Against Azure DataMarket in Windows Phone 7
The Azure DataMarket is a stellar resource for application data. At the current moment, it features over 100 datasets (from baseball to census to mortgage) in OData format, much of which is free, and the recently announced Self-Service Publishing Wizard should help to manifest even more hidden data into a common format. Mobile devices, being disconnected from pretty much everything, are ideal consumers of cloud-based data. In this post, we'll look at one way of querying data from the Azure DataMarket in a Windows Phone 7 application. Before we get rolling, a quick note about the free datasets. They are reg-ware, which means a Windows Live ID is required, and you'll need to generate an application key to use when making requests. I show you how to find your key below. Finding and Registering For A Dataset Time to get started--let's find a free dataset we can play with. Go to http://datamarket.azure.com/, click the Data button on the top menu, then on the left menu, click Free under the Price category. We're now browsing the list of all free datasets--56 of them at this time. If we use the search feature, the Data and Free filters will be honored. All the good BI demos involve despair of some kind, so let's see if there is crime data. Searching for crime brings us back two datasets--London Borough Profiles, and 2006-2008 Crime in the United States. Bingo on the second one! Clicking on the set's title takes us to a page with more information about the dataset. On the right is a large green "Sign Up" button, with subscription details right above it. This dataset is free for unlimited transactions per month. Fun! 
When you click the Sign Up button, you'll be asked to either log in with an existing Windows Live ID, or create a new one. Do what you need to here--remember that if you're signing up for a service that may incur charges, this Live ID will be used for the billing information. After you've logged in (and created a Live ID, if necessary), you'll be presented with another registration screen. This one creates your Azure DataMarket account, and and only needs to be completed the first time. You won't see this when you sign up for additional datasets. 
Next up is the usual gigantic page of legalese. These are the DataMarket Terms and Conditions, which you'll have to agree to if you want to consume data. For our sample apps, we're unlikely to run afoul of these, but if you'e an ISV, you might want to read these over. This is also a first-time-only requirement, until they change. 
Almost there! Now that we've signed up for DataMarket, we need to agree to the T&C of our dataset. Again, for our sample app using a free dataset, we're not likely to run afoul of these, but pay attention if there is money involved, or if the dataset is possibly copyrighted (e.g., non-government, like baseball stats or Zillow). You will see one of these forms for every dataset you sign up for. 
Finally, we're done! We can now start consuming our dataset. The Thank You page has links to some useful resources for us, notably the DataSet explorer, and using the dataset in Visual Studio. You can access a similar page from Account >> My Data, too. 
The dataset explorer is a really cool tool--here we can interactively build queries, chart and export our data. For some purposes, this may be all the tool you need. When building a Windows Phone Application, it's a great tool to debug your queries. Note that any query you make in the explorer counts against your allotment, if there is one. 
Groovy. At this point, it's time for a little architectural discussion regarding how we consume Azure DataMarket data. System.Data.Services.Client and DataMarket Queries DataMarket data is provided via the Open Data Protocol (OData). It's an incredibly rich system for querying online datasets, and data can be returned in both Atom and JSON. The workings of OData are beyond the scope of what we'll be doing here, but check out the Developer section on their site for more information. Data can be queried from the DataMarket using REST, but Windows Phone 7 provides us a library which wraps DataMarket calls into a more familiar method using Linq or traditional service operation. This library is System.Data.Services.Client.aspx), and is pretty impressive. Return types are DataServiceCollection
There is another piece of information given to us on the details tab of our dataset: 
Both of these tell us we can use a "flexible query" against this dataset--that is, we can write Linq queries in C# or VB.NET to access the data. This is a very fast, powerful and comfortable way to extract data from a cloud dataset. Contrast that with the alteryx Census Demographic Data. The DataMarket provides us with a C# class to include in our projects: 
And the Details tab tells us it's fixed query only: 
This post will focus on the flexible query, but I'll have a fixed query example shortly (follow me on Twitter at http://twitter.com/rj_dudley or watch this blog to see when that's done). Now that we know which query type we'll be using, let's take a closer look at our dataset. This is found on the Details tab of the dataset page. In a flexible query, the bold headings above the table are the entity names. Input parameters are what we can include in the WHERE clause of our query, and the results are what we get back. 
In a fixed query, the bold headings are the service operations we can call. More on that in the next post. Service URL and Application Key There are two more things we need from the DataMarket. One is the service root URL. It's on the details tab of our dataset's page, and you can see it in the screenshots above. You'll just copy and paste this when we need it in a moment. The other thing we need is an application key. This key is unique to every DataMarket customer, and if you want, you can generate a number of keys to make them unique to each application. You find application your keys under My Account >> Application Keys. You'll also copy and paste this in a few moments: 
Building our Application The time has arrived for the real fun! Open Visual Studio and create a new project in the language of your choice. Under Silverlight for Windows Phone, choose the Windows Phone Application template, and name it what you want. You should be prompted to choose the target phone OS; if you're not, your SDK is out of date and you need to get the latest Mango release from AppHub. First thing we should do is add a reference to the service. If you haven't done this before, open your Solution Explorer, select your project, then click the Show All Files icon: 
This exposes the References node to us. Right-click on References, and choose Add Service Reference: 
Paste the service root URL into the Address box and click Go. Visual Studio will download the metadata for the service and create the proxy classes we need. Before you click OK, make sure to change the Namespace to something useful: 
If all went well, it's now time for some small layout. We're going to use a simple listbox to display the year and number of violent crimes for a particular city. Dress this up as you see fit later; for now, find the ContentPanel grid in the XAML, and paste the following code inside the grid tags (this is also available in the sample project):
<ListBox.ItemTemplate> </ListBox.ItemTemplate>
Now for a little code behind. This is also available in the sample project. At the top of the code behind, add the following import/using statements:
Imports System.Linq
Imports System.Net
Imports Microsoft.Phone.Controls
Imports System.Data.Services
Imports
.CityCrimeService
For the sake of simplicity, we're not going to implement MVVM here; we'll simply execute our code when the page loads, and container the data on this page. It's easy to change this later if you want.
Inside the class, but above the constructor, add the following declarations:
Private _context As datagovCrimesContainer
Private ReadOnly _crimesUri = New Uri("https://api.datamarket.azure.com/data.gov/Crimes/") Private _crimes As DataServiceCollection(Of CityCrime) Private _userId As String = "fake email address" Private _key As String = "real account key here"
Where it says "fake email address", put any fake email address. At the current time, the DataMarket authentication ignores this field. However, to make a proper credential, you need to enter any gibberish text which looks like an email address. Where it says "real account key here", put your correct account key. This is required by DataMarket authentication. Inside the page constructor, add the following code. You can see where to customize your query, you can even add parameters based on GPS or text inputs if you want. For asynchronous requests (which is the only wayWP7 communicates with services), we need to tell the application what to do once the data is returned, so we use AddHandler to point to another method which does something with the data. The LoadAsync method then executes our query.
_context = New datagovCrimesContainer(_crimesUri) _crimes = New DataServiceCollection(Of CityCrime)(_context) Dim _query = From c In _context.CityCrime Where c.State = "Pennsylvania" And c.City = "Pittsburgh" Select c Order By c.Year _context.Credentials = New NetworkCredential(_userId, _key) AddHandler _crimes.LoadCompleted, AddressOf crimes_loadcompleted _crimes.LoadAsync(_query)
Finally, add the handler method to our class. All we're doing here is binding our returned data to the listbox. If we had any additional post-query processing, this is where we'd do it.
Private Sub crimes_loadcompleted() Me.LayoutRoot.DataContext = _crimes End Sub
At this point, you should be able to build and run your application! From here, practice changing your Linq query to filter or shape the data some more. You can grab the sample code below if you need to, and be sure to read the notes below. Download VB.NET Sample Code: Flexible Query VB Download C# Sample Code: Flexible Query CS A Note About MVVM
By executing the query and containering the data on this page, all the controls on this page will have access to the date, but other pages we add will not. We use MVVM to serve as an application-wide bucket of data (if you'd like a little more information about MVVM, check out Greg Lutz's video at http://vimeo.com/20347067). You can also read Implementing the Model-View-ViewModel Pattern in a Windows Phone Application.aspx) on MSDN. Using Your Key, Or Not? In this sample, we used our own API key in our application. Were we to distribute this application, everyone who uses it would count against our allotment of API calls. For the service we chose, not a big deal, but this will end up costing you a lot of money on a pay service. You can require your users to create their own DataMarket account, with their own billing information, and then enter their credentials into your application. OAuth is probably a better choice than our API key in this sample (see below). I also suggest adding a new key to your account, and using that in your application. If you have more than one application, use more than one key. This helps you keep your keys a little more private, and track usage of particular apps against your account should you need to. Additional References We've used HTTP Basic Authentication in the sample; OAuth 2.0 is also available. A discussion on which one to choose and how to implement can be found in the Marketplace Authentication on MSDN. How to: Execute Asynchronous Data Service Queries (WCF Data Services) http://msdn.microsoft.com/en-us/library/dd756367.aspx Open Data Protocol (OData) Client for Windows Phone http://msdn.microsoft.com/en-us/library/gg521146(v=VS.92).aspx.aspx) WCF Data Services Learning Center http://msdn.microsoft.com/en-us/data/bb931106 Consuming a Windows Azure Data Service by using the OData Client http://create.msdn.com/en-US/education/quickstarts/odataservice Studio for Windows Phone 7 Want to add some awesome charts, gauges and other controls to your application? Check out ComponentOne Studio for Windows Phone 7.
MESCIUS inc.