Azure IoT services are collection of services targeting the internet of things.
They provide event collection, processing, analytics and machine learning
In this series of articles I'm going to explain how to setup simple event
collection and processing pipeline. Target is to enchance it with analytic
and dashboard views as we go forwards.
Planned articles in this series
- Event collection
- Event dispatching
- Simple analytics
- Live dashboards
We're going to build a sample application showing all the capabilities we're
going trough in this series. Sources will be available on GitHub.
First part of collecting the events from a device or device gateway is to have
a endpoint in Azure which will receive the events. For such purpose Azure
provides a service called EventHub. This service will allow us to
receive events in big data scale from hundreds - thousands of event sources.
Diagram 1. Overview
Diagram 1. shows the architecture overview of how event collection could be
setup in most cases not involving tens of thousands of event sources with high
frequence of generated events. In the diagram you can find the event source
"Device" which will send the events to EventHub named "Event Aggregator".
Long-term event storage
In our case we will want to setup a long-term storage of the events for future
processing needs for example using HDInsight (Hadoop as service).
Long-term storage should be cheap and size should scale to fit future events. In
Azure we can leverage the Azure Blob Storage to store the raw events as
blobs of data.
To process the incoming events we could use Azure WebJobs,
Azure Worker Roles or Azure Stream Analytics to store
the events in blob storage. In our case we will use Stream Analytics as the
implementation will be scalable and simple by using it.
Diagram 1. contains a Stream Analytics block named "Event Dispatcher" connected
to both "Event Aggregator" and "Long-term storage". As events are processed
from the EventHub we will write them to blob storage using a Stream
Analytics query. Stream Analytics query language follows closely
syntax of SQL. It provides query extensions to query incoming events based on
time windows. In our query we will just read the event and output it to the blob
storage. Our input will be the "Event Aggregator".
Diagram 2. Stream Analytics query
Event source can be anything that can do an HTTPS request or AMQP 1.0 connection
to our EventHub. EventHub uses policies to control from whom it will accept
incoming requests from. For scalability reasons and to gain control over
invidual event sources we will use a publisher tokens derived from the policy.
This will also allow us to set the lifetime of the token and as such increasing
the security of our system.
Stream Analytics supports JSON, CSV and Avro as the format of the messages. We
will use JSON as the format of our messages.
Diagram 3. Sample event message
To send the event message to the EventHub we will need to create a publisher
token. First we will need a Shared Access Policy with Send -permission. You can
manage policies in the Azure portal under ServiceBus / YourHub / Configure.
That might change when the new Azure portal gets support for EventHubs. More
information found here Event Hubs Overview.
Currently there's no support for creating publisher tokens in the UI but we can
easily do that in code.
var timeToLive = TimeSpan.FromDays(365*2);
var token = SharedAccessSignatureTokenProvider.GetPublisherSharedAccessSignature(
Diagram 4. Publisher token generation
Note that the EventHub does not store the publishers anywhere. EventHub will
decrypt the token in message using the policy and check the lifetime based on
encryption and decryption. Set the token as the "Authorization" header of the
When using publishers the address where to send the request is similar to
You will receive
201 OK status code as response if the requst is successfull.
This should give you the beginning steps on how to setup the event pipeline
using Azure services and give you long-term storage of your events. In next
article we will add event dispatching based on the EventType.