In part 1 of this series, we covered the definition and types of streaming data.  That blog can be found here.  Once you have identified your business need to utilize streaming data in your Azure analytics solution, it is time to determine what options and services are available to connect to and ingest this data. 

Within the Azure Streaming Analytics platform, the following options are available, IoT Hubs, Event Hubs and Service Buses.  In part 2 of this series, we will be ingesting streaming data using Event Hubs.  We will be leveraging Python to generate some telemetry data to push to the Event Hub.

 

First, we need to create the event hub resource in our Azure Resource Group.

 

 

 

Create the Event Hub Namespace.  The namespace is a container that will allow you to create one or many event hubs.  I chose the basic pricing tier for demo purposes.  Throughput controls the Event Hub traffic.  One throughput unit allows 1 MB per second ingress and 2 MB per second egress.  Since this is only a proof of concept, I chose one Throughput.  This may be a setting to evaluate based on data size and frequency of ingestion to prevent throttling.

 

 

 

After we create the Event Hub namespace, we need to create and configure the Event Hub itself.  The Event Hub is the resource that receives, processes and stores data messages

 

 

 

There are several configurations in creating an event hub that can be important depending the amount and frequency of data being pushed to it.  The partition count determines how the subset of messages are sequenced and read.  Newer data is always added to the end of the partition.   The level of concurrency for reading event hubs has a huge impact on performance.  Since partition counts cannot be changed after the fact, they should be set according to the expected workload for scalability.  Microsoft recommends the number of partitions should be equal to or greater than the number of throughput units for best performance.  Message retention determines how long messages are stored in the event hub.

 

 

 

Now that we have the Event Hub instance created, we need to get the shared access key to define how to connect to the Event Hub from an application.  From the namespace window, select Shared access policies.

 

 

 

 

Once in the shared access policies, select RootManageSharedAccessKey.

 

 

 

You want to use the primary connection string of RootManageSharedAccessKey.  Click the copy button to place it in your clipboard.

 

 

 

For the sake of this demo, I created a python script to generate record sets to send as messages to the event hub.  Below you will see the test code and parameters configured.  The key components of this script are ensuring you have the json and service bus libraries for python.  There is also an event hub library as well so feel free to use whatever libraries you feel most comfortable with.  We will use the shared access key from above to make the connection to our Event Hub.  The remainder of the script is simply generating random weather data and sending to the service bus as events in the form of json.  Replace  the following items with your own configuration values:  ADDRESS, USER, KEY, service_namespace, send_event.

 

import uuid
import datetime
import random
import json
from azure.servicebus.control_client import ServiceBusService

# Address can be in either of these formats:
# "amqps://<URL-encoded-SAS-policy>:<URL-encoded-SAS-key>@<mynamespace>.servicebus.windows.net/myeventhub"
# "amqps://<mynamespace>.servicebus.windows.net/myeventhub"
# For example:
ADDRESS = "amqps://StreamWeatherData.servicebus.windows.net/weatherdatamessagestream"

# SAS policy and key are not required if they are encoded in the URL
USER = "RootManageSharedAccessKey"
KEY = "fh6m8IpVSACft9UQyConSxWIbs/TBtqUHfAEDJesWiw="

sbs = ServiceBusService(service_namespace='StreamWeatherData', shared_access_key_name=USER, shared_access_key_value=KEY)
devices = []
for x in range(0, 10):
    devices.append(str(uuid.uuid4()))

for y in range(0,10):
    for dev in devices:
        reading = {"WeatherTimeStamp": str(datetime.datetime.utcnow()), "Temperature": random.randint(20, 100), "Visibility":  random.randint(0, 10), "WindSpeed": random.randint(0,60) }
        s = json.dumps(reading)
        sbs.send_event('weatherdatamessagestream', s)
    #print (reading)

 

 

Once you modify and run the above python code in your IDE of choice according to your environment, you should be able to refresh the metrics section of the Event Hub.  Below you will see the 10 events we generated from the python script.

 

 

 

Now that we have data in our event hub, we have a couple of options for integrating and using the data.  We can set up a streaming job or create an event followed by some sort of trigger.  For the streaming jobs option, I will be releasing another blog shortly to finish the 3rd and final part of this series.

Stay tuned…

 

As businesses and corporations become exposed to more tools and begin adopting a wider variety of data points associated with their industry, more opportunities and challenges arise with integrating the disparate data sources to provide insights about the business. In this blog series, I want to talk about streaming data into Azure for analytics.

Generally, standard applications are tied to a relational backend database. These standard applications require specific user interaction to generate and modify data written to a transactional database. Historically, to get this data into your data warehouse, an Extraction, Transformation, and Loading (ETL) process is developed. Most of the time, the ETL process to refresh the data happens on specified intervals. From my experience, this type of solution is still suitable today but considered a warm or cold repository depending on the frequency of data loads. With the transition of technology, ETL has also turned into an ELT (Extraction, Load, Transformation process).

The Internet of Things (IoT) has introduced a new element of cloud architecture among modern businesses and corporations; we also see the transition of business intelligence and advanced analytics spanning into more, near real-time solutions. Data from sensors, logs, portable devices, social media, and control and network systems can be generated quickly by user interaction or through some form of automation. A few examples of streaming data in these platforms would be GPS information in transportation, social media posts, devices that measure temperature, and manufacturing equipment with sensors that generates logs.

Due to the frequency, these data points accumulate and are usually not stored on the devices and systems for long periods. This data needs to be captured and stored quickly in order to maintain history. The data can be structured, unstructured or semi-structured due to the many forms and devices it can be generated from. I look forward to providing some solutions for working with this type of data using Azure tools and services. Please stay tuned for future parts to this blog series.

My experience at PASS Summit 2017 was one of the most memorable times of my life.  In addition to attending some amazing presentations and spending valuable time with the SQL Family, I had the opportunity to compete with so many great speakers and colleagues in Speaker Idol. 

My Experience at PASS as a First Timer

In 2015, I attended PASS Summit for the first time.  I remember all of the hype leading up to the big event.  A month earlier, I had just spoken at my first SQL Saturday and was eager to learn more and get involved with the community.  As a first timer, I was blown away by the wealth of knowledge and the support shown by my peers.  They welcomed me along with so many others with open arms into the SQL family.  I was even more motivated to continue speaking with the goal of standing on the big stage at a future PASS Summit. 

I applied to as many SQL Saturdays as I could fit into my work and family schedule.  As a new speaker, I was and still am extremely grateful to the organizers and staff who accepted my submissions and gave me an opportunity to be a part of their event.   I also had the opportunity of speaking at user groups, both local and virtual.  The more I spoke the better I got.  I used the speaker evaluations and advice given to me by my close friends and colleagues to improve.

Taking the Next Step as a Frequent Speaker

At the beginning of 2017, I worked up the courage to submit several sessions to PASS Summit.  I knew it would be a stretch to be accepted but I wanted to get a feel for the process; besides, nothing beats a failure but a try.  I also submitted to Speaker Idol for a chance to land a spot in the top 12.  This could provide another opportunity to present at one of the biggest conferences in the SQL community.

After receiving the call for speakers results email in July of 2017 and finding out none of my sessions were accepted, I was not discouraged.  I was happy to have experienced the process and congratulated my colleagues and friends who were accepted.  A couple of months later I got the same results for Speaker Idol.  Some of my close friends were accepted and I was happy to root them on and show my support.  Sometimes when things do not work out how you anticipate, it is easy to get down on yourself and get discouraged.  I found myself reflecting and feeling blessed for all of the chances I had to speak at the many SQL Saturdays, user groups, etc. 

Selected For Speaker Idol at PASS Summit 2017

On Oct. 9 2017, I received an email from Denny Cherry with a spot available for Speaker Idol after a contestant had to drop out.  With such a short time frame to prepare, I was nervous; however, I stepped out on a limb and accepted the challenge.  I worked diligently to put some material together.  I presented my talk to a couple of close friends.  I was over time and had some other changes I had to make.  I was used to speaking for 60 minutes.  I was thinking, “How in the world can I present something front to back in 5 minutes?”  With much practice and some feedback from my friends, I was getting closer and closer.

Fortunately, I was on the schedule to present in the third round on Friday of PASS Summit.  This gave me more time to prepare and to get a feel how things would flow by attending the first two rounds.  I practiced diligently, almost memorizing the speech without adlibbing.  This really helped me save on time and provided for a tighter and consistent presentation.  There were many good presenters.  I knew I had to be on point in order to win.  The judges were critical but I loved that about the competition.  How can you improve if you do not get honest feedback from top notch/experienced individuals.  After a close third round, I was surprised and blessed to be selected the winner and to move on to the finals.

Winner of Round 3, Headed to the Finals!

I took to heart what the judges had to say and I knew improving on those areas would be crucial for the final round.  I found a corner right off the main crosswalk and began making changes and practicing.  I found a couple of friends to bounce some ideas off as well for a second set of eyes/ears.  The final was even grander than the prelim rounds (of course it would be, it is the FINALS!).  The stage was bigger and so was the audience.  I drew second and was prepared to go.  I even had two MVPs directly behind me try to rattle me in all fun.  With little time to fit the changes into the presentation and practice, I was a little concerned about time and delivery.  Once I took the stage, the nerves and worries subsided.  The adrenaline and passion for speaking took over. 

The contest was extremely close and everyone did a great job.  After the remaining finalists presented, the judges went to deliberate and determine a winner.  We had the privilege of enjoying some entertainment from @KillaDBA who put on a nice show while we awaited the results.  The judges were in the deliberation chambers for a good while.  I knew it would be a tough decision for them.   As they entered the room, everyone was on the edge of their seat.  They announced the runner up, Simon Whiteley.  Simon did an amazing job and could have very well won the competition. 

It was now down to myself and two other speakers.  The anticipation was enormous.  What felt like 5 minutes, but actually was a matter of seconds, they announced me as the winner!  It was a surreal moment and a joy that I will not ever forget.  I was and still am extremely grateful to Denny Cherry for giving me a chance to present, to my friends and colleagues for supporting and being there for me and for the judges and there feedback and constructive criticism that allowed me to better my presentation. 

Preparing For PASS Summit 2018!

Now on to the big stage!  It’s a dream come true to say “I am speaking at PASS Summit 2018”.  I am extremely excited to present my session on dynamic SSAS partition processing.  I encourage anyone who wants a great opportunity to learn and fellowship with the SQL Community, to register and attend the SQL Saturdays that are close to you as well is PASS Summit if possible.  For any speaker’s interested in submitting to speak at Speaker Idol, try and speak at as many SQL Saturdays as possible as well as local user groups.  If selected to speak at Speaker Idol, here are some tips from my experience:

  1. Tell a story.
  2. Do not try to cover too much material. Either focus on a high-level overview or pick one specific topic to give details.
  3. Tie your slides together well and have good pacing between each.
  4. Site your images.
  5. If you choose to talk about technical details, give negatives or things to look out for in addition to positives.
  6. Engage and make eye contact with the audience and judges.
  7. Be confident but not egotistical in the material you are covering.
  8. Finish under 5 minutes

My journey has been a blessing and opportunity of a lifetime!  I look forward to continuing my journey in the world of SQL Server and being a part of the #SQLFamily and community.  I am excited to see what the future holds.