As businesses and corporations become exposed to more tools and begin adopting a wider variety of data points associated with their industry, more opportunities and challenges arise with integrating the disparate data sources to provide insights about the business. In this blog series, I want to talk about streaming data into Azure for analytics.
Generally, standard applications are tied to a relational backend database. These standard applications require specific user interaction to generate and modify data written to a transactional database. Historically, to get this data into your data warehouse, an Extraction, Transformation, and Loading (ETL) process is developed. Most of the time, the ETL process to refresh the data happens on specified intervals. From my experience, this type of solution is still suitable today but considered a warm or cold repository depending on the frequency of data loads. With the transition of technology, ETL has also turned into an ELT (Extraction, Load, Transformation process).
The Internet of Things (IoT) has introduced a new element of cloud architecture among modern businesses and corporations; we also see the transition of business intelligence and advanced analytics spanning into more, near real-time solutions. Data from sensors, logs, portable devices, social media, and control and network systems can be generated quickly by user interaction or through some form of automation. A few examples of streaming data in these platforms would be GPS information in transportation, social media posts, devices that measure temperature, and manufacturing equipment with sensors that generates logs.
Due to the frequency, these data points accumulate and are usually not stored on the devices and systems for long periods. This data needs to be captured and stored quickly in order to maintain history. The data can be structured, unstructured or semi-structured due to the many forms and devices it can be generated from. I look forward to providing some solutions for working with this type of data using Azure tools and services. Please stay tuned for future parts to this blog series.