SprintBytes

Azure Data Explorer is a cloud service and a Platform tool which is provided by Azure.  The language ADX uses is Kusto Query Language which is heavily used in most cases when querying in Azure.  After working with ADX and Kusto since 2022, I can share some of the benefits of ADX and why I think Data Engineers should start adapting to this big data service. 

ADX is a big data analytics cloud platform service which provides all the requirements to serve applications and users with an end-to-end solution for data products.  It can ingest un-structured free text or semi-structured JSON format, then parse the data to become consumable in relational format.  Data can be ingested from various cloud sources such as Data Lake (ALDS Gen 2) or other cloud services by using Azure Event Hub.  Azure Event Hub is a data-streaming services which allows data to be processed and ingested into ADX. 

Once the data has been ingested into ADX we can parse and transform the data to create a Serving Layer for Visualization tools such as Power BI or Grafana.  We can then make powerful dashboards using the data from ADX to deliver to our business teams or clients.  ADX also provides visualization capabilities directly from the service itself using Kusto Query Language.  ADX has the capability to create effective dashboards and includes Time-Series insights. 

Data Engineers can really use ADX in a number of different ways.  Along with creating an efficient Data Product with the ADX data pipeline, we can access our ADX cluster through Python to further manipulate and perform datamining techniques required for applications.  ADX clusters can be optimized for performance and cost by managing instance counts, VM SKUs and defining policies (Cache and Retention).  Networking Security models can be placed on cluster to protect the data from public access, which ensures resources are protected from cyber criminals. 

ADX was found between 2014-2016 and since then Kusto has become the Query Language for Azure. Seasoned Data Engineers or Developers who are new in this field, will find ADX as a great tool to achieve their goals as well as a learning tool working with Big Data in the cloud.