In essence there are three tiers or layers to any business intelligence ecosystem. Bottom up, they are the Sources/Input Layer, the Aggregation/Transformation Layer (traditionally known as The “ETL Layer”) and the Visualization/Reporting Layer.

Whilst these are well known it is important that these layers are symbiotic and the approach an organization takes to each layer will have a significant impact on the options and flexibility that the resultant BI Ecosystem will provide. Looking at the big picture of your data landscape is crucial for data automation and real time reporting.

Frequently BI projects focus on the Visualization/Reporting aspects but place less emphasis on the layers that underpin this. All layers need equal focus to deliver a successful BI Ecosystem in today’s ever-evolving environment.

SME has mapped out three example tech stacks that our customers use most frequently:

 

Lakehouse Architecture

SME's lakehouse architecture is customizable to fit your already existing cloud data lake, like Azure Data Lake or Amazon S3. We utilize tools that seamlessly, and automatically, fill your data lake directly from your data sources.  Once the data is in the lake, a data lakehouse is installed to "sit on top of" that data lake or federated data sources. The data lakehouse serves as a query engine, semantic layer, and data catalog for use in your business intelligence and data analytics tools, like Power BI. 

This solution looks to solve common pain points like:

  • High infrastructure costs from having multiple copies of the data. 
  • No flexibility with it comes to the architecture and having to change it every time a new tool is added.
  • Legacy data warehouse is slow and causing our bottlenecks in our analytics. 

Learn more. 

 

Warehouse Architecture

SME's warehouse architecture is designed for organizations needing to consolidate their data into a scalable warehouse that they are able to connect to their analysis tools, like ThoughtSpot. For the data integration piece, SME utilizes a tool designed to connect to disparate data sources. This architecture includes the ability to perform data transformations inside the data warehouse with low code and no code methods. An agile data governance approach is taken, including a data catalog, data dictionary, and business glossaries. 

This solution looks to solve common pain points like:

  • Current warehouse is too slow and we pay for more compute than we use.
  • Too many data sources organized. 
  • Database Architect's already know SQL and we don't want them to have to spend the time learning a new language. 

Learn more. 

 

Azure Architecture

SME's Azure architecture uses a wide variety of Microsoft solutions to analyze data or prepare it for reporting, machine learning, and data science. This architecture utilizes Azure Data Factory, Azure Data Lake Storage, Azure Synapse for the analytics workspace, and Databricks for data science and machine learning. The Azure Data Catalog is implemented to connect to the data lake or Azure data systems to catalog your data/metadata and add a governance layer. The data can then be analyzed in a Business Intelligence (BI) tool like Power BI.

This solution looks to solve common pain points like:

  • Not using Azure properly or to it's full ability, minimizing the return on investment. 
  • Have Azure Data Factory but not using it.
  • Do not know how to incorporate data science using Synapse and Databricks.

Learn more.