For anyone interested in learning more about SME in the news, use cases, and customer stories.
In this case study, the SME Team addresses a bank's fraud analysis need by utilizing Azure Data Factory and Snowflake to engineer a solution that improved their time to insight for potential fraud.
5 Billion to 1: Data Engineering for Fraud AnalysiS
Fraud analytics is one of the 3 top big data use cases in financial services.
XYZ Bank processes about 60 million transactions a day and they need to continuously integrate the new data with historical data, consisting of about 5 billion rows of transactions every 90 days. In order to better predict credit/debit card fraud, analysis must be done on this massive amount of data. The below contains the challenges, goals, initiatives, and impacts for the fraud analytics case study.
The bank is facing extreme difficulty with effectively accessing and leveraging over 5 billion rows of credit and debit card transactions in their transactional database.
They need this transactional data to be integrated into the data warehouse while also being archived in the data lake.
They want to use Power BI for analysis but they cannot connect Power BI directly to entire data set because...
If they try to load the data into the report, the memory limit is reached and the report fails to reload.
Using direct query requires the largest warehouse that would increase their costs drastically if used throughout the day.
The bank partnered with SME to architect a solution that would:
Leverage 3 months of data to more accurately capture attempted fraudulent transactions.
Set the bank up for improved accuracy when they shift to the predictive use case: fraud detection.
Improve analytical report performance, speed, and reliability by connecting to curated data marts.
In order to provide a solution that met the bank's goals, SME assessed their current data ecosystem. From the assessment, a plan was constructed and implemented. These were the key steps our technical team performed for XYZ Bank:
Use Azure Data Factory to move data from the transactional database, Azure SQL, to Azure Data Lake Storage
Use Azure Data Factory to partition the data, store in the data lake, and load into Snowflake
Perform complex data transformations within Snowflake to prepare a data mart for analysis
Direct query the transformed data mart inside Snowflake with Power BI
Following the deployment of the solution, it was determined that not only were the goals met but exceeded them by providing real business impacts:
Actionable insights unearthed after visualizing outliers in the data
Drilled down from 5 billion records to one record, identifying which accounts have unusual spend activity
SME's George Barrett was the lead Solutions Engineer on the project. After scrubbing and anonymizing the data, he led a webinar that went into detail on this case study and includes an end-to-end demo of “5 Billion to 1: Data Engineering for Fraud Analysis”.