Lessons learned - ingesting, securing and processing 200 million messages a day

By Jess Panni

Talk Abstract:

Even with the scale of the cloud, processing 200 million messages per day is not trivial, especially when those messages contain personally identifiable information (PII) which means that you have to make GDPR considerations a fundamental part of not only your architecture, but also your operational processes to ensure your security protocols are enforced. This session will dig into the use cases that a modern data platform needs to fulfil to support advanced analytical workloads; gathering data in real-time using Azure Event Hubs, creating a "hot path" using Azure Functions and Power BI and creating a "cold path" to Azure Data Lake so that historic data can be analysed at scale. It will also cover the role of DevSecOps and how many of the required security protocols can be enforced using automation and Visual Studio Team Services. The session consists of 15 minutes of presentation and 30 minutes of technical walkthrough and demos.

About Jess Panni

Jess has over 20 years’ experience helping companies succeed through the smart use of technology. He has spent most of his career working for leading Microsoft partners across the UK and Australia and is now Principal at endjin, working with clients to help them transform into modern, data-driven cloud-first organisations.