Over the past decade, we have observed open source powered big data and analytics platforms evolve from large data storage containers to massively scalable advanced modeling platforms that seamlessly operate on-premises and in a multi-cloud environment. Cloudera Enterprise is the market leader in this space and working closely with the wider open source community to integrate the latest innovations in machine learning and artificial intelligence (AI). Our platform efforts in this regard are being led by Hilary Mason, founder of Fast Forward Labs, and now general manager of Cloudera’s Machine Learning business unit, whose passion for analytics and innovation has no bounds!
My narrower vision of the next advancement in analytics is driven (or biased) by my quantitative risk management background and the critical role that computational simulation capabilities have played in many advances in the world of finance. These examples are well covered by many others (e.g., Derman (2016), Cesa (2017) & Bouchard (2018)). From this perspective, it is natural to see the next stage of advancement for the modern platform to bring the world of computational simulation into the big data and analytics space. Together with our close partner Simudyne (www.simudyne.com), we are leading the way on this. As announced on April 9th (press release), Cloudera & Simudyne provide the first computational simulation platform built for big data to the financial services sector. Now banks and financial companies can design and run any detailed simulation model at massive scale.
Computational simulation provides methods for studying a wide variety of models of real-world systems. Only through detailed simulations are we able to incorporate the dynamics, feedback and related complexities that are required to represent real-world events accurately. Computational simulation provides a range of tools that include discrete event simulation, system dynamics, network analytics and agent-based models (ABMs).
It’s exactly these ABMs that offer a way to develop high fidelity models of social and financial systems. ABMs consist of an environment that is composed of (software) agents who interact with and influence one another, learn from their experiences and adapt their behaviours so that they are better suited to their environment. They can consist of simple rules, machine learning derived models or even AI-based models such as reinforcement learning models. ABMs are primarily constructed in a bottom-up fashion, where the micro-constituents of a system are represented as interacting agents within a simulation framework. The strength of these models is that they show how even very simple behaviours can combine from the ‘bottom-up’ to recreate the more complex behaviors observed in the real world.
Adding ABM simulations to an organisation’s analytics toolbox allows them to account for emergent phenomena, tail risk events, contagion risks as well as examine states of the world in which traditional machine learning models would break down due to their dependence upon historical data and events.
ABMs can be applied to more effectively model credit, market and liquidity risk exposures, for stress testing purposes, to support financial stability analysis, balance sheet and customer impact analysis from an unlimited range of shocks and to derive new customer insights for competitive advantage. Together with Simudyne, Cloudera makes it possible.
Please visit us at our booth at Strata Data London to learn how to take advantage of the next analytics wave in the big data and analytics world.
Dr. Richard L. Harmon
Global Industry Leader – Financial Services
London – UK
Emanuel Derman. “A Stylized History of Quantitative Finance”. Blog Post, Nov-2016
Jean-Phillipe Bouchard, J. Bonart, J. Connier & M. Gould. Trades, Quotes and Prices: Financial Markets Under the Microscope, Cambridge University Press, 2018.
Mauro Cesa. “A Brief History of Quantitative Finance”. Probability, Uncertainty and Quantitative Risk (2017) 2:6.