If you’re new to Cloud services like AWS, Azure and Google Cloud, you may be in for a shock when you arrive. Prepare yourself for…..The Microservice Maze.
If you’re going to take advantage of The Cloud (and those advantages are huge, so you should be considering it), you’ll need to learn to navigate The Maze.
I was prepping for the Amazon Web Services Certified Cloud Practitioner exam this summer (which I passed, thankfully), and quickly found myself feeling lost. AWS now has over two hundred individual “services” as a part of their platform, and they release more by the month. Azure, Google Cloud, and other platforms are similarly bewildering. The Maze I found myself in was very different from how I’m used to learning enterprise software, because the architecture is so fundamentally different.
I’m used to enterprise software being linear.
Import data -> do some things -> export some stuff. If you’re familiar with standard pieces of oil & gas software like The Big Six (ERP, reserves/planning, well lifecycle, land, production allocation, and SCADA), this is the path they follow. Import -> do -> export. REPEAT. Complexity exists, but it’s usually hidden from the business user.
Modern cloud infrastructure isn’t like that at all.
The microservices swirl around and interact with each other – storage feeds and is fed, compute spins up and spins down, security touches each endpoint and the paths between. It’s an Escher drawing, but even nerdier.
A long-time friend and cloud computing guru used an analogy that crystallized this for me. He said that a modern cloud platform is like Lowe’s or Home Depot, not a department store. The platform offers you all kinds of raw materials like lumber, screws, or pipe. It’s up to you to figure out how to put them together for your specific project. They don’t tell you what you’re allowed to build, or even what you should build – that’s the job of an architect, an electrician, or a landscaper. Or even better, a weekend warrior that should know better than to mess with plumbing (like myself).
To stretch the analogy a bit more, consider the lawnmowers, ceiling fans, and pellet smokers. These are complex, valuable pieces of equipment that allow you to do interesting things (although I strongly prefer brisket to yard work). Similarly, cloud services like AWS Kinesis (streaming data management), Azure Purview (data governance), and GCP AutoML (assisted machine learning models) provide incredibly powerful tools that make the developer dramatically more productive than they would be on their own.
As the big three cloud providers compete against each other, their platforms look increasingly similar to my eyes. The components have different names, but the underlying functions are the same. Satish Chandra Gupta draws beautiful diagrams that communicate this eloquently – different names and different branding, but the lumber, screws, and pipe are the same.
So, what is a petroleum engineer, oil and gas accountant, or IT professional to do with this information?
Start learning your way around the maze. Take some classes (we offer them), talk to some friends, read blog posts like this. Learn about the components and how effective implementations stitch them together. Maybe even try building a treehouse (aka website). I find these tools very easy to learn vs. conventional on-premise solutions, but you’ve got to put in the effort.
The opportunities modern cloud platforms present are too good to ignore. The performance, cost, security, and reliability are innovating at blistering speeds, in a way that on-premises infrastructure can’t match. Whether you’re using IaaS, PaaS, or SaaS, innovative companies have consistently proven that cloud-first strategies aren’t just viable, they’re preferable. Just be ready to get lost in the Maze once or twice along the way. And maybe bring along someone with a map.
Big data pipeline architecture on Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). Image is by Satish Chandra Gupta, and released under Creative Commons BY-NC-ND 4.0 International license.