Stranger Things: Scaling teams and micro services in alternate universes
Over my journey of scaling teams and leading teams that build data products, I learnt and recognised a lot patterns. These are not necessarily design patterns but real life patterns. I have become so creepy sometimes comparing software patterns and real life patterns comes naturally to me but still within my senses and encouraging others to see the alternate universes that I see.
I am no where to call myself as specialist in design patterns about adopting micro services architecture or on how to break monoliths into micro-services compared to brilliant minds in the industry. If you are looking for deep dive on micro service architecture and patterns, I would happily redirect you to Sam Newman or Martin Fowler. However having conversations with many of my teams (past/present) and industry peers, I always found interesting patterns in the evolution of the company that dictates the need to move to micro services as the teams / companies grow. This is what I would like to highlight in this article so that teams can focus on the outcomes rather than technology trends.
Pattern 1: Guard rail your risks
When you begin a start-up or SMB journey, probably the application architecture and the teams RACI around it, is very simple. It would be absurd if someone already thinks about design patterns when you want to market at a rapid phase. Probably monoliths fit all the needs but to be honest, I dont really care at that point in time of my company / team’s growth phase. After a couple of years of hyper growth phase, you suddenly realise there are around 40 engineers in the ecosystems all building new features to acquire new customers or retain them or engaging them with tons of new features in your product. You will see patterns where there are lot of dependencies between teams, roles and functions. You spot bugs due to conflicting changes. Deploying features suddenly becomes a high risk / impact and directly proportional to team sizes. The best example I can provide is we developed a serverless ETL framework that overtime became a monolithic giant. The repo was too big, tons of lambda bundled together, huge dependencies between different instances of these ETL components. Anything we touched had a big probability that it will break things. Test specs were too big, longer to run, high risk to deploy. This is where we found moving to a micro service architecture which automatically builds an organic guard rail around the engineering services with clear segregation of duties. The scope, risk and impact becomes clear to the teams and increases efficiency of the engineering teams.
Pattern 2: Data Complexity
Second pattern is around, the complexity of data landscape. As business grows/scales, data landscape becomes complex. you will see name game / blame game in all companies and usually the last team which provides the data to consumers is in sandwich position. Moving into data domain architectures becomes obvious to create isolation of data. This also created at the same time E2E accountability model for teams rather than centralised data ownership.
Pattern 3: Functional structures
Last trigger comes with evolving organisation structure as you scale your engineering organisation. The adoption of micro-services allows you to give engineers autonomy, creating a more flexible and dynamic working environment. These patterns were natural transitions that created an environment where it made sense to adopt micro-services, and focus on their outcomes rather than technology trends.
You can watch my speech here
