The Evolution of Software Development and Future Outlook
Overview of software development evolution, with future insights and suggestions for Engineering Managers and Engineers
In the past 50 years, software development has evolved significantly, and software development teams have evolved with it. In the 70s and 90s, engineers were often viewed as tech wizards, and product managers were seen as brave daredevils willing to tame them.
However, the software development landscape has undergone substantial changes, and this gap is closing rapidly, requiring a shift in development teams and their engineering managers.
While everything continues to evolve rapidly, it's crucial to try to understand where we are headed and prepare for future careers, whether in Engineering or Engineering Management.
Today, after a quick overview of the last fifty years in software development, I will share my perspective on what might come next and some actionable suggestions on how you should prepare, both as Engineering Managers and Engineers.
📝 TL;DR
Over the past five decades, software development has evolved significantly, transitioning from a niche field to a mainstream industry.
Initially, there was a considerable gap between Development Teams and Product Management, but this gap has drastically narrowed in recent years.
Rather than introducing new intermediary roles between Development Teams and Product Management (Product Owners I’m looking at you! 😀), both Engineering Managers and Engineers should collaborate closely with Product Teams and Customers to gain a deeper understanding of their perspectives and needs.
📚 Software Development History
If we take a look back at the last 50 years in software development, we can identify six main phases that have characterized this evolution. Let's delve a bit deeper into each, trying to understand their main traits.
The Early Days (1970s)
Back in the 70s, software development was just beginning. Things were simple but not very organized, as there wasn't a significant demand.
👨💻 Programming Languages: the era of assembly languages and early high-level languages like FORTRAN, COBOL, and C.
🔁 Development Process: often ad hoc with limited formal structure or methodologies.
👥 Teams: small, often consisting of a few individuals who were both developers and operators (and often also hardware experts!)
Structured Programming and PC Revolution (1980s)
In the 80s, things got a bit more serious. People started to make rules for how software should be made.
👨💻 Programming Languages: emergence of object-oriented languages like C++.
🔁 Development Process: introduction of structured programming and early SDLC (Software Development Life Cycle) models.
👥 Teams: growing in size, with more specialization among roles.
The Internet and Open Source Movement (1990s)
The 90s brought us the internet era, and suddenly, everyone could work together from different places.
👨💻 Programming Languages: rise of languages like Java and Python; open-source languages gain popularity.
🔁 Development Process: waterfall methodology dominates; early agile and iterative approaches.
👥 Teams: increasingly complex with the rise of the internet; open-source projects introduce distributed development.
Agile and the Rise of Web Apps (2000s)
When the 2000s came around, the focus was on making software faster and getting it out on the web.
👨💻 Programming Languages: consolidation around a few key languages (Java, C#, JavaScript); beginning of web-focused languages and frameworks.
🔁 Development Process: agile methodologies become mainstream; focus on continuous delivery.
👥 Teams: multidisciplinary teams with close collaboration between developers, testers, and business stakeholders.
DevOps and Cloud Computing (2010s)
The 2010s were all about making things work together smoothly and using the internet to store and access our software.
👨💻 Programming Languages: further diversification with languages like Go and Swift; heavy focus on JavaScript frameworks for web development.
🔁 Development Process: DevOps practices integrate development and operations; widespread adoption of cloud computing.
👥 Teams: more automation and tooling; emphasis on continuous integration and deployment (CI/CD).
AI, Machine Learning, and Remote Work (2020s)
This is where we are still today, the era where smart machines and working from anywhere are becoming normal.
👨💻 Programming Languages: languages and frameworks that facilitate AI and ML development, such as Python with its rich ecosystem. Consolidation around JavaScript and its frameworks.
🔁 Development Process: AI-augmented coding tools; shift to microservices and serverless architectures.
👥 Teams: cross functional teams focused on product features; increase in AI-assisted development.
🧩 Filling the Gaps
When we take a closer look at the past, it's evident that we've gone through a significant transformation. In the earlier years, software development was a bit like black magic, known by a few people and lacking any well-defined procedures. But over time, people started recognizing the business potential of software, even those outside the tech industry. This recognition prompted the creation of structured approaches, known as software development life cycles, aimed at turning software into a marketable product integrated into various business processes.
Software Development Life Cycles
A Software Development Life Cycle (SDLC) is a structured process or framework that outlines the steps and stages involved in designing, creating, testing, and maintaining software applications. It helps ensure that software is developed efficiently, with quality, and in a predictable manner from start to finish.
Over the last 50 years, we have seen two main types of SDLCs.
Waterfall
Waterfall is probably the most traditional software development life cycle, and it's based on a linear process following the concept, design, build, and deploy approach.
The main actors of the waterfall development process are:
Stakeholders/Product Manager: focused on defining the product strategy, aligning it with business goals, and ensuring the product's success in the market.
Engineering Manager: focused on managing the engineering team, ensuring efficient project execution, fostering a productive team environment, and facilitating communication within and outside the team.
Engineering Team: focused on developing the product following the directions of the roles above.
While there's nothing wrong with the waterfall approach (and there are certainly companies that still use it), the need for more flexibility, speed, and adaptability soon arose.
Agile
Agile Software Development was born as an answer to this demand of speed and flexibility. While it shares some similarities with traditional waterfall development from a technical perspective, it is fundamentally based on adaptive iterations and embraces a more flexible and customer-centric approach.
The trade-off for this agility was the need to identify new roles that could help Product Management keep pace and understand both the development process and the technology.
That's why for Scrum, probably the most famous Agile Framework, the main actors are the same as waterfall, with two main additions:
Product Owners: focused on representing the user and business needs within the development team, defining and prioritizing the product backlog, and ensuring the product is built according to requirements.
Scrum Masters: A Scrum Master is like a coach for a team that works on projects. They help the team follow the "Scrum" rules, which are like a game plan for getting work done efficiently and well.
Having profiles like Product Owners and others in the development process, in the last couple of decades, has led to tech teams and product teams slowly filling the huge gap they had initially.
Even though every situation is different, also talking with many colleagues, I believe today Product Managers are more involved in the development process; they have a more comprehensive understanding of technology and its potential. On the other hand, development teams are closer to products and their customers, and for this reason, they should be prepared for it.
🔮 What's Next?
Now that we saw what happened until now, what should we expect and how should we prepare for it, both as Engineering Managers and Engineers in general?
As software becomes increasingly high-level, and with the advent of no-code tools and AI, I expect the gap between product and development teams to continue narrowing down.
Building on this, I have five suggestions for both Engineering Managers and Engineers:
For Engineering Managers
Adopt Cross-Functional Training: invest in cross-functional training for your teams. Ensure that engineers have a basic understanding of product management, and vice versa. This mutual understanding can foster better collaboration and a more cohesive product vision.
Focus on Soft Skills Development: with technical tasks becoming increasingly automated, soft skills like communication, empathy, and leadership will become even more critical. Develop training programs to enhance these skills within your teams.
Facilitate Joint Goal Setting: start each project by setting common goals for both product and development teams. Shared objectives ensure everyone is aligned and working toward the same outcomes.
Implement Regular Cross-Team Meetings: schedule regular meetings where both teams can discuss progress, challenges, and feedback. I do this kind of meeting on a bi-weekly basis, and it's very helpful to improve alignment and build relationships.
Promote User-Centric Development: encourage engineers to participate in user research and testing sessions. This firsthand exposure can lead to more empathetic design and development choices.
For Engineers
Develop Product Sensitivity: take the initiative to understand the product lifecycle, customer needs, and market dynamics. This broader perspective will make you a more valuable team member and open up new career opportunities.
Participate in Product Strategy Discussions: take an active role in product strategy meetings. Your technical insights can help shape a more feasible and innovative product vision.
Adopt a T-Shaped Skillset: we already wrote about different roles shapes. While deep technical expertise is important, complement it with a broad set of soft skills and a basic understanding of other domains. This T-shaped skillset will make you more adaptable and versatile.
Improve Communication Skills: as collaboration becomes more integral, effective communication is key. Work on clearly articulating technical concepts to non-technical team members.
Understand the Business and User Perspective: Step out of your technical bubble and learn about the market, competitors, and user pain points. This broader understanding can inform better development decisions.
That’s all folks
That's all for today! As always, I would love to hear from my readers (and if you've made it this far, you're definitely one of the bravest). Please don't hesitate to connect with me on LinkedIn and send a message. I always respond to everyone!
As always, great overview! I especially loved the tech/product knowledge gap diagram, nice idea there.
I recently read Leo's article from last week about 'Product Engineers':
https://engineercodex.substack.com/p/the-1-trait-of-the-most-valuable
As the roles and responsibilities between EMs (and engineers) and PMs start to blend, I won't be surprised if they'll be joined. In lots of places the PMs are almost scrum masters, creating tickets and managing dailys. Or, the other extreme might happen - PMs will move towards marketing (as in 'Product Marketing' roles), and the gap will start to increase again.
Not all companies operate in the same decade. Some still operate in the 2000s, some in the 10s and some in the 20s.