Ambassador post originally published on Medium by Mathieu Benoit
Platform Engineering was a huge theme at KubeCon NA 2023 in Chicago. More than ever before.
More and more companies have initiatives to streamline platform engineering best practices and build internal developer platforms.
KubeCon has historically been an infrastructure and operations-focused event, but we mustn’t forget about our primary (platform) customers: the developers.
— Daniel Bryant
This KubeCon NA 2023 edition was a blast! That’s the second time I’m attending this conference in-person. First time as Cloud Native Ambassador (CNA) and first time working for Humanitec. And again, my key takeaway is that learnings and big outcomes of attending conferences in person are coming from the people you meet there, the discussions you have, the demo or talk you deliver or see, etc. That’s why I’m less and less attending talks to be honest. I watch the sessions afterward when they are available a few days after the conference on YouTube: CNCF [Cloud Native Computing Foundation].
What’s also important is what happens around the main conference itself: Cloud Native Rejekts, Co-located events like BackstageCon, IstioCon, ArgoCon, the solutions showcase area, etc. And also all the announcements from communities, products, etc. I will mention a lot of them in this blog post.
My key takeaways
With this KubeCon NA 2023 edition, here are the themes I captured around Platform Engineering that I will cover throughout this blog post:
- Capabilities and building blocks of a Platform, many technologies are the enablers for shaping your platform.
- Developer Experience — associated with Developer Productivity are the main business drivers for Platform Engineering.
- Maturity, best practices and lessons learned—We are now seeing more and more talks sharig concrete tips, failures, success, and recipes about their own journey with platform engineering on top of Kubernetes. So insightful and informative!
- The rise of Platform Orchestrators — A Platform Orchestrator is what will improve the experience of the interactions and abstractions of the different stakeholders with your Platform, at scale. Key building block here!
Capabilities and building blocks of a Platform
Built on top of OSS and community standards, the CNCF TAG App Delivery announced the whitepaper on Platforms for Cloud-native Computing:
It presents how today’s CNCF projects fit together as the foundation of complete platform initiatives.
The talk Introducing the BACK Stack! by Ritesh Patel (Nirmata) & David Murphy (Upbound) is a concrete example of reusing this reference architecture in this whitepaper and provide opinionated choice of tools with Backstage, Argo, Crossplane and Kyverno:
Which is also very well aligned with the recent CNOE initiative:
Cloud Native Operational Excellence (aka, CNOE, pronounced Kuh.noo) is a joint effort to share developer tooling, thoughts, and patterns to help organizations make informed technology choices and resolve common pain points.
— CNOE — A Joint Effort (Adobe, AWS, Autodesk, Salesforce and Twilio) to Share Internal Developer Platform Tools and Best Practices. | CNOE
Note: CNOE was presented during the CNCF TAG App-Delivery Meetup @ KubeCon NA 2023.
One of the building blocks that piqued my interest during this KubeCon NA 2023 was Dapr. I recently wrote a blog post about my experience with it here: Dapr with Score and Humanitec — Improving the Developer Experience of your Platform, on steroids!
Another popular building block is Backstage. Backstage got its dedicated co-located event BackstageCon NA 2023 and the videos are already available here. But what I’m still hearing everywhere is “Backstage is key and very powerful to build your own custom unified Developer portal/UI, but it’s complex to build and maintain”. There is also and still a lot of confusion between a Developer Portal versus a Developer Platform.
Here are some talks and notes I captured in order to gain more knowledge from people who have been implementing Backstage for years now:
- Five Backstage lessons in 5 minutes — Backstage is not magic and requires an organizational journey. Backstage must align with both Leadership and Developer goals. If you are struggling with adoption, you are not focusing on real problems.
- 4 Common Pitfalls when Adopting Backstage (KCD Austria) — Backstage is a great tool to have a unified developer portal, but it could be complicated to start with as well as deal with day 2 operations with it.
- Jorge Lainfiesta — Is Backstage for me? (KCD Denmark) — Great overview of the Backstage features with practical best practices to see when and if you need them or not.
So, you now have your platform on top of Kubernetes, with Backstage, Argo/Flux, Crossplane/Terraform, Istio/Linkerd, Kyverno/OPA Gatekeeper, etc.
Great!
But at the end of the day, that’s just tools and technologies, right? Have you resolved the most important part of abstracting your platform to your users — the developers of the products of your company? Do they have the right level of abstraction? Maybe not, right?
From here, Developers are still writing Kubernetes YAML files, Helm charts or Kustomize overlays. Is it the most efficient experience for them? Do they need to be Kubernetes (at least at the CKAD level) experts? Maybe not, right?
Do they need to be Security experts too? That’s what this Cloud Native Rejekts talk from Melissa McKay (JFrog) is really well tackling:
An expectation that all developers by default are equipped to handle this additional workload is unreasonable.
— Don’t Expect Developers to be Security Experts! :: Cloud Native Rejekts NA (Chicago) 2023
The real miss at this point is still the actual Developer Experience!
Developer eXperience (DX)
From my previous blog post Why Platform Engineering is so important? And what is it by the way?, here is how I could summarize how to define Platform Engineering:
To increase velocity while maintaining stability, Platform Engineering is about the Developer eXperience. To avoid frictions and accelerate adoption with well-supported Golden Paths exposed by your platform abstracting complexity. Platform that is continuously improved with a product mindset.
This KubeCon NA 2023 keynote: Blueprint Banter: Cloud Native Conversations by the Fireside provided great insights about the Developer Experience perspective:
As Developer, infrastructure is abstracted, it just works. We need to ship code, not doing Kubernetes nor Infrastructure, focused on velocity and business value. Security is centralized and shifted-down to the platform, not shifted-left to the developers…
That’s also exactly what Kelsey Hightower and Nick Durkin discussed recently:
Do not shift left to the devs, shift back/down to the platform, and serialize months/years of learnings in the platform instead of asking all your developers to do Kubernetes, Infrastructure, Security, etc.
During the Open Source Summit China 2023, Puja Abassi did this talk and I think he really nailed it too:
Capabilities need to be from the user perspective, not from tech or platform capabilities — Not “I want Grafana”, also not “I need Observability”, but rather “I need to debug my application and I need data to help me with that”.
Jobs-To-Be-Done (JTBD) — Not “I want a drill”, “or I want a hole in the wall”, but rather “I want to hang this picture”.
Maturity, best practices and lessons learned
The CNCF Platforms Working Group (WG) has recently launched the first release of a platform engineering maturity model. This is organizing patterns and observations into a progressive maturity model with 5 aspects (Investment, Adoption, Interfaces, Operations and Measurement) and 4 levels of maturity (Provisional, Operational, Scalable and Optimizing).
Speaking of maturity and patterns, I found 2 great talks during KubeCon NA 2023 where speakers shared concrete tips, failures, success, and recipes about their own journey with platform engineering on top of Kubernetes! So insightful and informative! Kudos to the authors!
Talk given in KCD UK 2023 too.
Join us in this session as we delve into the successes, failures, and valuable lessons learned throughout the building of our platform.
Their platform based on Kubernetes brought different benefits to the company and developer teams:
- Standardization and unification
- Security and compliance
- Cost optimization
- Faster deployment cycles
Issues and lessons learned:
- Internal team — Having a core team and not just counting on external consultants is important.
- Documentation — Technical documentation, release notes, blog posts are very important to share knowledge.
- Feedback — Development teams feedback is what matters, importance of having a Product Owner.
- Migration plan — Change in technologies and processes shouldn’t be under-estimated. You need to have testing scenarios, rollback plans in order to avoid any major incidents.
- Other aspects — Cost optimization with resource tages and stop/start automation, prevent incidents with locks and soft-delete policies.
This talk will help your platform team in handling the current and upcoming post-merger scenario from the unique experience and lessons learned in how our team manages the challenges in onboarding multiple engineering teams to our IDP that we think is still rarely talked in the community.
In their reference architecture (template from McKinsey – developer platform reference architecture), they show their different building blocks and tools, and something to call out is the need of their custom Platform Orchestrator. This is an extensible tool, generating Kubernetes manifests in order to then have ArgoCD deploying them. I think this is key, and exactly what I will discuss later in this blog post in my Humanitec section.
Another best practice highlighted below is about the importance of the concept of Task Force (Enabling Team in Team Topologies), bridging and connecting the dots between leadership, product and platform teams:
Their key takeaways:
- Consolidating tools bring cost benefits, recover productivity, and improve collaboration.
- Finding allies across companies to address gaps and advocating for adoption.
- Favor consolidation by movements, but ask leadership mandates to speed up alignment.
If you want to hear more about their story, they also shared more insights about their ArgoCD implementation: GoTo Financial’s Story: Towards 10k ArgoCD Apps to Support Billions of $ Transactions — Yudi Andrean Phanama & Giri Kuncoro, GoTo Financial (video).
The rise of Platform Orchestrators!
Throughout most of the content of this blog post I’m wearing my Cloud Native Ambassador (CNA) hat. Here, I will wear my Humanitec hat. I’m Customer Success Engineer there, implementing concrete internal developer platforms with customers and our Humanitec Orchestrator product. Exciting times!
At KubeCon NA 2023, we were one of the sponsors and we had a booth. And I was fortunate to be there to talk to customers, prospects and the community. So inspiring and important for me to listen and talk about real problems. We were able to meet with hundreds of people, demo our products and show what are the values for companies having already a platform but still having issues with how to scale and how to optimize how their developers consume their platform.
My key takeaways and the pain points I’m still hearing all the time from these discussions (disclaimer: I’m talking at scale here, with enterprises already having a platform in place and some level of maturity):
- The shift left to the Developers doesn’t work. Developers shouldn’t do Kubernetes manifests, Kubernetes security nor Terraform. They want to focus on their code. And they should.
- The burden on the Platform Engineers is too high. They need to serialize security and governance standards and common best practices down to the Platform, without being a bottleneck nor constraining the Developers velocity.
And that’s exactly what Humanitec is solving, our products enable platform teams to reduce cognitive load, drive standardization and slash time to market.
Humanitec empowers platform engineers to build the perfect Internal Developer Platform for the enterprise. Our products enable platform teams to reduce cognitive load, drive standardization and slash time to market.
- Score: The driving principle here is that you want your developers to describe their workload and related dependencies in abstract terms, using a workload specification like Score.
- Humanitec Orchestrator: It takes the abstract request and the relevant context and matches it to baseline resource definitions to create or update the associated infrastructure. It then generates the config files and wires everything up, to eventually deploy them in a targeted Kubernetes cluster.
- Humanitec Portal: This UI provides a single pane of glass into usage, metrics, health scores and actions. An intuitive interface for self-service actions.
With Humanitec, our developers don’t need to touch Kubernetes unless they want to. This eased a lot of pressure from the team, allowing them to focus on what they do best — creating amazing solutions for our customers.
— Igor Kantor, Director of Software Engineering at Convera
Note: Score was also presented during the CNCF TAG App-Delivery Meetup @ KubeCon NA 2023 by Atulpriya Sharma.
Oh, and by the way, if you want to see this illustrated, live, with a concrete scenario, have a look at this recent blog post I wrote: Dapr with Score and Humanitec — Improving the Developer Experience of your Platform, on steroids!
Our Product & Engineering teams have been very busy in the last few weeks and months releasing new features for our customers and I was able to do some demos to show these great stuffs!
- Humanitec reference architectures for Internal Developer Platforms are out! Checkout the intro and the associated tutorials.
- Humanitec CLI
- Humanitec Operator with External Secret Stores
- Custom Workload Profiles
- New Service User UI
- Terraform Driver as GA
- Simplify change promotion with our new cloning UI
- Introducing the redesigned Platform Orchestrator UI
That’s a Wrap!
As mentioned, I had a blast attending this KubeCon NA 2023, meeting and discussing with people was so insightful. As well as demoing Humanitec products with our Internal Developer Platform reference architectures.
KubeCon conferences and talks are now more and more focused on customers, and developers.
Platform Engineering was everywhere, have you seen any talk not talking about Platform Engineering? 🙂
The questions are not anymore about how to build a platform on top of Kubernetes. We have a lot of pointers, whitepapers and reference architectures for that. Great coverage and great content if you are still at this stage.
Developer Experience is one of the primary drivers and outcome when building a Platform. Seeing this Platform as a Product is one of the good practices.
Now, what the community and end users want to share (and learn from each other) is how to scale and optimize their own Platform implementation: shifting down not shifting left, real experiences, best practices, success/failure stories, etc. that’s what we are seeing more and more.
Last but not least, the cherry or the icing on the cake, is the ultimate building block you want for your Platform, the Orchestrator:
I’m demonstrating this Platform Orchestration part in this other blog post if you are interested in seeing this in action with more details: Dapr with Score and Humanitec — Improving the Developer Experience of your Platform, on steroids!
Platform Orchestration is one of the techniques described by ThoughtWorks and also recently documented by Microsoft (they just released Radius aligned with that too):
A developer platform orchestrator allows developers or systems to create requests to perform an action using a template. It doesn’t perform these actions itself, but instead coordinates with a task engine, workflow engine or another orchestrator to do so. It’s one of the critical pieces you’ll want to be sure is a part of your self-service foundation.
— Design a developer self-service foundation | Microsoft Learn
Spot on!
What’s next?
More to come about Platform Engineering in 2024, that’s for sure!
During KubeCon + CloudNativeCon Europe (March 2024) where you will hear and learn more about Platform Engineering:
Also, I highly recommend you to join the global home for Platform Engineers community (platformengineering.org) and register to the coming PlatformCon 2024 — The Platform Engineering Conference in June 2024, and submit talks there! You can see how the 2023 edition was a huge success here: Top PlatformCon 2023 insights and highlights.
Last but not least, if you want to directly hear from the WG Platforms of the CNCF TAG App Delivery, here you are: CNCF TAG App Delivery. You can also read more about getting started with contributing in WG Platforms | CNCF TAG App Delivery.
I want to hear from you!
What about your own takes from KubeCon NA 2023? Any talks, announcements, etc. around Platform Engineering I missed in this blog post?
Please leave a comment to share what you learned and what are your key takeaways!
Other great write ups of KubeCon NA 2023
Here are other great write ups of KubeCon NA 2023, not just focusing on Platform Engineering, so with that you hear more about trends around Gen AI, Security, WASM, etc.
- KubeCon North America 2023 Retrospective: What A Difference A Year Makes | by Adri Villela
- KubeCon Chicago Key Takeaways. The (slow) rise of AI, the domination… | by Daniel Bryant
- KubeCon + CloudNativeCon North America 2023 Wrap-up | by Allesandro Vozza
- KubeCon NA 2023 | by Mauricio Salatino
- Key Takeaways from KubeCon + CloudNativeCon North America 2023 | by Ankur Kumar
- Kubernetes Enters Its Second Decade: Insights from KubeCon Chicago (chkk.io)
- My KubeCon + CloudNativeCon Adventure | by Hope Oluwalolope
- The Cloud SustaiNative Era: A Recap of KubeCon + CloudNativeCon NA 2023
- My Adventure at KubeCon: A Techie Teen’s Take on the Tech Universe — DEV Community
- My KubeCon + CloudNativeCon NA Chicago 2023 Experience | by Atulpriya Sharma
- Kubernetes Podcast from Google: KubeCon NA 2023
Hope you enjoyed that one! Happy sailing, happy platforming! 🙂