As we learned from the first blog in our series, RBC has taken a holistic approach on our cloud journey, as it enables technologies across our enterprise. In today’s post, we’re going to share how Capital Markets has integrated Kubernetes and FluxCD into our technology stack.
In the fast-paced world of capital markets, agility, reliability, and scalability are paramount. At RBC Capital Markets, we are constantly evolving our technology stack to meet the needs of our clients. One of the key enablers of this evolution has been our adoption of Kubernetes for application deployment. In this blog post, we’ll take you through our journey of deploying applications on Kubernetes, with a particular focus on how we leverage FluxCD to manage our platform and user applications as tenants.
The need for Kubernetes in capital markets
Capital markets operate in a highly dynamic environment where milliseconds can make a difference. Our applications need to be highly available, scalable, and resilient to failures. Kubernetes, with its capabilities, is an industry standard for managing containerized applications at scale. It allows us to:
- Scale applications dynamically based on market demand.
- Ensure high availability with self-healing capabilities.
- Roll out updates seamlessly with zero downtime.
- Optimize resource utilization across our infrastructure.
However, managing Kubernetes at scale, especially in a multi-tenant environment like ours, comes with its own set of challenges. This is where FluxCD comes into play.
Introducing FluxCD: GitOps for Kubernetes
FluxCD is a GitOps tool that helps us automate the deployment and management of applications on Kubernetes. It operates on the principle of declarative infrastructure as code, where the desired state of our applications and infrastructure is defined in Open Container Initiative (OCI) repositories. FluxCD continuously monitors these repositories and ensures that the actual state of the cluster matches the desired state. It allows us to:
- Automate deployments across multiple environments.
- Enforce consistency by using OCI as the single source of truth.
- Simplify rollbacks by reverting to previous OCI version.
- Enable self-service for tenants while maintaining strict governance.
Our deployment architecture
We operate across multiple teams (e.g., trading desks, risk management teams, equities application team), where each of them deploy and manage their own applications on our Kubernetes platform. Here’s how we’ve structured our deployment pipeline using FluxCD:
1. Platform deployment
The platform layer includes all the shared services and infrastructure components that are required to run our applications. This includes:
- Ingress controllers for routing traffic.
- Monitoring and logging tools like Prometheus, Loki, and Grafana.
- Cert management and network configurations.
- Custom resource definitions (CRDs) for tenant-specific configurations.
We use a dedicated OCI repository to define the desired state of our platform. FluxCD monitors this repository and applies any changes to the Kubernetes cluster automatically. This ensures that our platform is always up-to-date and consistent across all environments.
2. Tenant application deployment
Each tenant has its own OCI repository where they define the desired state of their applications. These repositories include:
- Kubernetes manifests (Deployments, Services, ConfigMaps, etc.).
- Helm charts for packaging applications.
- Environment-specific configurations (e.g., dev, qa, production).
FluxCD is configured to monitor these tenant repositories and deploy applications to their respective namespaces in the Kubernetes cluster. This allows tenants to manage their applications independently while adhering to the platform’s governance policies.
3. Multi-tenant isolation
To ensure isolation between tenants, we use Kubernetes namespaces and role-based access control (RBAC). Each tenant is assigned a dedicated namespace where they can deploy and manage their applications. FluxCD enforces RBAC policies to ensure that tenants can only modify resources within their own namespaces.
4. Continuous Delivery Pipeline
FluxCD integrates seamlessly with our CI/CD (Continuous Integration/Continuous Dployment) pipeline. When a tenant pushes changes to their Git repository, FluxCD automatically picks up the changes and deploys them to the appropriate environment. This enables us to achieve continuous delivery with minimal manual intervention.
Benefits of using FluxCD at RBC Capital Markets
The improvements we have seen in our deployment process are:
- Faster time-to-market: Automated deployments reduce the time it takes to roll out new features and updates.
- Improved reliability: GitOps ensures that our deployments are consistent and reproducible, reducing the risk of errors.
- Enhanced collaboration: Tenants can manage their applications independently, while platform teams focus on maintaining the underlying infrastructure.
- Better governance: Git and OCI serves as an audit trail, allowing us to track changes and enforce compliance.
Challenges and lessons learned
While FluxCD has been a great fit for our use case, we’ve also encountered some challenges along the way:
- Managing large Repositories: As the number of tenants and applications grow, managing large Git repositories can become cumbersome. We’ve addressed this by breaking down repositories into smaller, more manageable components.
- Sensitive Information: Managing information requires careful consideration. We’ve integrated FluxCD and external management tools like HasiCorp Vault to improve processing of information.
- Onboarding tenants: Educating tenants on GitOps practices and Kubernetes concepts has been an ongoing effort. We’ve invested in training and documentation to help tenants get up to speed.
Looking ahead
As we continue to scale our Kubernetes platform, we’re exploring new ways to enhance our deployment process. Some of the areas we’re focusing on include:
- Advanced GitOps patterns: Implementing canary deployments, blue-green deployments, and other advanced deployment strategies.
- Policy enforcement: Using tools like Open Policy Agent (OPA) to enforce policies across tenants.
- Observability: Enhancing our monitoring and alerting capabilities to provide better visibility into the health of our applications.
In a highly competitive environment, we will continue to innovate and are excited to see how these technologies evolve.