Simplifying Complexity: Using Dapr to Build a More Maintainable Software Ecosystem
DataGalaxy is a Data & AI governance platform built for the speed of modern business. It helps organizations align strategy, technology, and execution to drive data-driven success.
Learn more: https://www.datagalaxy.com/en/
Challenge
Our organization was facing significant challenges with our existing legacy codebase. Over time, the code had become increasingly difficult to maintain, hindering our ability to deliver value to clients. We needed a fresh start with a modernized, structured approach that would enable us to scale seamlessly. To address this, we decided to migrate our project to a modular monolith architecture, allowing us to maintain consistency while enabling gradual evolution toward a more service-oriented model.
Beyond the architectural overhaul, we also sought to enhance our analytics capabilities. Our existing tracking mechanisms were insufficient to provide valuable insights into feature usage and platform state. Without a reliable way to collect, store, and analyze data efficiently, decision-making was often based on incomplete or outdated information. We needed a more dynamic and event-driven approach to analytics.
Finally, we aimed to introduce trustworthy AI-powered features. To boost adoption and automate data governance, our platform required AI-driven automation such as intelligent content generation and translation. But integrating AI in a scalable and fault-tolerant manner was a challenge. Ensuring reliability while minimizing disruptions was key to making AI a core component of our platform.
Solution
To tackle these challenges, we leveraged Dapr’s powerful capabilities, particularly Pub/Sub and Service-to-Service invocation. By incorporating Dapr, we streamlined the transition from our legacy system to a modern modular monolith. Dapr abstracted away the complexity of building resilient and scalable services, allowing our developers to focus on business logic rather than infrastructure concerns. This drastically reduced the effort needed to implement robust communication patterns, making our migration smoother and more efficient.
For analytics, we adopted an event-driven approach powered by Dapr’s Pub/Sub. We now publish structured JSON event payloads into our data lake, where our data team can easily consume and process them. This has empowered us to generate in-depth analytics dashboards, providing real-time insights into platform usage and user behavior. The ease of integration and scalability of this solution has transformed how we monitor and optimize our product.
Incorporating AI into our platform was also made seamless with Dapr. We use Pub/Sub to send data requiring AI processing – such as translation tasks or content generation – to our central AI cluster. The platform then return results asynchronously, ensuring fault tolerance and resiliency. This approach has allowed us to integrate AI-driven features without compromising system stability or performance, making AI a reliable and scalable component of our offering.
Impact
Being a leader in data governance and dealing with millions and millions of metadata objects, DataGalaxy needed a trustful solution for the scale up in analytics and AI – as these capacities begin to be the drivers of user adoption and data governance support.
By adopting a standardized modular monolith architecture with Dapr, our development teams have experienced a significant boost in productivity. Developers can now focus on feature development without being bogged down by legacy constraints. The streamlined architecture has improved maintainability, making it easier to onboard new team members and accelerate development cycles.
The shift to event-driven analytics has provided us with deeper insights into platform usage. With more precise feature tracking, we can make data-driven decisions that align with market trends and user needs. This has allowed us to continuously refine and optimize our platform, ensuring we stay competitive in an ever-evolving industry.
Finally, integrating AI through resilient asynchronous communication has enhanced our platform’s capabilities while reducing operational risks. AI-driven automation has improved efficiency and reliability, allowing us to deliver smarter and more automated features. As a result, our platform is now better positioned in the market, offering cutting-edge AI capabilities while maintaining high availability and stability.
Projects used
By the numbers
~25 Million Messages
processed monthly throughout the year
2 Months
from proof of concept to full production deployment
130+ Clusters
Running Dapr in production
Dapr empowered us to modernize our architecture, streamline analytics, and integrate scalable AI capabilities – effortlessly and reliably – by abstracting complex infrastructure into simple, composable building blocks.
Further details
To provide a deeper look under the hood, here’s a technical overview of our platform’s architecture and key integrations. Built on modern cloud-native principles, we leverage .NET 9 with the Dapr .NET SDK for service-to-service and pub/sub communications, ensuring scalable and resilient interactions. Our messaging backbone combines RabbitMQ for internal module coordination and Kafka for handling communication with our centralized AI cluster. The entire system runs on Kubernetes, delivering robust container orchestration and deployment. For analytics, we ingest event data into an Azure Databricks data lake and visualize insights with PowerBI dashboards. Below, we explain how these technologies come together across our modular monolith, event-driven pipeline, and asynchronous AI integration.
Modular Monolith Architecture with Dapr
Our existing monolithic codebase had, over time, developed significant component interdependencies. This complexity meant that deploying new features became an increasingly challenging and high-risk endeavor. Our primary objective was to attain the clear component isolation and separation of concerns typically associated with microservices, but without incurring the substantial operational overhead inherent in managing fully distributed systems. The Modular Monolith architecture emerged as the optimal strategy to achieve this balance.
To facilitate this transition and empower our new architecture, we leveraged key capabilities within Dapr.
Leveraging Dapr for Asynchronous Communication via Pub/Sub
The adoption of Dapr’s publish-subscribe building block was instrumental in enabling an event-driven architecture within our modular monolith. This allows distinct domains to publish significant business events, with other modules subscribing and processing these events asynchronously. Dapr abstracts the complexities of reliable message delivery, utilizing RabbitMQ as the configured message broker in our deployment. This approach effectively eliminated rigid, synchronous dependencies for many workflows, resulting in a system with highly decoupled modules, improved resilience, enhanced scalability in event processing, and the critical ability for individual modules to evolve independently.
Enhancing Inter-Module Communication with Dapr Service Invocation
Within our modular monolith design, Dapr’s service-to-service invocation building block enables each domain to expose a well-defined and independently addressable API. This capability not only streamlines communication between the newly defined modules but also provides secure and reliable pathways for interaction with existing legacy components or internal “toolboxes.” By abstracting direct inter-module dependencies, Dapr’s service invocation contributes to a cleaner, more decoupled codebase. The outcome is significantly improved system maintainability and a more robust, evolvable architecture prepared for future requirements.
Event-Driven Analytics Pipeline
To address the lack of observability, we introduced an event-driven pipeline. Every module now emits events (e.g., user interaction, job execution, errors) via Dapr Pub/Sub. These events are now captured as streamlined, lightweight JSON messages and ingested into a centralized data lake. From there, we built real-time dashboards that surface insights for both operational monitoring and product decision-making.
Asynchronous AI Integration
Instead of synchronous calls to AI services – which impacts system reliability – we designed an asynchronous pattern using Dapr. The system pushes tasks such as translation and content generation to an AI queue. Once processed, the system returns results through a dedicated channel, ensuring loose coupling and built-in fault tolerance.
