Charles is a software engineer currently working on a data transformation program at Springer Nature. He has experience building distributed systems and micro-service software platforms as an academic and as a consultant, independently and with companies such as ThoughtWorks and EqualExperts.
More about Charles Kubicek...
Charles Kubicek is talking about
Building a data mesh at springer nature
Digital transformation increasingly relies on diverse datasets to drive complex features and customer interactions. Organisations must put datasets to work in various user-facing settings far beyond their intended use case, which requires data to be reliably and continually shared instead of the “throw it over the wall and hope” data lake model.
Data Mesh is an architectural model that combines the ideas of lean/agile product delivery, such as domain orientation, APIs, product ownership and self-serve platforms, and applies them to the analytical data space to enable deliberate and trustworthy data sharing.
In this case study, I’ll tell you how we applied our agile tools and mindset to the data space. I’ll introduce a real-life use case we delivered to drive the change and how the data-as-a-product concept enabled teams to begin delivering data fit for numerous use cases. Topics I’ll cover include:
- How we established a data engineering discipline based on XP practices such as TDD and pair programming, applied to data tools and languages
- What we needed in our cloud-based minimum-viable data platform
- How we are working to improve trust in data
- Our approach to turning poorly managed data in entrenched vendor systems into in-demand data products
Key takeaways…
- Become familiar with implementing data mesh through use cases
- Have a solution for organising and connecting data for unforeseen use cases, such as forecasting, timely and valuable user communications, and machine learning.
- Apply familiar lean/agile methodologies to analytical data
- Decrease the divide between operational and analytical data