Summary

In early September 2020, the client shared a concept called “Datamesh” with the Google project team and Google Cloud Product Leadership, which initiated discussions that established a new product partnership.

This partnership evolved as Google Cloud allocated a budget and formed a new team to deliver the client’s vision by leveraging their standards. Within a matter of months, the client’s interest and the project team's momentum led to the empowerment of multiple strategic customers, generating additional sponsorship for what is now known as “Dataplex.”

This product has since been released in general availability (GA) and is published on the Google Cloud website. Many features in Dataplex exist today because the client’s project team effectively envisioned the 10x potential of Datamesh, collaborated with GCP product leadership, and mobilized internal resources to push the client’s product design requirements. Their responsive, humble, and customer-centric approach enabled them to outpace AWS in delivering these capabilities.

Customer Challenge

Like many organizations leveraging cloud technology, the client faced the challenge of having “too much data, not enough organization and oversight.” With over 500 petabytes of data in BigQuery and more than 300 petabytes in Google Cloud Storage (GCS), the client possessed one of the largest data universes on Google Cloud, which continues to grow rapidly. The volume of labor required to manage this data, ensure its security, and maintain compliance from a governance standpoint became overwhelming.

To address these challenges, the client began designing what they called “Datamesh” in Q2-Q3 2020. This concept aimed to create a portable data fabric solution that would catalog and intelligently surface essential information about their total data universe, such as who is using it, how often, and for what purpose, down to the metadata level. Initially, the client envisioned Datamesh as a means to inventory metadata, images, text messages, define metrics, and track jobs, as leadership often encountered discrepancies when asking data science questions, receiving multiple answers that highlighted operational and governance gaps across products within Google Cloud and third-party cloud providers.

Additionally, the client sought to future-proof their existing data architecture and mitigate operational challenges in a multi-cloud environment. They wanted to understand the lifecycle of their data—where it originates (which service and cloud), who accesses it (user tracking), its journey in BigQuery (end-to-end tracing), and the value it generates for the company (with every data job needing trackable KPIs). To take their Datamesh concept to the next level, they requested Google build a new product leveraging Google’s engineering expertise.

Project Deliverables

The project team led product scoping and design efforts, performing a variety of responsibilities beyond typical functions. This summarizes the first phase of the project, with the next phase focused on onboarding, ramping, product testing, and in-production implementation delivery.

1. Lead Product Vision

   • Brainstormed potential product ideas based on customer needs, market trends, or new technologies.

   • Conducted research to assess market demand, competition, feasibility, and alignment with company goals.

   • Product concept was defined, including target audience, unique features, and positioning.

   • Roadmap was developed, timelines were set, and resources were allocated.

2. Scheduled Regular Check-ins with Project Management and the Client

   • Set realistic expectations with both the project manager and the client.

   • Led weekly meetings with product leadership.

   • Led monthly meetings with senior product leadership.

   • Led quarterly meetings with executive product leadership.

3. Introduced Initial Concepts and Enablement Strategy to GTM Team

   • Initial designs were created, and prototypes were built for testing and validation.

   • The product prototype was tested with potential users to gather feedback and identify issues.

   • Marketing, sales, and support strategies were planned; documentation and training were finalized.

4. Monitored Ongoing Engagement of Engineering, Product, and Other Customers

   • The product was released to the market, and early feedback was monitored.

   • Ongoing customer feedback was gathered, performance was monitored, and improvements and new features were iterated.

Impact

The Google project team successfully collected feature requests and technical requirements, significantly influencing Google Cloud’s product roadmap and driving a deeper partnership between Google and the client. Within the first couple of months, Google Cloud's project team provided enough momentum and strategic input to expand the design from a “client-only” focus to a broader concept that is now publicly available, known as “Dataplex.”

1. The client’s usage of BigQuery increased from 300PB to 500PB in 6 months due to the Datamesh/Dataplex initiative.

2. The client released their project-team scorecard, highlighting excitement for Dataplex, which has become a focal point for their $3-4B contract renewal.

3. The client’s business was positioned to scale seamlessly and effectively to the Zettabyte range, addressing many previously identified challenges.