SBIR Topic Opens: Chief Digital and Artificial Intelligence Office (CDAO) Data Mesh Reference Design (REFDES)

The Department aims to establish a set of software services to allow data users across the Department of Defense (DoD) enterprise to discover DoD data products, understand their structure and meaning, seamlessly negotiate access, and consume them via self-service API. Per DoD rulemaking, data access must support attribute-based access control (ABAC) and operate in a zero-trust environment.

The Department has determined that data sharing models with a unified schema and single system of record (sometimes called a “Data Warehouse”) or even semi-unstructured data in a single system of record (“Data Lake”) are not good operational fits for the Department’s requirements. A closer analogue is a “Data Mesh” as described in Dehghani (2022).1

This program consists of three phases, described in the sections below. First, however, a few descriptions.

Peer-Cooperative Microservices

Each unique microservice is able to communicate with all other similar/identical microservices to form a specific community.

Services vs. Microservices

The rest of this document will reference services and microservices as “services”, however it recognizes the distinction between them. Larger applications, built on a single code base, typically consist of a client-side UI, a database, and a server-side application. These are Services. On the other hand microservices are built for a fully distributed system to accomplish a single feature or business logic. Instead of exchanging data within the same code base, microservices communicate with an API.

The DoD has identified 15 core functional capabilities that an enterprise data mesh at the Department must have:

1. UIDs: Tools to describe how data transforms and flows as it is transported from source to destination across the entire data lifecycle. Data versioning for tracking data and models as they change. [A prototype of this is available, accompanied by a whitepaper describing its recommended structure]
2. Semantic Services: Tools to promote sharing, collaboration, and reuse of data models and ontologies; alias re-referencing to build a canonical controlled vocabulary. [A prototype of this is available, accompanied by a whitepaper describing its recommended structure]
3. Federated Data Catalog: Virtually federated catalog enabling defense-wide visibility of data and interfaces through pointers to DOD assets and services. [Multiple instantiations exist]…

More here.

🔊 Listen to Story