Data governance¶

Who owns a dataset, what is in it, who may see it, how it got there, and how anyone knows it is right. Governance is not a tool you buy. It is a short list of habits (owners, classifications, contracts, checks) and a catalog that makes them findable.

flowchart TB
  OWN[Ownership<br/>who answers for this table?] --> CAT[Catalog<br/>find it, read its description]
  CLS[Classification<br/>PII, financial, public] --> ACC[Access control<br/>roles, masking, row filters]
  CON[Contracts<br/>producer promises a schema] --> QA[Quality checks and observability]
  LIN[Lineage<br/>where it came from, who uses it] --> CAT
  QA --> LIN
  ACC --> AUD[Audit trail]

The pieces of governance and how they depend on each other. Ownership and classification come first; the catalog, access control, quality checks and lineage are how they are enforced and discovered.

Learn¶

• Data governance and catalogs - owners, classification, access and discovery
• Data contracts - producer-consumer agreements enforced in CI
• Data quality and lineage - assertions, and tracing a number back to its source
• Data observability - freshness, volume and schema monitoring
• Backup and point-in-time restore - the recovery half of data protection
• Slowly changing dimensions - keeping history so an audit can ask "what did we know then?"

Compare¶

• Data warehouses - the access-control and masking features differ per platform
• Databases (cloud-native) - encryption, auditing and backup options per service

Reference¶

• Architecture pattern: data mesh - federated ownership, data products and computational governance
• Architecture pattern: medallion - where raw, PII-bearing data stops being visible
• Architecture pattern: semantic layer - one definition per metric
• Postmortem: GitLab database incident, 2017 - backups nobody had restored
• Postmortem: Atlassian site deletion, 2022 - a deletion script, and recovery at scale
• Compliance guides - GDPR and the rest, where lineage stops being optional
• Identity and IAM - the access-control foundations on the security site

Build¶

• Build a dbt project - tests, contracts and lineage in code
• Backup and restore drill - prove a backup restores before you need it
• Design a star schema - declared grain and conformed dimensions are governance too

Certify¶

• Databricks Data Engineer Professional - Unity Catalog permissions, masking and governance tags
• Databricks Data + AI Platform Administrator Accreditation
• Snowflake SnowPro Advanced Administrator - roles, masking and row access policies
• Snowflake SnowPro Core (COF-C03)
• GCP Professional Data Engineer - data security and governance sections
• AWS Data Engineer Associate (DEA-C01) - security and governance domain

Related topics: Data engineering for the pipelines these controls sit on, and Analytics and BI for where the governed data is consumed.