Introducing Databricks Unity Catalog: Fine-grained Governance for Data and AI on the Lakehouse

Data lake systems such as S3, ADLS, and GCS store the majority of data in today’s enterprises thanks to their scalability, low cost, and open interfaces. Over time, these systems have also become an attractive place to process data thanks tolakehousetechnologies such asDelta Lakethat enable ACID transactions and fast queries. However, one area where data lakes have remained harder to manage than traditional databases is governance; so far, these systems have only offered tools to manage permissions at the file level (e.g. S3 and ADLS ACLs), using cloud-specific concepts like IAM roles that are unfamiliar to most data professionals.

That’s why we’re thrilled to announce ourUnity Catalog, which brings fine-grained governance and security to lakehouse data using a familiar, open interface. Unity Catalog lets organizations manage fine-grained data permissions using standard ANSI SQL or a simple UI, enabling them to safely open their lakehouse for broad internal consumption. It works uniformly across clouds and data types. Finally, it goes beyond managing tables to govern other types of data assets, such as ML models and files. Thus, enterprises get a simple way to govern all their data and AI assets:

What’s hard with data lake governance tools today?

Although all cloud storage systems (e.g. S3, ADLS and GCS) offer security controls today, these tools arefile-orientedandcloud-specific, both of which cause problems as organizations scale up. We’ve often seen customers run into four problems:

• Lack of fine-grained (row, column and view level) security:Cloud data lakes can generally only set permissions at the file or directory level, making it hard to share just a subset of a table with particular users. This makes it tedious to onboard enterprise users who should not have access to the whole table.
• Governance tied to physical data layout:因为治理控制是在文件级别,data teams must carefully structure their data layout to support the desired policies. For example, a team might partition data into different directories by country and give access to each directory to different groups. But what should the team do when governance rules change? If different states inside one country adopt different data regulations, the organization may need to restructure all its data.
• Nonstandard, cloud-specific interfaces:Cloud governance APIs such as IAM are unfamiliar to data professionals (e.g., database administrators), and different across clouds. Today, enterprises increasingly have to store data in multiple clouds, (e.g., to satisfy privacy regulations), so they need to be able to manage data across clouds.
• No support for other asset types:Data lake governance APIs work for files in the lake, but modern enterprise workflows produce a wide range of other types of data assets. For example, SQL workflows often revolve around views, data science workloads produce ML models, and many workloads connect to data sources other than the lake (e.g., databases). In the modern compliance landscape, all of these assets need to be governed the same way if they contain sensitive data. Thus, data teams have to reimplement the same security policies in many different systems.

Unity Catalog's approach

Unity Catalog solves these problems by implementing a fine-grained approach to data governance based on open standards that works across data asset types and clouds. It is designed around four key principles:

• Fine-grained permissions:Unity Catalog can enforce permissions for data at the row, column or view level instead of the file level, so that you can always share just part of your data with a new user without copying it.
• An open, standard interface:Unity Catalog’s permission model is based on ANSI SQL, making it instantly familiar to any database professional. We’ve also built a UI to make governance easy for data stewards, and we’ve extended the SQL model to support attribute-based access control, allowing you to tag many objects with the same attribute (e.g., “PII data”) and apply one policy to all of them. Finally, the same SQL based interface can be used to manage ML models and external data sources.
• Central control:Unity Catalog can work across multiple Databricks workspaces, geographic regions and clouds, allowing you to manage all enterprise data centrally. This central position also enables it to track lineage and audit all accesses.
• Secure access from any platform:虽然我们爱砖平台,我们知道bob体育客户端下载that many customers will also access the data from other platforms and that they’d like their governance rules to work across them. Unity Catalog enforces security permissions from any client that connects through JDBC/ODBC or throughDelta Sharing, the open protocol we’ve launched to exchange large datasets between a wide range of platforms.

Let’s look at how the Unity Catalog can be used to implement common governance tasks.

Easily manage permissions using ANSI SQL

Unity Catalog brings fine-grained centralized governance to all data assets across clouds through the open standard ANSI SQL Data Control Language (DCL). This means administrators can easily grant permission to arbitrary user-specific subsets of the data using familiar SQL -- no need to learn an arcane, cloud-specific interface. We’ve also added a powerful tagging feature that lets you control access to multiple data items at once based on attributes to further simplify governance at scale.

Below are a few examples of how you can use SQL grant statements with the Unity Catalog to add permissions to existing data stored on your data lake.

First, you can create tables in the catalog either from scratch or by pointing to existing data in a cloud storage system, such as S3, accessed with cloud-specific credentials:

CREATEEXTERNALTABLEiot_events LOCATION s3:/...WITHCREDENTIAL iot_iam_role

You can now simply use SQL standardGRANTstatements to set permissions, as in any database. Below is an example of how to grant permissions to iot_events to an entire group such as engineers, or to just the date and country columns to the marketing group:

GRANTSELECTONiot_eventsTOengineersGRANTSELECT(date, country)ONiot_eventsTOmarketing

The Unity Catalog also understands SQL views. This allows you to create SQL views to aggregate data in a complex way. Here is how you can useView-Based Access Controlto grant access to only an aggregate version of the data for business_analysts:

CREATEVIEWaggregate_dataASSELECTdate, country,COUNT(*)ASnum_eventsFROMiot_eventsGROUPBYdate, countryGRANTSELECTONaggregate_dataTObusiness_analysts

In addition, the Unity Catalog allows you to set policies across many items at once using attributes (Attribute-Based Access Control), a powerful way to simplify governance at scale. For example, you can tag multiple columns as PII and manage access to all columns tagged as PII in a single rule:

ALTERTABLEiot_eventsADDATTRIBUTE piiONemailALTERTABLEusersADDATTRIBUTE piiONphoneGRANTSELECTONDATABASE iot_dataHAVINGATTRIBUTENOTIN(pii)TOproduct_managers

Finally, the same attribute system lets you easily govern MLflow models and other objects in a consistent way with your raw data:

GRANTEXECUTEONMODELSHAVINGATTRIBUTE (eu_data)TOeu_product_managers

Discover and govern data assets in the UI

Unity Catalog's UI makes it easy to discover, describe, audit and govern data assets in one place. Data stewards can set or review all permissions visually, and the catalog captures audit and lineage information that shows you how each data asset was produced and accessed. The UI is designed for collaboration so that data users can document each asset and see who uses it.

Share data across organizations with Delta Sharing

Every organization needs to share data with customers, partners and suppliers to collaborate. Unity Catalog implements the open sourceDelta Sharingstandard to let you securely share data across organizations, regardless of which computing platform or cloud they run on (any Delta Sharing client can connect to the data).

Open interfaces for easy access

Unity Catalog works with your existing catalogs, data, storage and computing systems so you can leverage your existing investments and build a future-proof governance model. It can mount existing data inApache HiveMetastores or cloud storage systems such as S3, ADLS and GCS without moving it. It also connects with governance platforms like Privacera and Immuta to let you define custom workflows for managing access to data. Finally, we designed Unity Catalog so that you can also access it from computing platforms other than Databricks: ODBC/JDBC interfaces and high-throughput access viaDelta Sharingallow you to securely query your data any computing system.