POST_START

Protecting Sensitive Production Columns with Unity Catalog Column Masks

Setting the Stage

I recently joined a team that’s responsible for maintaining a data warehouse built on Databricks. One of the key responsibilities I inherited was ensuring that sensitive data, especially in production environments, is properly protected. The team had a number of tables with personal information, including email addresses, and they were looking for a way to mask these columns without altering the underlying data or breaking existing queries.

I learned that Databricks Unity Catalog provides a powerful feature called column masks, which allows you to define rules for how data is displayed based on user roles. This means that sensitive data can be hidden from certain users while still being accessible to others who need it. The goal was to mask email addresses in the customer master table, but only for users who are not part of the PII readers group.

Creating the Masking Function

I started by writing a simple function that would apply the masking logic. The function needed to check if the user belongs to the ‘pii_readers’ account group, and if so, return the actual email address. Otherwise, it should return a masked value.

CREATE OR REPLACE FUNCTION production.security.email_mask(email STRING) 
RETURN CASE 
  WHEN is_account_group_member('pii_readers') THEN email 
  ELSE '***MASKED***' 
END;

I chose to place the function in the ‘production.security’ schema to keep it organized and to follow the team’s naming conventions. This function would be used as a column mask, so it needed to be simple and efficient. I verified that the function would return the correct values based on the user’s group membership, which is a built-in function in Unity Catalog.

Applying the Mask to the Column

Once the function was in place, I moved on to applying it to the email column in the customer master table. The team had a table called ‘customer_master’ in the ‘production.customers’ schema, and they wanted the email column to be masked for non-PII readers.

ALTER TABLE production.customers.customer_master 
ALTER COLUMN email SET MASK production.security.email_mask;

I ran this command in the Databricks notebook, making sure to use the correct schema and table names. The ALTER TABLE command is used to apply the mask to the column. I noticed that the operation was quick and didn’t require any data movement, which is a big plus for production environments.

As the command executed, I monitored the logs to ensure there were no errors. The system confirmed that the mask was successfully applied to the ’email’ column. This means that any future queries against this column would automatically use the mask function based on the user’s group membership.

Verifying the Masking Works

To ensure everything was working as expected, I ran a SELECT query to retrieve some sample data from the customer master table.

SELECT * FROM production.customers.customer_master LIMIT 20;

I chose a small number of rows to inspect, and I checked the output to see if the email column was masked correctly. I logged in as a regular user who is not part of the ‘pii_readers’ group and confirmed that the email addresses were displayed as ‘***MASKED***’. Then, I switched to an account that is a member of the ‘pii_readers’ group and verified that the actual email addresses were visible.

This step was crucial because it confirmed that the masking logic was working as intended. It also gave me confidence that the column mask was correctly applied and that the data was being protected according to the team’s security policies.

Wrapping Up

Through this process, I learned how to use Unity Catalog column masks to protect sensitive data in production environments. By creating a simple function and applying it to the relevant column, I was able to ensure that email addresses were masked for users who shouldn’t have access to them. This approach is both efficient and secure, and it aligns with the team’s data governance practices.

I now have a better understanding of how Unity Catalog can be used to enforce data privacy and security in a data warehouse. This experience has also made me more confident in working with Databricks and implementing similar protections for other sensitive columns in the future.