POST_START

Classifying Sensitive Columns with Governance Tags

I recently took on a new project at work that required me to improve data governance for a customer database. One of the key tasks was to classify sensitive columns, such as email addresses, so they could be properly managed and protected. I decided to use Databricks Unity Catalog to apply governance tags to these columns, which would help our team understand and enforce data privacy policies.

Identifying the Sensitive Column

Before I could assign any tags, I needed to locate the column that contained sensitive information. I checked the schema of the `training.sales.customers` table and identified the `email` column as a potential candidate for classification. This column stores customer email addresses, which are personal and require special handling.

I verified that the column exists and is accessible by running a simple query to confirm its presence. This step was important to ensure I was working with the right data and to avoid any accidental misclassification.

Applying the PII Tag

Once I confirmed the column was correct, I decided to apply the `pii` (Personally Identifiable Information) tag to it. This tag is used to mark data that can be used to identify an individual, which is critical for compliance and data security.

SET TAG ON COLUMN training.sales.customers.email `pii` = `true`;

I noticed that the command executed without any errors, and the system confirmed that the tag was applied. This was a clear indicator that the change was successful and that the column was now marked as PII in the Unity Catalog.

By applying this tag, I ensured that any future access or usage of this column would be subject to the appropriate governance policies. This helps protect customer data and ensures that the team is aware of the sensitivity of the information stored in this column.

Verifying the Tag Application

To double-check that the tag was applied correctly, I ran a query to inspect the metadata of the column. This allowed me to confirm that the `pii` tag was indeed set to `true` for the `email` column. This step was crucial to ensure that the data classification was accurate and that I could trust the system to enforce the correct policies.

I also reviewed the Unity Catalog UI to visually confirm the tag was present. This provided a clear and intuitive way to understand the data governance state of the table and gave me confidence that the classification was correctly implemented.

Removing the Tag (if Necessary)

Later, I considered a scenario where the `email` column might no longer need to be classified as PII. For example, if the data was anonymized or if the column was no longer in use. In that case, I would need to remove the tag.

UNSET TAG ON COLUMN training.sales.customers.email `pii`;

I noticed that the command executed successfully, and the system confirmed that the tag was removed. This was an important step to ensure that the data governance policies remained accurate and up to date.

By using these commands, I was able to both apply and remove the `pii` tag on the `email` column, demonstrating how Unity Catalog supports dynamic data governance. This flexibility is essential for maintaining compliance and adapting to changing data requirements.

Conclusion

Through this process, I learned how to effectively use Databricks Unity Catalog to classify sensitive columns with governance tags. Applying and removing tags is a straightforward yet powerful way to manage data privacy and ensure that the right policies are enforced. This experience has given me a deeper understanding of how data governance works in practice and how it supports compliance and security in real-world scenarios.