POST_START

Identifying Unexpected Unity Catalog Ownership Changes in Production

I was working on a routine performance optimization task in our production environment when I noticed something unusual. One of the tables we rely on for critical reporting, production.sales.orders, was behaving unexpectedly. Queries were timing out, and some users were reporting access issues. I suspected that the problem might be related to changes in ownership or permissions, so I decided to investigate using the Unity Catalog tools available in Databricks.

Checking Current Permissions

I started by running the SHOW GRANTS ON TABLE command to see what permissions were currently assigned to the table. This is a standard way to verify who has access to a specific resource and what actions they can perform.

SHOW GRANTS ON TABLE production.sales.orders;

As the results came in, I noticed that the table was no longer owned by the expected service account. Instead, it had been reassigned to a different team’s user account. This was a red flag because the table is used across multiple departments, and any change in ownership could lead to access issues or unintended modifications.

Looking for Audit Logs

To understand when and why this change occurred, I turned to the audit logs. I ran the following query to look for any recent activity related to the production.sales.orders table:

SELECT * FROM system.access.audit WHERE request_params LIKE '%production.sales.orders%' ORDER BY event_time DESC;

The audit logs showed that the ownership change had happened just a few days ago. It appeared that a team member had manually changed the owner without proper approval or documentation. This was a significant oversight because it could have introduced security risks and disrupted workflows.

Verifying the Impact

I cross-checked the audit logs with the current grant permissions to ensure that the ownership change was the root cause of the access issues. I also reviewed the access patterns of the users who were affected. The new owner had different access controls, and some users were no longer able to query the table due to permission mismatches.

By correlating the audit events with the current access rights, I was able to identify the exact point in time when the ownership was changed and who had initiated the change. This information was crucial for the team to understand what went wrong and how to prevent similar incidents in the future.

Conclusion and Next Steps

Through a combination of checking current permissions and reviewing audit logs, I was able to trace the unexpected ownership change in the Unity Catalog and understand its impact on the production environment. This experience reinforced the importance of monitoring access changes and maintaining a clear audit trail for all operations in a production system.

Going forward, I plan to implement automated alerts for any changes to critical assets and ensure that all teams are trained on the proper procedures for modifying Unity Catalog resources. This will help prevent similar issues and maintain the integrity of our data governance processes.