POST_START

Comparing Current and Historical Production Data After a Suspected Bad Write

Today, I encountered an issue with the production data in our Databricks Unity Catalog. A team member reported that a recent write operation might have caused some data to be incorrect. My task was to investigate and compare the current data with a historical version to determine if there was any change that could have introduced the problem.

Understanding the History of the Table

First, I wanted to understand what changes had been made to the production.sales.orders table over time. I ran the DESCRIBE HISTORY command to see the version history of the table.

DESCRIBE HISTORY production.sales.orders;

version	timestamp	userName	operation
12	2026-09-11 14:32:10	analyst@demo.com	WRITE
11	2026-09-11 13:18:42	engineer@demo.com	MERGE
10	2026-09-11 09:05:17	admin@demo.com	CREATE TABLE

From the output, I saw that the latest version was 12, created by the analyst at 14:32. The previous version was 11, created by the engineer at 13:18. Version 10 was the initial creation of the table. This helped me identify the timeline of operations and the users involved.

Reviewing the Historical Data

Next, I wanted to look at the data from version 10 to see what it looked like before the most recent changes. I ran a query to retrieve the data from that version.

SELECT * FROM production.sales.orders VERSION AS OF 10;

order_id	customer_id	region	amount	status
50001	2001	EMEA	1250.00	COMPLETE
50002	2002	AMER	890.50	COMPLETE
50003	2003	EMEA	430.25	PENDING

I noted the data from version 10, which included three orders with specific customer IDs, regions, amounts, and statuses. This was the baseline data before the more recent operations.

Checking the Current Data

To compare, I ran the same query on the current version of the table to see if any changes had occurred.

SELECT * FROM production.sales.orders;

order_id	customer_id	region	amount	status
50001	2001	EMEA	1250.00	COMPLETE
50002	2002	AMER	890.50	COMPLETE
50003	2003	EMEA	430.25	PENDING

Interestingly, the current data was identical to the data from version 10. This suggested that the recent write operation might not have affected the data, or that the changes were not visible in the query results. However, the timestamp of the latest version (version 12) was after the version 10 data, so there must have been some change in the metadata or structure of the table.

Conclusion and Next Steps

Although the data appeared the same, I knew that the version history indicated a recent write. I planned to dig deeper by checking the exact changes in version 12. I also considered the possibility that the write operation might have been a merge or update that didn’t affect the data I was querying. I would need to review the exact changes made in version 12 and the user who performed the operation to fully understand the impact of the write.