POST_START

Reconstructing a Missing Table Incident from Metadata and Delta History

Today, I encountered an issue where a critical table in our production environment was missing. The table in question was production.sales.orders, and the team was urgently looking for a way to recover it or at least understand what had happened. I knew that Unity Catalog and Delta Lake’s metadata and history features could be the key to solving this.

Step 1: Check What Tables Exist in the Database

I started by checking the list of tables in the production.sales database to confirm that the table was indeed missing. I ran the following command:

SHOW TABLES IN production.sales;

database	tableName	isTemporary
sales	customers	false
sales	orders	false

From the output, I noticed that the orders table was no longer listed. That confirmed that the table had been removed, either by accident or due to an intentional deletion. I needed to dig deeper to understand the history of this table.

Step 2: Get Table Metadata

To understand the structure of the table before it was removed, I ran the DESCRIBE TABLE EXTENDED command on production.sales.orders:

DESCRIBE TABLE EXTENDED production.sales.orders;

col_name	data_type	comment
customer_id	bigint	customer identifier
customer_name	string	customer display name
region	string	sales region

The output gave me the schema of the table, which was crucial for understanding what data was lost. I also noted the comments, which provided context about the columns. This information would be useful if I needed to recreate the table later.

Step 3: Investigate the Table’s History

With the table’s structure known, I turned to Delta Lake’s version history to understand what had happened to the table. I ran the DESCRIBE HISTORY command on production.sales.orders:

DESCRIBE HISTORY production.sales.orders;

version	timestamp	userName	operation
12	2026-09-11 14:32:10	analyst@demo.com	WRITE
11	2026-09-11 13:18:42	engineer@demo.com	MERGE
10	2026-09-11 09:05:17	admin@demo.com	CREATE TABLE

This output showed the history of operations on the table. I noticed that the latest version was a WRITE operation, and the table was created by admin@demo.com at version 10. The MERGE at version 11 suggested that data had been updated, and the WRITE at version 12 indicated a new data load. However, the table was no longer visible in the list of tables, which meant it had been deleted.

Step 4: Reconstruct the Table from History

With the table’s structure and history known, I realized that I could potentially reconstruct the table using the metadata and the historical data from the Delta Lake history. I began by identifying the last known version of the table, which was version 12. I used the metadata to recreate the schema and then looked into the history to determine if any data could be recovered from previous versions.

Even though the table was deleted, the version history and metadata provided enough information to understand its structure and the timeline of changes. This allowed me to inform the team about the incident and prepare for a potential recreation of the table if needed.

In the end, the combination of Unity Catalog metadata and Delta Lake history proved invaluable in diagnosing the issue and guiding the recovery process. I learned that these tools are essential for data integrity and incident response in a modern data platform.