sql
-
Recovering a Production Delta Table to a Known Good Version
POST_START Recovering a Production Delta Table to a Known Good Version Today, I encountered an issue with a production Delta table that had been accidentally modified. The table in question was production.sales.orders, and the latest version contained incorrect data that was causing downstream processes to fail. I needed to recover the table to a known…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, Delta, education, good, known, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, recovering, sql, table, unity catalog, version -
Comparing Current and Historical Production Data After a Suspected Bad Write
POST_START Comparing Current and Historical Production Data After a Suspected Bad Write Yesterday, I noticed some unusual behavior in our production data pipeline. A new batch of data was written to the production.sales.orders table, but when I checked the latest records, it looked like some entries were missing or had incorrect values. I suspected a…
A1, after, bad, category:business, category:education, category:software, category:technology, coding, Comparing, current, data, data engineering, data governance, databricks, education, historical, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, suspected, unity catalog -
Protecting Sensitive Columns with Column Masks
POST_START Protecting Sensitive Columns with Column Masks I recently needed to ensure that sensitive data, like customer emails, was only accessible to specific users in our Databricks environment. I decided to use column masks to control access to the email column in the customers table. This approach allows me to hide the email addresses from…
A1, category:business, category:education, category:software, category:technology, coding, column, columns, data engineering, data governance, databricks, education, Lakehouse, masks, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, protecting, sensitive, sql, unity catalog -
Removing Column Masks from Protected Columns
POST_START Removing Column Masks from Protected Columns Understanding the Need to Remove Column Masks I recently took on the responsibility of managing sensitive customer data in our Databricks environment. One of the key tasks was to ensure that data access adhered to strict privacy regulations. As part of this, certain columns, like the customer email,…
A1, category:business, category:education, category:software, category:technology, coding, column, columns, data engineering, data governance, databricks, education, Lakehouse, masks, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, protected, removing, sql, unity catalog
