troubleshooting
-
🚨 The Dashboard Numbers Were Wrong — I Traced the Databricks Pipeline with My Team
🚨 The Dashboard Numbers Were Wrong — I Traced the Databricks Pipeline with My Team My morning started with a message from Maya, one of our business analysts: “The revenue dashboard is showing yesterday’s numbers.” A few seconds later, Daniel from the product team added another problem. Several orders visible in the operational application were…
A1, A2, auto-loader, B1, B2, C1, C2, category:databricks, cdc, chat, conversational, data engineering, data-freshness, data-quality, databricks, delta lake, difficulty:240, incident-response, production-pipelines, profession:data engineer, structured-streaming, team-conversation, troubleshooting, unity catalog, Workflows. -
🔎 I Investigate Databricks Production Incidents
A successful data pipeline can still hide a serious problem. I learned this after working through production-style Databricks incidents where a job could finish successfully while the resulting data was stale, duplicated, incomplete, unexpectedly expensive, or available to the wrong audience. That changed the way I think about data engineering. I no longer begin with…
A1, A2, auto-loader, B1, B2, C1, C2, category:databricks, cdc, chat, conversational, data engineering, data-engineer-professional, data-quality, databricks, delta lake, difficulty:240, incident-response, investigation, production-pipelines, profession:data engineer, root-cause-analysis, structured-streaming, troubleshooting, unity catalog, Workflows. -
Troubleshooting Column-Level Security with Function and Grant Inspection
POST_START Troubleshooting Column-Level Security with Function and Grant Inspection I was working on a data pipeline where certain users needed access to specific columns in the customer data, but not all. I noticed that when I tried to query the email column, it wasn’t showing up in the results, even though the column existed in…
A1, category:business, category:education, category:software, category:technology, coding, column-level, data engineering, data governance, databricks, education, function, grant, inspection, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, security, sql, troubleshooting, unity catalog -
Troubleshooting a Service Principal That Lost Production Table Access
POST_START Troubleshooting a Service Principal That Lost Production Table Access Today, I encountered an issue where a service principal that was previously able to access a table in the production catalog suddenly lost access. The team was trying to run a critical ETL job, and the error message indicated that the service principal didn’t have…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, Lakehouse, Lost, paragraph ordering, paragraph structure, paragraphorder, principal, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, service, sql, table, That, troubleshooting, unity catalog -
Troubleshooting Sensitive Column Visibility and Unity Catalog Column Mask Behavior
POST_START Troubleshooting Sensitive Column Visibility and Unity Catalog Column Mask Behavior I was working on a data pipeline that involved accessing the production.customers.customer_master table. My task was to understand why certain sensitive columns weren’t showing up in the results, even though I had the necessary permissions. I wanted to troubleshoot the issue using Unity Catalog’s…
A1, behavior, category:business, category:education, category:software, category:technology, coding, column, data engineering, data governance, databricks, education, Lakehouse, mask, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sensitive, sql, troubleshooting, unity catalog, visibility -
Troubleshooting Unexpected Row-Level Access Results from Unity Catalog Row Filters
POST_START Troubleshooting Unexpected Row-Level Access Results from Unity Catalog Row Filters I was working on a query against the production.sales.customers table and noticed that some rows were not being returned as expected. I suspected that Unity Catalog row filters might be at play, but I wasn’t sure why the results didn’t align with what I…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, filters, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, results, row, row-level, sql, troubleshooting, Unexpected, unity catalog
