data
-
Validating an External Location Before Production Data Onboarding
POST_START Validating an External Location Before Production Data Onboarding I recently had the task of onboarding a new data source into our production environment. Before I could proceed, I needed to validate the external location to ensure it met our security, access, and governance standards. The first step was to check what external locations were…
A1, before, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, External, Lakehouse, location, onboarding, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, validating -
Comparing Current and Historical Production Data After a Suspected Bad Write
POST_START Comparing Current and Historical Production Data After a Suspected Bad Write Yesterday, I noticed some unusual behavior in our production data pipeline. A new batch of data was written to the production.sales.orders table, but when I checked the latest records, it looked like some entries were missing or had incorrect values. I suspected a…
A1, after, bad, category:business, category:education, category:software, category:technology, coding, Comparing, current, data, data engineering, data governance, databricks, education, historical, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, suspected, unity catalog -
Inspecting Storage Credentials for External Data Access
POST_START Inspecting Storage Credentials for External Data Access I recently needed to understand how our team is accessing external storage systems for data processing. I started by checking the list of storage credentials that are currently registered in Unity Catalog. This would give me a high-level overview of what credentials are available and who owns…
A1, Access, category:business, category:education, category:software, category:technology, coding, credentials, data, data engineering, data governance, databricks, education, External, inspecting, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, Storage, unity catalog -
Controlling Table Data Changes with MODIFY Privileges
POST_START Controlling Table Data Changes with MODIFY Privileges I recently had the task of managing access to a table in our data warehouse called training.sales.customers. The goal was to ensure that only specific teams could modify the data, while others could only read it. I learned that using the MODIFY privilege is essential for allowing…
A1, category:business, category:education, category:software, category:technology, changes, coding, controlling, data, data engineering, data governance, databricks, education, Lakehouse, modify, paragraph ordering, paragraph structure, paragraphorder, privileges, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, table, unity catalog -
Creating Identity-Aware Views for Conditional Data Access
POST_START Creating Identity-Aware Views for Conditional Data Access I recently had the task of implementing a secure data access pattern for customer information in our sales database. The goal was to create a view that would expose customer data conditionally based on the user’s identity. This way, only users who are part of the ‘pii_readers’…
A1, Access, category:business, category:education, category:software, category:technology, coding, Conditional, creating, data, data engineering, data governance, databricks, education, identity-aware, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, Views -
Upserting Data with Delta MERGE
POST_START Upserting Data with Delta MERGE Introduction I recently needed to synchronize customer data between two tables in my Delta Lake environment. One table, called customers, was the source of truth, and the other, customer_updates, contained new or modified records. The challenge was to efficiently update existing records and insert new ones in a single…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, Delta, education, Lakehouse, merge, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, upserting -
Loading Table Data from Queries with INSERT SELECT
POST_START Loading Table Data from Queries with INSERT SELECT I recently needed to extract a subset of customer data from our sales database for a specific analysis project. The data I needed was all customers located in Germany. I decided to use the Unity Catalog feature in Databricks to load this data into a new…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, insert, Lakehouse, loading, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, queries, select, sql, table, unity catalog -
Designing Region-Based Row-Level Security for Shared Production Data
POST_START Designing Region-Based Row-Level Security for Shared Production Data I recently took on the responsibility of securing access to a shared production dataset in Databricks. The dataset, production.sales.regional_orders, contains sensitive customer information, and I needed to ensure that users only saw data relevant to their region. The goal was to implement region-based row-level security using…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, designing, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, region-based, row-level, security, shared, sql, unity catalog
