data
-
Creating Identity-Aware Views for Conditional Data Access
POST_START Creating Identity-Aware Views for Conditional Data Access I recently had the task of implementing a secure data access pattern for customer information in our sales database. The goal was to create a view that would expose customer data conditionally based on the user’s identity. This way, only users who are part of the ‘pii_readers’…
A1, Access, category:business, category:education, category:software, category:technology, coding, Conditional, creating, data, data engineering, data governance, databricks, education, identity-aware, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, Views -
Upserting Data with Delta MERGE
POST_START Upserting Data with Delta MERGE Introduction I recently needed to synchronize customer data between two tables in my Delta Lake environment. One table, called customers, was the source of truth, and the other, customer_updates, contained new or modified records. The challenge was to efficiently update existing records and insert new ones in a single…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, Delta, education, Lakehouse, merge, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, upserting -
Loading Table Data from Queries with INSERT SELECT
POST_START Loading Table Data from Queries with INSERT SELECT I recently needed to extract a subset of customer data from our sales database for a specific analysis project. The data I needed was all customers located in Germany. I decided to use the Unity Catalog feature in Databricks to load this data into a new…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, insert, Lakehouse, loading, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, queries, select, sql, table, unity catalog -
Designing Region-Based Row-Level Security for Shared Production Data
POST_START Designing Region-Based Row-Level Security for Shared Production Data I recently took on the responsibility of securing access to a shared production dataset in Databricks. The dataset, production.sales.regional_orders, contains sensitive customer information, and I needed to ensure that users only saw data relevant to their region. The goal was to implement region-based row-level security using…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, designing, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, region-based, row-level, security, shared, sql, unity catalog -
Restricting Finance Data Access to an Approved Production Group
POST_START Restricting Finance Data Access to an Approved Production Group I was recently asked to help secure access to sensitive finance data in our Databricks environment. The goal was to ensure that only a specific group of finance analysts could access certain data in the production catalog, while maintaining strict access controls to prevent unauthorized…
A1, Access, approved, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, finance, group, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, restricting, sql, unity catalog -
Designing Service Principal Access for Production Data Pipelines
POST_START Designing Service Principal Access for Production Data Pipelines I recently took on the task of securing access for our ETL pipelines to a production data catalog in Databricks. The goal was to ensure that the service principals used by our data pipelines had the right level of access without exposing sensitive data unnecessarily. I…
A1, Access, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, designing, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, Pipelines., principal, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, service, sql, unity catalog -
Enforcing Separation of Duties for Sensitive Production Data with Unity Catalog Privileges
POST_START Enforcing Separation of Duties for Sensitive Production Data with Unity Catalog Privileges I recently took on the responsibility of securing access to a critical dataset in our production environment. The dataset, called customer_master, contains sensitive customer information that must be handled with care. My task was to enforce a separation of duties model using…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, duties, education, enforcing, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sensitive, separation, sql, unity catalog
