data engineering
-
Auditing Security Function Ownership Used by Column Masks
POST_START Auditing Security Function Ownership Used by Column Masks I recently had to audit the security function ownership used by column masks in our Databricks Unity Catalog environment. The goal was to ensure that the functions used for data masking were properly secured and that only authorized users had access to them. I started by…
A1, auditing, category:business, category:education, category:software, category:technology, coding, column, data engineering, data governance, databricks, education, function, Lakehouse, masks, ownership, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, security, sql, unity catalog, Used -
Investigating a Column Mask That Blocks an Approved Production User
POST_START Investigating a Column Mask That Blocks an Approved Production User I recently encountered an issue where an approved production user was unable to access certain data in the customer_master table. The user had the necessary permissions, but when they tried to query the table, some columns were masked or hidden. My task was to…
A1, approved, blocks, category:business, category:education, category:software, category:technology, coding, column, data engineering, data governance, databricks, education, investigating, Lakehouse, mask, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, That, unity catalog, user -
Investigating a Column Mask That Exposes Sensitive Production Values
POST_START Investigating a Column Mask That Exposes Sensitive Production Values I recently encountered an issue while reviewing the customer data in our production environment. A column mask was supposed to protect sensitive information, but it appeared to be exposing some production values. I needed to investigate the table structure and data to understand what was…
A1, category:business, category:education, category:software, category:technology, coding, column, data engineering, data governance, databricks, education, exposes, investigating, Lakehouse, mask, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sensitive, sql, That, unity catalog, values -
Designing Department-Based Row-Level Security for Enterprise Tables
POST_START Designing Department-Based Row-Level Security for Enterprise Tables Setting Up the Security Function I started by thinking about how to secure our enterprise tables based on department access. I needed a way to filter rows so that users could only see data relevant to their department. The solution was to create a security function that…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, department-based, designing, education, Enterprise, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, row-level, security, sql, tables, unity catalog -
Protecting Production PII While Preserving Analytical Access
POST_START Protecting Production PII While Preserving Analytical Access I recently joined a team that’s working with a large customer dataset in Databricks. Our goal is to ensure that sensitive personal information (PII) is protected in production while still allowing analysts to perform their work. As part of my onboarding, I was asked to examine the…
A1, Access, analytical, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, pii, preserving, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, protecting, sql, unity catalog, while -
Testing Column-Level Security for Privileged and Standard Users
POST_START Testing Column-Level Security for Privileged and Standard Users I recently needed to verify how column-level security is enforced in our Databricks environment. My task was to test whether privileged users could access certain columns in a table, while standard users were restricted. To do this, I started by examining the structure of the customer_master…
A1, category:business, category:education, category:software, category:technology, coding, column-level, data engineering, data governance, databricks, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileged, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, security, sql, standard, Testing, unity catalog, users -
Investigating a Row Filter That Hides Expected Production Records
POST_START Investigating a Row Filter That Hides Expected Production Records I recently encountered an issue where some expected production records were missing from our financial transactions table. My task was to investigate why these records weren’t showing up, even though they should have been in the dataset. I started by checking the table structure to…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, expected, filter, hides, investigating, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, records, row, sql, That, unity catalog -
Designing Region-Based Row-Level Security for Shared Production Data
POST_START Designing Region-Based Row-Level Security for Shared Production Data I recently took on the responsibility of securing access to a shared production dataset in Databricks. The dataset, production.sales.regional_orders, contains sensitive customer information, and I needed to ensure that users only saw data relevant to their region. The goal was to implement region-based row-level security using…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, designing, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, region-based, row-level, security, shared, sql, unity catalog -
Testing Row-Level Security for Multiple Production User Roles
POST_START Testing Row-Level Security for Multiple Production User Roles I recently took on the task of verifying that row-level security (RLS) policies are correctly configured for multiple user roles in the production environment. My goal was to ensure that each role only sees the data it’s authorized to access. To start, I needed to understand…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, Lakehouse, multiple, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, roles, row-level, security, sql, Testing, unity catalog, user -
Investigating a Row Filter That Returns Too Many Production Records
POST_START Investigating a Row Filter That Returns Too Many Production Records I recently encountered a problem where a row filter was returning more records than expected from the production.finance.transactions table. The filter was intended to limit results to only active transactions, but the query was returning a large number of records, including some that were…
(many), A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, filter, investigating, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, returns, row, sql, That, Too, unity catalog
