data engineering
-
Protecting Sensitive Columns with Column Masks
POST_START Protecting Sensitive Columns with Column Masks I recently needed to ensure that sensitive data, like customer emails, was only accessible to specific users in our Databricks environment. I decided to use column masks to control access to the email column in the customers table. This approach allows me to hide the email addresses from…
A1, category:business, category:education, category:software, category:technology, coding, column, columns, data engineering, data governance, databricks, education, Lakehouse, masks, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, protecting, sensitive, sql, unity catalog -
Removing Column Masks from Protected Columns
POST_START Removing Column Masks from Protected Columns Understanding the Need to Remove Column Masks I recently took on the responsibility of managing sensitive customer data in our Databricks environment. One of the key tasks was to ensure that data access adhered to strict privacy regulations. As part of this, certain columns, like the customer email,…
A1, category:business, category:education, category:software, category:technology, coding, column, columns, data engineering, data governance, databricks, education, Lakehouse, masks, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, protected, removing, sql, unity catalog -
Removing Row-Level Filters from Tables
POST_START Removing Row-Level Filters from Tables I recently encountered a situation where a table in our data lake had been filtered at the row level, and we needed to remove those filters to access the full dataset. The table in question was training.sales.customers, and the filters were applied as part of a row-level security policy.…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, filters, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, removing, row-level, sql, tables, unity catalog -
Implementing Row-Level Access Control with Row Filters
POST_START Implementing Row-Level Access Control with Row Filters I recently needed to implement row-level access control for a sales team in Databricks. The goal was to ensure that only members of the ‘global_sales’ account group could see all customer data, while others could only access data for Germany. I decided to use Unity Catalog’s row…
A1, Access, category:business, category:education, category:software, category:technology, coding, control, data engineering, data governance, databricks, education, filters, implementing, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, row, row-level, sql, unity catalog -
Creating and Managing External Volumes
POST_START Creating and Managing External Volumes in Databricks Unity Catalog I recently started working on a new data pipeline for the sales team, and I needed to organize the raw files they’re ingesting from various sources. I learned that Databricks Unity Catalog provides a way to manage external volumes, which are useful for storing and…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, education, External, Lakehouse, Managing, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, volumes -
Controlling File Access with Volume Privileges
POST_START Controlling File Access with Volume Privileges I recently needed to set up a secure environment for my team to access specific files in the Databricks Unity Catalog. The goal was to ensure that only certain roles could read or write to a particular volume. I decided to use volume privileges to control access, which…
A1, Access, category:business, category:education, category:software, category:technology, coding, controlling, data engineering, data governance, databricks, education, file, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, Volume. -
Creating and Managing External Delta Tables
POST_START Creating and Managing External Delta Tables I recently needed to create an external Delta table in Databricks Unity Catalog to integrate data from an external source. This table would be used to store sales orders, and I wanted to ensure it was properly managed within the catalog. I started by defining the structure of…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, Delta, education, External, Lakehouse, Managing, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, tables, unity catalog -
Creating and Managing Managed Volumes for Non-Tabular Files
POST_START Creating and Managing Managed Volumes for Non-Tabular Files I recently needed to organize a growing collection of non-tabular files, such as PDFs, CSVs, and text documents, for a data pipeline. These files were not structured in a traditional table format, so using standard tables wasn’t the right approach. Instead, I decided to leverage Databricks…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, education, files, Lakehouse, Managed, Managing, non-tabular, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, volumes -
Managing Read, Write, and Table-Creation Access on External Locations
POST_START Managing Read, Write, and Table-Creation Access on External Locations Setting Up Access for Data Engineers I recently needed to set up access for the data_engineers team to an external location called raw_data. The goal was to grant them the ability to read files from this location, but also to manage their permissions carefully to…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, External, Lakehouse, locations, Managing, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, read, sql, table-creation, unity catalog, Write
