data engineering
-
Creating and Using Reusable SQL Functions
POST_START Creating and Using Reusable SQL Functions in Databricks Unity Catalog I recently needed to calculate tax amounts for various sales transactions in my Databricks environment. To make this process more efficient and maintainable, I decided to create a reusable SQL function using Unity Catalog. This function would allow me to apply the same tax…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, education, Functions, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, reusable, sql, unity catalog -
Creating Identity-Aware Views for Conditional Data Access
POST_START Creating Identity-Aware Views for Conditional Data Access I recently had the task of implementing a secure data access pattern for customer information in our sales database. The goal was to create a view that would expose customer data conditionally based on the user’s identity. This way, only users who are part of the ‘pii_readers’…
A1, Access, category:business, category:education, category:software, category:technology, coding, Conditional, creating, data, data engineering, data governance, databricks, education, identity-aware, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, Views -
Creating and Managing Permanent Unity Catalog Views
POST_START Creating and Managing Permanent Unity Catalog Views I recently needed to create a permanent view in Unity Catalog to simplify access to customer data for my team. I decided to use the CREATE OR REPLACE VIEW command to define a view that selects specific columns from the customers table in the training.sales namespace. CREATE…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, education, Lakehouse, Managing, paragraph ordering, paragraph structure, paragraphorder, permanent, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, Views -
Creating and Using Session-Scoped Temporary Views
POST_START Creating and Using Session-Scoped Temporary Views I recently needed to analyze a subset of customer data for a reporting task. The dataset was large, and I wanted to filter for active customers in Germany first. I decided to create a temporary view to simplify my analysis and keep the data isolated for the session.…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, session-scoped, sql, temporary, unity catalog, Views -
Dropping Unity Catalog Tables
POST_START Dropping Unity Catalog Tables Starting the Process I recently needed to clean up some old data in my Unity Catalog to make space and ensure that our data lake remains organized. One of the tables I identified for removal was training.sales.customer_master. Before I proceed, I wanted to make sure I understood the implications of…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, dropping, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, tables, unity catalog -
Removing All Rows While Preserving a Table with TRUNCATE
POST_START Removing All Rows While Preserving a Table with TRUNCATE I recently had the task of cleaning up some data in the training.sales.customer_master table. The goal was to remove all rows from the table while keeping the table structure intact. This is a common operation when you need to reset a table for testing or…
(all), A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, preserving, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, removing, rows, sql, table, truncate, unity catalog, while -
Upserting Data with Delta MERGE
POST_START Upserting Data with Delta MERGE Introduction I recently needed to synchronize customer data between two tables in my Delta Lake environment. One table, called customers, was the source of truth, and the other, customer_updates, contained new or modified records. The challenge was to efficiently update existing records and insert new ones in a single…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, Delta, education, Lakehouse, merge, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, upserting -
Deleting Selected Records from Delta Tables
POST_START Deleting Selected Records from Delta Tables I recently needed to clean up some customer data in my Delta table. The task was to remove a specific customer record by their ID. I started by checking the current state of the table to make sure I understood what I was working with. SELECT * FROM…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, deleting, Delta, education, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, records, selected, sql, tables, unity catalog -
Updating Existing Records in Delta Tables
POST_START Updating Existing Records in Delta Tables I recently needed to update a specific customer’s country in our Delta table. The customer had an ID of 1, and I wanted to change their country to the Netherlands. I knew that Delta Lake supports updates, but I wanted to make sure I did it correctly using…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, Delta, education, existing, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, records, sql, tables, unity catalog, Updating
