data
-
Granting Least-Privilege RBAC Write Access to Production Data Engineering Teams
POST_START Granting Least-Privilege RBAC Write Access to Production Data Engineering Teams Granting Least-Privilege RBAC Write Access to Production Data Engineering Teams I started my day by reviewing the latest security requirements for our data engineering team. We needed to grant them the ability to write to the production data catalog, but with the strictest access…
A1, Access, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, engineering, granting, Lakehouse, least-privilege, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, rbac, sql, unity catalog, Write -
Finding Excessive or Unexpected Unity Catalog Privileges on Production Data
POST_START Identifying Excessive or Unexpected Unity Catalog Privileges on Production Data I recently had to investigate a concern raised by our security team about potential excessive or unexpected privileges on our production data in Unity Catalog. The team was worried that some users might have access to data they shouldn’t, which could lead to data…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, excessive, finding, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, Unexpected, unity catalog -
Inspecting Storage Credentials for External Data Access
POST_START Inspecting Storage Credentials for External Data Access Inspecting Storage Credentials for External Data Access I recently needed to inspect the storage credentials used by my team to access external data sources. Our data pipeline relies on several external systems, and ensuring that the credentials are correctly configured and secured is crucial for both performance…
A1, Access, category:business, category:education, category:software, category:technology, coding, credentials, data, data engineering, data governance, databricks, education, External, inspecting, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, Storage, unity catalog -
Controlling Table Data Changes with MODIFY Privileges
POST_START Controlling Table Data Changes with MODIFY Privileges I recently had the task of ensuring that only specific data engineers could modify the customer data in our training database. The goal was to prevent accidental or unauthorized changes to the training.sales.customers table while still allowing the team to perform necessary updates. I decided to use…
A1, category:business, category:education, category:software, category:technology, changes, coding, controlling, data, data engineering, data governance, databricks, education, Lakehouse, modify, paragraph ordering, paragraph structure, paragraphorder, privileges, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, table, unity catalog -
Upserting Data with Delta MERGE
POST_START <h1>Upserting Data with Delta MERGE</h1> <p> I'm working on a project to keep our customer data up to date in the Delta table <code>training.sales.customers</code>. Every day, we receive new data in the <code>training.sales.customer_updates</code> table, and we need to make sure that any changes to existing customers are reflected, while new customers are added. I…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, Delta, education, Lakehouse, merge, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, upserting -
Loading Table Data from Queries with INSERT SELECT
POST_START # Loading Table Data from Queries with INSERT SELECT In this tutorial, you'll learn how to load data from one table into another using the `INSERT INTO … SELECT` statement in Databricks Unity Catalog. This operation is useful for copying subsets of data, filtering based on conditions, and maintaining data integrity across different tables.…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, insert, Lakehouse, loading, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, queries, select, sql, table, unity catalog -
Databricks Unity Catalog: Ensure Secure Publication of Governed Data Products with Lineage Tracking
Databricks Unity Catalog: Secure Publication with Lineage Tracking Databricks Unity Catalog: Secure Publication of Governed Data Products with Lineage Tracking This tutorial demonstrates the secure publication of governed data products with lineage tracking using Databricks Unity Catalog and Databricks SQL. We’ll focus solely on the publication and lineage aspects, utilizing a simplified scenario to illustrate…
A1, Catalog, category:data cable, category:digital product, category:education, category:networking, category:software, coding, data, data engineering, databricks, databricks sql, delta lake, education, ensure, governed, paragraph ordering, paragraph structure, paragraphorder, profession:data analyst, profession:data engineer, profession:data scientist, profession:linguist, profession:student, profession:teacher, profession:translator, progressive learning, publication, secure, sql, sql engineering, unity -
Databricks Unity Catalog: Address Privilege Drift After a Data Migration to Unity Catalog
Databricks Unity Catalog Tutorial: Addressing Privilege Drift After Data Migration Databricks Unity Catalog Tutorial: Addressing Privilege Drift After Data Migration This tutorial focuses on mitigating privilege drift after migrating data to Databricks Unity Catalog using Databricks SQL. We’ll walk through three scripts demonstrating progressive steps in identifying and correcting privilege discrepancies. Script 1: Initial Privilege…
A1, address, after, Catalog, category:computer, category:data cable, category:education, category:networking, category:software, coding, data, data engineering, databricks, databricks sql, delta lake, drift, education, paragraph ordering, paragraph structure, paragraphorder, privilege, profession:data analyst, profession:data engineer, profession:data scientist, profession:linguist, profession:student, profession:teacher, profession:translator, progressive learning, sql, sql engineering, unity -
Databricks SQL: Reduce Data Scans with Selective Predicates
Databricks SQL Tutorial: Reduce Data Scans with Selective Predicates Databricks SQL Tutorial: Reduce Data Scans with Selective Predicates This tutorial focuses on reducing data scans in Databricks SQL by strategically using selective predicates. We’ll explore how to filter data as early as possible in your queries to minimize the amount of data processed. Script 1:…
A1, category:education, category:software, coding, data, data engineering, databricks, databricks sql, delta lake, education, paragraph ordering, paragraph structure, paragraphorder, predicates, profession:data analyst, profession:data engineer, profession:data scientist, profession:linguist, profession:student, profession:teacher, profession:translator, progressive learning, Reduce, scans, selective, sql, sql engineering -
Databricks SQL: Detect Data Quality Failures with SQL Metrics
Databricks SQL: Detect Data Quality Failures with SQL Metrics Databricks SQL: Detect Data Quality Failures with SQL Metrics This tutorial guides you through detecting data quality failures using SQL metrics within Databricks SQL. We’ll build three progressively complex queries to identify issues in sample data. Script 1: Detecting Null Values This script demonstrates the simplest…
A1, category:computer, category:data cable, category:education, category:software, coding, data, data engineering, databricks, databricks sql, delta lake, detect, education, failures, Metrics., paragraph ordering, paragraph structure, paragraphorder, profession:data analyst, profession:data engineer, profession:data scientist, profession:linguist, profession:student, profession:teacher, profession:translator, progressive learning, quality, sql, sql engineering
