data engineering
-
Hardening Row-Level Security with a Controlled Administrator Bypass
POST_START Hardening Row-Level Security with a Controlled Administrator Bypass I recently had the task of hardening row-level security for a customer table in our data lake. The goal was to enforce a region-based filter, but also allow for a controlled bypass for administrators. I needed to implement a function that would restrict access to data…
A1, administrator, bypass, category:business, category:education, category:software, category:technology, coding, controlled, data engineering, data governance, databricks, education, hardening, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, row-level, security, sql, unity catalog -
Designing Multi-Group Row-Level Security for Regional Sales Data
POST_START Designing Multi-Group Row-Level Security for Regional Sales Data I’ve been tasked with implementing row-level security for the sales data in our Databricks environment. The goal is to ensure that users can only access sales data relevant to their region. This involves setting up a function that filters data based on user group membership and…
A1, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, designing, education, Lakehouse, multi-group, paragraph ordering, paragraph structure, paragraphorder, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, regional, row-level, sales, security, sql, unity catalog -
End-to-End Production Security Review for Row and Column Protection
POST_START End-to-End Production Security Review for Row and Column Protection I recently had the opportunity to perform a comprehensive security review of a customer data pipeline in Databricks Unity Catalog. The goal was to ensure that both row-level and column-level data protection were properly implemented and enforced. I started by understanding the functions and tables…
A1, category:business, category:education, category:software, category:technology, coding, column, data engineering, data governance, databricks, education, End-to-End, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, protection, review, row, security, sql, unity catalog -
Documenting a Production Table and Sensitive Column for Governance Review
POST_START Documenting a Production Table and Sensitive Column for Governance Review I recently took on the responsibility of ensuring that our production data assets are properly documented for governance review. One of the key tables in our system is the customer_master table in the production.customers schema. This table holds the core customer information, and it’s…
A1, category:business, category:education, category:software, category:technology, coding, column, data engineering, data governance, databricks, documenting, education, Governance, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, review, sensitive, sql, table, unity catalog -
End-to-End Production Access Investigation from Namespace to Audit Trail
POST_START End-to-End Production Access Investigation from Namespace to Audit Trail I recently needed to investigate access to the production catalog in Databricks Unity Catalog. My goal was to understand who had access to what, what actions they could perform, and how their interactions were recorded in the audit trail. This is a common task when…
A1, Access, audit, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, End-to-End, investigation, Lakehouse, namespace, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, trail, unity catalog -
Testing a Production Table Schema Change Before Grant Review
POST_START Testing a Production Table Schema Change Before Grant Review I recently needed to test a schema change for a production table before submitting it for grant review. The table in question was production.sales.orders, and the goal was to add a new column to track the status of a review process. Before making any changes,…
A1, before, category:business, category:education, category:software, category:technology, change, coding, data engineering, data governance, databricks, education, grant, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, review, schema, sql, table, Testing, unity catalog -
Investigating a Production Table Rename Through Object Discovery
POST_START Investigating a Production Table Rename Through Object Discovery Today, I was tasked with investigating a recent table rename in the production environment. The team had renamed a table from orders to orders_archive, and I needed to verify that the change was correctly applied and understand the impact on the system. I started by checking…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, discovery, education, investigating, Lakehouse, object, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, rename, sql, table, through, unity catalog -
Reviewing Production Table Properties During an Operational Investigation
POST_START Reviewing Production Table Properties During an Operational Investigation I recently encountered an issue with the production table production.sales.orders that required me to investigate its properties to understand its structure and behavior. As part of the operational investigation, I needed to check the table’s metadata to determine if there were any settings that might be…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, during, education, investigation, Lakehouse, operational, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, properties, Reviewing, sql, table, unity catalog
