data engineering
-
Validating an External Location Before Production Data Onboarding
POST_START Validating an External Location Before Production Data Onboarding I recently had the task of onboarding a new data source into our production environment. Before I could proceed, I needed to validate the external location to ensure it met our security, access, and governance standards. The first step was to check what external locations were…
A1, before, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, External, Lakehouse, location, onboarding, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, validating -
Investigating a Specific User During a Production Access Incident
POST_START Investigating a Specific User During a Production Access Incident I was called in to investigate a potential access incident involving a user with the email address engineer@example.com. The incident report mentioned that a user might have accessed production data, and I needed to trace their activity using Databricks Unity Catalog’s audit logs. I started…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, during, education, incident, investigating, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, Specific, sql, unity catalog, user -
Investigating User Activity Against a Production Table with Audit Logs
POST_START Investigating User Activity Against a Production Table with Audit Logs I recently needed to investigate user activity against a production table in our data warehouse. The table in question was production.sales.orders, and I wanted to understand who had accessed it, what actions they performed, and when. Since we use Databricks Unity Catalog for access…
A1, Activity, against, audit, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, investigating, Lakehouse, logs, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, table, unity catalog, user -
Auditing Production Catalog Privileges Through INFORMATION_SCHEMA
POST_START Auditing Production Catalog Privileges Through INFORMATION_SCHEMA I recently needed to audit the privileges assigned to the production catalog in our Databricks environment. This was part of a routine security review to ensure that only authorized users had access to sensitive data. I decided to use the INFORMATION_SCHEMA to get a clear and structured view…
A1, auditing, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, information, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, schema, sql, through, unity catalog -
Auditing Production Table Privileges Through INFORMATION_SCHEMA
POST_START Auditing Production Table Privileges Through INFORMATION_SCHEMA Today, I needed to audit the privileges assigned to a specific table in the production environment. The table in question was orders from the sales schema. My goal was to understand who has access to this table and what kind of privileges they hold. I decided to use…
A1, auditing, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, information, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, schema, sql, table, through, unity catalog -
Auditing Production Schema Privileges Through INFORMATION_SCHEMA
POST_START Auditing Production Schema Privileges Through INFORMATION_SCHEMA I recently needed to audit the privileges assigned to the sales schema in the production catalog. This is a common task when ensuring that only authorized users have access to sensitive or critical data. I decided to use the INFORMATION_SCHEMA to get a clear and structured view of…
A1, auditing, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, information, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, schema, sql, through, unity catalog -
Removing an Obsolete Governance Tag from a Production Table
POST_START Removing an Obsolete Governance Tag from a Production Table I recently had to address an outdated governance tag on a production table in our data lake. The table, production.customers.customer_master, was marked with a legacy_classification tag that was no longer relevant. My task was to remove this tag and verify that it was successfully removed.…
A1, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, Governance, Lakehouse, obsolete, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, removing, sql, table, tag, unity catalog
