data engineering
-
Reviewing Storage Credentials Used by Production External Locations
POST_START Reviewing Storage Credentials Used by Production External Locations I recently needed to review the storage credentials used by our production external locations to ensure they were correctly configured and secure. This was part of a routine audit to verify that all external data sources are properly managed within the Databricks Unity Catalog. I started…
A1, category:business, category:education, category:software, category:technology, coding, credentials, data engineering, data governance, databricks, education, External, Lakehouse, locations, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, Reviewing, sql, Storage, unity catalog, Used -
Granting Read Access to a Governed Production External Location
POST_START Granting Read Access to a Governed Production External Location I recently had the task of granting read access to a governed production external location in Databricks Unity Catalog. The location, named production_raw, is used by data engineers to access raw data stored in an S3 bucket. My goal was to ensure that the data_engineers…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, External, governed, granting, Lakehouse, location, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, read, sql, unity catalog -
Validating an External Location Before Production Data Onboarding
POST_START Validating an External Location Before Production Data Onboarding I recently had the task of onboarding a new data source into our production environment. Before I could proceed, I needed to validate the external location to ensure it met our security, access, and governance standards. The first step was to check what external locations were…
A1, before, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, External, Lakehouse, location, onboarding, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, validating -
Investigating a Specific User During a Production Access Incident
POST_START Investigating a Specific User During a Production Access Incident I was called in to investigate a potential access incident involving a user with the email address engineer@example.com. The incident report mentioned that a user might have accessed production data, and I needed to trace their activity using Databricks Unity Catalog’s audit logs. I started…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, during, education, incident, investigating, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, Specific, sql, unity catalog, user -
Investigating User Activity Against a Production Table with Audit Logs
POST_START Investigating User Activity Against a Production Table with Audit Logs I recently needed to investigate user activity against a production table in our data warehouse. The table in question was production.sales.orders, and I wanted to understand who had accessed it, what actions they performed, and when. Since we use Databricks Unity Catalog for access…
A1, Activity, against, audit, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, investigating, Lakehouse, logs, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, table, unity catalog, user -
Auditing Production Catalog Privileges Through INFORMATION_SCHEMA
POST_START Auditing Production Catalog Privileges Through INFORMATION_SCHEMA I recently needed to audit the privileges assigned to the production catalog in our Databricks environment. This was part of a routine security review to ensure that only authorized users had access to sensitive data. I decided to use the INFORMATION_SCHEMA to get a clear and structured view…
A1, auditing, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, information, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, schema, sql, through, unity catalog -
Auditing Production Table Privileges Through INFORMATION_SCHEMA
POST_START Auditing Production Table Privileges Through INFORMATION_SCHEMA Today, I needed to audit the privileges assigned to a specific table in the production environment. The table in question was orders from the sales schema. My goal was to understand who has access to this table and what kind of privileges they hold. I decided to use…
A1, auditing, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, information, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, privileges, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, schema, sql, table, through, unity catalog
