data engineering
-
Granting Read-Only Access to a Production Volume
POST_START Granting Read-Only Access to a Production Volume I recently needed to grant read-only access to a production volume in Databricks Unity Catalog so that the data analysts could explore the data without modifying it. The volume in question was production.files.landing, and the target principal was the data_analysts group. Granting Read-Only Access I started by…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, granting, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, read-only, sql, unity catalog, Volume. -
Removing Write Access from a Production External Location
POST_START Removing Write Access from a Production External Location I recently had the task of securing a production external location in Databricks Unity Catalog. The location, named production_raw, was being used by several teams, but we needed to remove write access for a legacy ingestion service to prevent accidental or unauthorized data modifications. Understanding the…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, External, Lakehouse, location, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, removing, sql, unity catalog, Write -
Creating a Governed External Volume for Production Landing Files
POST_START Creating a Governed External Volume for Production Landing Files I’m working on setting up a new data pipeline for our production environment. As part of the data governance process, I need to create a governed external volume for the landing files that will be ingested from S3. I started by checking what volumes already…
A1, category:business, category:education, category:software, category:technology, coding, creating, data engineering, data governance, databricks, education, External, files, governed, Lakehouse, landing, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, Volume. -
Reviewing Storage Credentials Used by Production External Locations
POST_START Reviewing Storage Credentials Used by Production External Locations I recently needed to review the storage credentials used by our production external locations to ensure they were correctly configured and secure. This was part of a routine audit to verify that all external data sources are properly managed within the Databricks Unity Catalog. I started…
A1, category:business, category:education, category:software, category:technology, coding, credentials, data engineering, data governance, databricks, education, External, Lakehouse, locations, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, Reviewing, sql, Storage, unity catalog, Used -
Granting Read Access to a Governed Production External Location
POST_START Granting Read Access to a Governed Production External Location I recently had the task of granting read access to a governed production external location in Databricks Unity Catalog. The location, named production_raw, is used by data engineers to access raw data stored in an S3 bucket. My goal was to ensure that the data_engineers…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, education, External, governed, granting, Lakehouse, location, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, read, sql, unity catalog -
Validating an External Location Before Production Data Onboarding
POST_START Validating an External Location Before Production Data Onboarding I recently had the task of onboarding a new data source into our production environment. Before I could proceed, I needed to validate the external location to ensure it met our security, access, and governance standards. The first step was to check what external locations were…
A1, before, category:business, category:education, category:software, category:technology, coding, data, data engineering, data governance, databricks, education, External, Lakehouse, location, onboarding, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, sql, unity catalog, validating -
Investigating a Specific User During a Production Access Incident
POST_START Investigating a Specific User During a Production Access Incident I was called in to investigate a potential access incident involving a user with the email address engineer@example.com. The incident report mentioned that a user might have accessed production data, and I needed to trace their activity using Databricks Unity Catalog’s audit logs. I started…
A1, Access, category:business, category:education, category:software, category:technology, coding, data engineering, data governance, databricks, during, education, incident, investigating, Lakehouse, paragraph ordering, paragraph structure, paragraphorder, production, profession:cloud engineer, profession:data analyst, profession:data engineer, profession:data scientist, profession:database administrator, profession:software engineer, profession:solution architect, progressive learning, Specific, sql, unity catalog, user
