POST_START
Reviewing Storage Credentials Used by Production External Locations
I recently needed to review the storage credentials used by our production external locations to ensure they were correctly configured and secure. This was part of a routine audit to verify that all external data sources are properly managed within the Databricks Unity Catalog. I started by checking the list of available storage credentials in the environment.
SHOW STORAGE CREDENTIALS;
| name | credential_type | owner |
|---|---|---|
| production_storage | SERVICE_PRINCIPAL | data_platform_admins |
I noticed that there was a single storage credential named production_storage of type SERVICE_PRINCIPAL, owned by the data_platform_admins group. This credential was likely used for accessing our production storage accounts. To understand more about it, I decided to describe the credential in detail.
DESCRIBE STORAGE CREDENTIAL production_storage;
| name | credential_type | owner |
|---|---|---|
| production_storage | SERVICE_PRINCIPAL | data_platform_admins |
The description confirmed what I had already seen. The credential was of type SERVICE_PRINCIPAL, which means it was associated with an Azure Active Directory service principal. This is a common setup for secure access to cloud storage in production environments. The ownership by the data_platform_admins group indicated that this credential was managed by the data platform team, which aligns with our security practices.
Next, I wanted to check which external locations were using this credential. I looked up the production_raw external location, which was expected to be one of the primary data sources accessed by our production pipelines.
DESCRIBE EXTERNAL LOCATION production_raw;
| name | url | credential_name | owner |
|---|---|---|---|
| production_raw | s3://company-prod/raw | production_storage | data_platform_admins |
The result showed that the production_raw external location was using the production_storage credential. The URL s3://company-prod/raw pointed to the raw data lake in our AWS S3 bucket. The ownership by the data_platform_admins group reinforced that this location was managed by the data platform team, and the credential was appropriately assigned.
By reviewing these credentials and their associated external locations, I confirmed that the production storage access was correctly configured. The production_storage credential was being used by the production_raw external location, and both were owned by the same administrative group. This consistency helped ensure that access controls were properly enforced across our data pipeline.
Leave a Reply