POST_START

Validating an External Location Before Production Data Onboarding

I recently had the task of onboarding a new data source into our production environment. Before I could proceed, I needed to validate the external location to ensure it met our security, access, and governance standards. The first step was to check what external locations were already registered in Unity Catalog.

SHOW EXTERNAL LOCATIONS;

name	url	credential_name
production_raw	s3://company-prod/raw	production_storage

I ran the SHOW EXTERNAL LOCATIONS; command, which returned a list of all external locations in the catalog. The output showed that there was already a location named production_raw, pointing to an S3 bucket at s3://company-prod/raw, and using the credential production_storage. This was the location we intended to use for onboarding new production data.

Next, I wanted to understand more about the production_raw external location, including who owns it and any additional metadata. I ran the DESCRIBE EXTERNAL LOCATION production_raw; command to get more details.

DESCRIBE EXTERNAL LOCATION production_raw;

name	url	credential_name	owner
production_raw	s3://company-prod/raw	production_storage	data_platform_admins

The output confirmed the URL and credential, and also showed that the location was owned by the data_platform_admins group. This was important because it meant that the location had been set up by the data platform team and was under their governance. I noted this down for reference, as it helped me understand the ownership and responsibility for the location.

To ensure that the location was properly secured and accessible to the right teams, I decided to check the access permissions. I ran the SHOW GRANTS ON EXTERNAL LOCATION production_raw; command to see which roles or users had been granted access.

SHOW GRANTS ON EXTERNAL LOCATION production_raw;

principal	actionType	objectType
data_analysts	USE CATALOG	CATALOG
data_engineers	USE CATALOG	CATALOG

The output showed that the data_analysts and data_engineers groups had been granted the USE CATALOG action on the catalog. This meant they could access the external location and use the data it contained. I verified that these were the correct groups, as they were the ones who would be using the data in production.

With this information, I felt confident that the production_raw external location was properly set up, secured, and accessible to the right teams. I was ready to proceed with the onboarding process, knowing that the location had been validated and met all the necessary requirements.