POST_START

Creating and Managing External Volumes in Databricks Unity Catalog

I recently started working on a new data pipeline for the sales team, and I needed to organize the raw files they’re ingesting from various sources. I learned that Databricks Unity Catalog provides a way to manage external volumes, which are useful for storing and organizing data that’s not managed by the catalog itself. I decided to create an external volume to hold these raw files and then manage it as part of my workflow.

Creating an External Volume

I began by creating an external volume called training.sales.raw_files to store the incoming raw data. I used the CREATE EXTERNAL VOLUME command and specified the location where these files would be stored.

CREATE EXTERNAL VOLUME training.sales.raw_files LOCATION '/raw-files';

I noticed that the command executed successfully, and the volume was created in the training.sales catalog. This step allowed me to define a dedicated space for the raw files, making it easier to manage and access them later.

Describing the Volume

Next, I wanted to verify the details of the volume I just created. I ran the DESCRIBE VOLUME command to get information about its configuration.

DESCRIBE VOLUME training.sales.raw_files;

<

name	catalog	volumeType
documents	production	MANAGED

I verified that the volume was indeed created in the training.sales catalog and that it was of type EXTERNAL. This confirmed that the volume is managed outside of the catalog, which aligns with my goal of organizing external data sources.

Listing All Volumes in the Catalog

To better understand the structure of the training.sales catalog, I decided to list all the volumes that exist there. I used the SHOW VOLUMES IN command to get a clear view of the volumes and their types.

SHOW VOLUMES IN training.sales;

database	volumeName	volumeType
files	documents	MANAGED
files	landing	EXTERNAL

I noticed that there were already some volumes in the catalog, such as documents and landing. The landing volume was also of type EXTERNAL, which made me realize that this catalog was already being used to organize external data sources. This helped me understand how volumes are leveraged within the Unity Catalog for data governance.

Dropping the Volume

After confirming that the volume was correctly set up, I decided to test the DROP VOLUME command to understand how to remove it if needed. I ran the command to delete the training.sales.raw_files volume.

DROP VOLUME training.sales.raw_files;

I verified that the volume was successfully dropped, and I noticed that it no longer appeared in the list of volumes. This showed me how easy it is to manage volumes within the Unity Catalog, which is a key part of data lifecycle management.

Through this exercise, I learned how to create, describe, list, and drop external volumes in Databricks Unity Catalog. This knowledge will help me design more robust data pipelines and maintain better control over data sources in the future.