Transferring Data with Azure Storage Accounts

This page explains how to transfer data to/from your Azure storage account with a terminal. You can use the methods on this page for all Azure storage accounts, whether you created them on the PW platform or outside the platform.

Working With azcopy

To transfer data to/from an Azure storage account, you’ll use Azure's command-line tool, azcopy.

Azcopy is pre-installed on cloud clusters provisioned by the PW platform, so you can enter commands directly into the IDE after logging in to the controller of an active Azure cluster.

If you’re transferring data between Azure storage and your local machine or an on-premises cluster, you will need to install azcopy first.

Check for azcopy

Open a terminal and enter which azcopy.

If azcopy is installed, you’ll see a message that shows its location, such as /usr/local/bin/azcopy. Otherwise, you’ll see a message such as /usr/bin/which: no azcopy or azcopy not found.

Install azcopy

If you need to install azcopy, we recommend following the Azure installation guide, which includes OS-specific instructions for Linux, macOS, and Windows as well as troubleshooting tips.

Export Your Azure Credentials

You can see our page Obtaining Credentials for information on finding your Azure credentials.

In your terminal, enter the following commands with your own Azure credentials substituted:

export BUCKET_NAME=your-bucket-name
export BUCKET_ACCOUNT_NAME=your-account-name
export AZURE_CLIENT_ID=your-Azure-client-ID
export AZURE_TENANT_ID=your-Azure-tenant_ID
export AZCOPY_SPA_CLIENT_SECRET=your-Azure-client-secret
export AZCOPY_BUCKET_URL=https://your-account-name.blob.core.windows.net/your-bucket-name
azcopy login —service-principal —application-id $AZURE_CLIENT_ID —tenant-id $AZURE_TENANT_ID

List Files in an Azure Storage Account

In your terminal, enter azcopy ls $AZCOPY_BUCKET_URL to display the files in your bucket. For this guide, we used a small text file named test.txt, so our command returned this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ azcopy ls $AZCOPY_BUCKET_URL
2023-02-08 16:37:56         28 test.txt

Transfer a File To/From an Azure Storage Account

The azcopy command mimics the Linux cp command for transferring files. To transfer a file, enter azcopy cp SOURCE DESTINATION in your terminal.

Below is an example of the azcopy cp command.

In your terminal, enter azcopy cp $AZCOPY_BUCKET_URL/file/in/bucket.txt fileName to download a file from Blob storage to your current directory. You’ll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ azcopy cp $AZCOPY_BUCKET_URL/test.txt test_download.txt
download: https://test-storage.blob.core.windows.net/test-bucket/test.txt to test_download.txt

To upload a file to your Azure storage account, simply reverse the order of SOURCE and DESTINATION in the azcopy cp command.

Delete a File From an Azure Storage Account

In your terminal, enter azcopy rm $AZCOPY_BUCKET_URL/file/in/bucket.txt to delete a file. You’ll see this message:

demo@pw-user-demo:~/pw/cloud-data-transfer$ azcopy rm $AZCOPY_BUCKET_URL/test_upload.txt
delete: https://test-storage.blob.core.windows.net/test-bucket/test_upload.txt

Transferring Data with Azure Storage Accounts

Working With azcopy​

Check for azcopy​

Install azcopy​

Export Your Azure Credentials​

List Files in an Azure Storage Account​

Transfer a File To/From an Azure Storage Account​

Delete a File From an Azure Storage Account​

Further Reading​