Transferring Data with AWS S3
This page explains how to transfer data to/from your AWS S3 buckets with a terminal. You can use the methods on this page for all S3 buckets, whether you created them on the PW platform or outside the platform.
To transfer data to/from S3 bucket storage, you’ll use the AWS Command-Line Interface (CLI).
The AWS CLI is pre-installed on cloud clusters provisioned by the PW platform, so you can enter commands directly into the IDE after logging in to the controller of an active AWS cluster.
If you’re transferring data between S3 buckets and your local machine or an on-premises cluster, you’ll likely need to install the AWS CLI first.
Check for AWS CLI
Open a terminal or command line and navigate to your data’s destination. Enter which aws.
If the AWS CLI is installed, you’ll see a message that shows its location, such as /usr/local/bin/aws.
If the AWS CLI is not installed, you’ll see a message such as /usr/bin/which: no aws or aws not found.
Install AWS CLI
If you need to install the AWS CLI, we recommend following the AWS installation guide, which includes OS-specific instructions for Linux, macOS, and Windows as well as troubleshooting tips.
Export Your AWS Credentials
You can see our page Obtaining Credentials for information on finding your AWS credentials.
In your terminal, enter export AWS_ACCESS_KEY_ID="_____" with your AWS access key ID in the blank space.
Enter export AWS_SECRET_ACCESS_KEY="_____" with your AWS access key ID in the blank space.
Please be sure to include the quotation marks on both ends of your key IDs. There are characters inside AWS key IDs that, without quotation marks, systems will try to read as commands.
List Files in S3
In your terminal, enter aws s3 ls s3://bucket_name to display the files in your bucket. For this guide, we used a small text file named test.txt, so our command returned this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 ls s3://cloud-data-test
2023-02-08 16:37:56 28 test.txt
Transfer a File To/From S3
The AWS CLI mimics the Linux cp command for transferring files. To transfer a file, enter aws s3 SOURCE DESTINATION in your terminal.
SOURCE and DESTINATION can be local files or S3 buckets. S3 buckets are formatted as s3://bucket_name/file_name.
Below is an example of the aws s3 command.
In your terminal, enter aws s3 cp s3://bucket_name/file_name ./ to download a file from S3 storage to your current directory. You’ll see this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./
download: s3://cloud-data-test/test.txt to ./test.txt
To download a file from S3 storage to a specific directory, enter its absolute or relative path (e.g., /home/username/ or ./dir_relative_to_current_dir) in place of ./ with the aws s3 cp command. In the example below, we used the storage directory, which is located inside the current working directory:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 cp s3://cloud-data-test/test.txt ./storage
download: s3://cloud-data-test/test.txt to ./storage
To upload, simply reverse the order of SOURCE and DESTINATION in the aws s3 command.
Delete a File From S3
In your terminal, enter aws s3 rm s3://bucket_name/file_name to delete a file. You’ll see this message:
demo@pw-user-demo:~/pw/cloud-data-transfer$ aws s3 rm s3://cloud-data-test/test_upload.txt
delete: s3://cloud-data-test/test_upload.txt
Further Reading
For a comprehensive guide to installing the AWS CLI in different environments, please see this page of the AWS documentation.
To configure your AWS CLI settings, please see this page.
For the full list of AWS CLI transfer commands, please see this reference guide.