AWS CLI S3 Commands

Hello, In this post we will see some of the commands that will make your job easier while working with AWS S3 buckets.

What is AWS S3?

Amazon Simple Storage Service also known as Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 is designed for 99.999999999% (11 9's) of durability, and stores data for millions of applications for companies all around the world.

To know more about AWS S3 you can read here.

Commands

1. Create S3 Bucket in Default Region

Enter the following command to create a new bucket in the default region you have specified while configuring the AWS CLI. If the bucket name is already owned by you or someone else, you will get the error stating that.

aws s3 mb s3://bucket_name

2. Create S3 Bucket in Another Region

Enter the below command to create a new bucket in the region you have specified after the region flag. You can find the region names on the AWS website.

aws s3 mb s3://bucket_name --region region-name

3. Delete Empty S3 Bucket

Make sure the bucket you want to delete is empty or else you will get an error saying that bucket is not empty. Make sure the bucket exists in your account.

aws s3 rb s3://bucket_name

4. Delete Non-Empty S3 Bucket

To delete the non-empty bucket make sure to put the force flag after the bucket name.

aws s3 rb s3://bucket_name --force

5. List All S3 Bucket

Use the ls command in the following way to list all the buckets in your account.

aws s3 ls 

OR

aws s3 ls s3://

6. List All Objects in S3 Bucket

To list all objects in the bucket use the ls command and specify the bucket name in the command. It will display the list of files and folders in the S3 bucket.

aws s3 ls s3://bucket_name

7. List All  objects in S3 Bucket Recursively

To list all objects recursively use the ls command and specify the bucket name in the command. It will display the list of files and folders and all of its contents recursively.

aws s3 ls s3://bucket_name --recursive

8. Copy File to S3 Bucket

To copy file from the local machine to the S3 bucket. Make sure you enter the correct file location which you want to copy and also the location where you want to copy in the bucket.

aws s3 cp /home/file_to_copy.txt s3://bucket_name

9. Copy Folder to S3 Bucket

To copy the whole folder to the S3 bucket use the recursive flag after the bucket name. Make sure to enter the correct folder location.

aws s3 cp /home/folder_to_copy s3://bucket_name --recursive

10. Download File from S3 Bucket

To download file from an S3 bucket use the following command. This copies file.txt from the S3 bucket to the directory specified on the local machine. To download the file on the local machine with a different name, you can enter the new name in the destination field.

aws s3 cp s3://bucket_name/file.txt /home/downloads

11. Download All Files from S3 Bucket

To download all the files from an S3 bucket use the following command. This copies all the files and sub-folders present in the S3 bucket to the directory specified on the local machine.

aws s3 cp s3://bucket_name /home/project --recursive

12. Copy File Among S3 Buckets

To copy file among S3 buckets, use the following command. This copies file.txt from the source S3 bucket to the target S3 bucket. To copy file on the target bucket with a different name, you can enter the new name in the destination field with the target bucket name.

aws s3 cp s3://source_bucket/file.txt s3://target_bucket

13. Copy All Files Among S3 Buckets Recursively

To copy all the files among the S3 buckets, use the following command. This copies all the files and sub-folders present in the source S3 bucket to the destination S3 bucket.

aws s3 cp s3://source_bucket s3://target_bucket --recursive

14. Find Size Objects in S3 Bucket

To display the size of objects present in the S3 bucket. Use the recursive flag to display the size of all the files and sub-folders, a human-readable flag to display the size of the file in a readable format and summarize flag to display the last two lines in the output with the total number of objects in the S3 bucket and the total size of all those objects.

aws s3 ls s3://bucket_name --recursive --human-readable --summarize

15. Delete a File from S3 Bucket

To delete a file from the S3 bucket use the rm command and specify the file location in the S3 bucket.

aws s3 rm s3://bucket_name/file.txt

16. Delete All Files from S3 Bucket

To delete all the files from an S3 bucket, use the –recursive flag. This will clear the bucket.

aws s3 rm s3://bucket_name --recursive

17. Move File to S3 Bucket from Local

To move a file to the S3 bucket from the local machine, use the following command. To move file on the S3 bucket with a different name, you can enter the new name in the destination field with the location of the file on the bucket.

aws s3 mv file.txt s3://bucket_name

18. Move File from S3 Bucket to Local

To move a file to the local machine from the S3 bucket, use the following command. To move file on the local machine with a different name, you can enter the new name in the destination field with the location.

aws s3 mv s3://bucket_name/file.txt /home/downloads

19. Move Folder to S3 Bucket Recursively

To move all files and sub-folders recursively to S3 bucket from local machine use the following command. Add recursive flag to the move command.

aws s3 mv /home/folder s3://bucket_name/folder --recursive

20. Move Folder from S3 Bucket to Local Recursively

To move all files and sub-folders recursively to local machine from S3 bucket use the following command. Add recursive flag to the move command.

aws s3 mv s3://bucket_name/folder /home/downloads/folder --recursive

21. Move File among S3 Buckets

To move a file from the target bucket to the source bucket use the following command. To move file on source bucket with different name, you can enter the new name in the destination field with the location on the source bucket.

aws s3 mv s3://source_bucket/file.txt s3://target_bucket

22. Move Folder among S3 Buckets

To move all files and sub-folders among the S3 buckets use the following command. This will move all the files and sub-folder and it's contents to the target bucket from the source bucket.

aws s3 mv s3://source_bucket s3://target_backup --recursive

23. Sync Files from Local to S3 Bucket

To sync files from the local machine to the S3 bucket, use the following command with the appropriate location of files. It will only upload the file which doesn't exist or is not updated on the S3 bucket.

aws s3 sync /home/folder s3://bucket_name

24. Sync Files from S3 Bucket to Local

To sync files from the S3 bucket to local machine, use the following command with the appropriate location of files. It will only download the file which doesn't exist or is not updated on to the local machine.

aws s3 sync s3://bucket_name /home/downloads

25. Sync Files among S3 Buckets

To sync files among two S3 buckets, use the following command.

aws s3 sync s3://source_bucket s3://target_bucket

26. Host website on S3 Bucket

We can also host a static website on S3 bucket. Just upload the files to the bucket and specify the index and error page name in the command. If your bucket is in the us-east-1 region, then you can access your website on the URL: http://bucket_name.s3-website-us-east-1.amazonaws.com

aws s3 website s3://bucket_name/ --index-document index.html --error-document error.html

For more details on AWS CLI you can visit official documentation.

I hope you would have found these commands useful.

Stay Home Stay Safe.

Bbyee 👋