AWS

How to transfer files from EC2 to S3

EC2 (elastic compute cloud) is a scalable compute service provided by AWS. You can use the AWS EC2 service to launch virtual servers on the cloud in seconds or minutes. AWS provides hardware-level selection while initializing an EC2 instance (EC2 virtual machine). You can select hardware architecture, operating system, storage, and different configurations of CPU and memory depending upon your requirements.
S3 (a simple storage service) is a highly available and scalable storage service provided by AWS. It provides almost infinite storage, but you will only be charged for what you use from this storage. S3 stores your data in multiple places so you do not lose your data in case of disaster. That is why S3 can be used to backup your important data.
While working on EC2, sometimes you need to store some important data from EC2 instance to S3 regularly, like database backups or some other important data as a backup because EC2 servers may crash on AWS, and you may lose all your data. In this blog, we will discuss how we can move important data from EC2 to S3.

Installing awscli package

First, you need to install the awscli package on your EC2 instance. The awscli package is used to interact with AWS using the command line interface. Check the version of the awscli package to verify if it is already installed or not.

ubuntu@ubuntu:~$ aws --version

To install the awscli package, first, download the packaged zip file using the curl command.

ubuntu@ubuntu:~$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

Unzip the package using the unzip command.

ubuntu@ubuntu:~$ unzip awscliv2.zip

Install the awscli package using the following command.

ubuntu@ubuntu:~$ sudo ./aws/install

Enable S3 access on EC2 instance

After installing the awscli package, now enable S3 access on the EC2 instance so the EC2 instance can store data to S3. There are two ways to grant access to the EC2 instance. You can use any of them to grant access.

  • Grant access using IAM role (recommended way)
  • Grant access using Access key ID

Grant access using IAM role (recommended way)

EC2 instances can be granted access to upload files on S3 using the IAM role. An IAM role with access to upload data on S3 is created and attached to the EC2 instance.

NOTE: Never grant extra permissions using the IAM role. If someone else gets access to your EC2 instance, he can use it to provide extra resources to your account.

To create an IAM role, first, create an IAM policy with specific permissions. Login to the AWS management console and search for IAM in the search bar.

Click on the ‘Policies’ from the left side panel under ‘Access management’.

Now click on the ‘Create Policy’ button appearing on the right side.

It will display a page to create a policy. Select the ‘Visual editor’ tab from the top.

From the visual editor, select S3 as service, PutObject under the writing category as action, and All Resources as a resource.

After specifying the service, action, and resource, now click on the ‘Next’ button at the bottom right corner.

The tags are optional and can be skipped by clicking on the ‘Next’ button at the bottom right corner.

Add the name of the policy on the review page and click on the ‘Create Policy’ button to create the policy.

After creating the IAM policy, click on the ‘roles’ from the left side panel in the IAM console.

Click on the ‘create role’ button to create a new role.

Select ‘AWS service’ as a trusted entity and ‘EC2’ as a use case and click on the ‘Next’ button to add permissions.

For permissions, select the IAM policy created in the previous step and click on ‘Next’.

Add the role name and click on the ‘create role’ button to create the role.

Now the IAM role has been created; it is time to attach it to the EC2 instance. Search for the EC2 in the AWS management console.

Click on the ‘instances’ from the left side panel, and it will display all the instances.

Select the instance you want to grant access to upload files on S3 and click on the ‘Actions’ button on the top right corner of the console. Select the security > Modify IAM role from the drop-down menu.

Select the previously created IAM role and click on the save button. Now the EC2 instance has been granted access to upload files on S3.

Grant access using Access key ID

To grant access to EC2 an instance using the access key, first generate a new access key from the IAM console. From the IAM console, click on the ‘Users’ under the ‘Management access’ from the left side panel.

Click on your user account and go to the ‘security credentials’ tab from the user’s list.

Under the ‘security credentials’ tab, click on the ‘create access key’ to generate a new access key.

Download the csv file containing the access key id and secret access key.

After generating the Access key ID and secret access key, log into the EC2 instance using SSH and configure the access key.

ubuntu@ubuntu:~$ aws configure

It will ask for an access key ID and the secret access key. Provide the credentials we just generated.

Now the EC2 instance has access to upload the files on S3 using the command line interface.

Upload files to S3 from EC2

Before uploading the files to S3, first, create an S3 bucket. From the management console, search for S3.

From the S3 console, click on the ‘create bucket’ button.

Enter the name and the region of the bucket, leave the rest of the settings by default and create the bucket. The name of the S3 bucket must be universally unique.

Now log into your EC2 instance using SSH and upload the file to S3 using the command line interface. The syntax to upload the file to S3 is as follows.

ubuntu@ubuntu:~$ aws s3 cp [source file] [destination on S3] --region [s3 bucket region]

To copy a file named file.txt to S3, use the following command.

ubuntu@ubuntu:~$ aws s3 cp file.txt s3://linuxhint-demo-bucket/ --region us-east-1

To verify if the file is present in the S3 bucket or not, use the following command.

ubuntu@ubuntu:~$ aws s3 ls s3://linuxhint-demo-bucket/ --region us-east-1

The file has been copied to the S3 bucket. Instead of copying a file to S3, we can also move the file to S3.

ubuntu@ubuntu:~$ aws s3 mv new-file.txt s3://linuxhint-demo-bucket/ --region us-east-1

Conclusion

Files from EC2 instances can be saved as a backup by uploading them on S3 (simple storage service). This blog describes the procedure to upload files from EC2 to S3 using two different ways, i.e., using the IAM role and Access key ID. After reading this blog, I hope you can easily transfer files from EC2 to S3 using either way.

About the author

Zain Abideen

A DevOps Engineer with expertise in provisioning and managing servers on AWS and Software delivery lifecycle (SDLC) automation. I'm from Gujranwala, Pakistan and currently working as a DevOps engineer.