AWS

How to Work With Amazon S3 Event Notifications?

Amazon Simple Storage Service (S3) was introduced by AWS in the year of 2006. The core purpose of the S3 bucket is to store objects in the Cloud for long-term usage. While taking the advantage of cloud infrastructure of AWS, it offers industrial-leading scalability, encryption, high performance, and data availability, along with versioning to recover and restore the objects. With its flexibility to grow and shrink in size in compliance with the application’s requirements, users can upload files of any format to S3 bucket such as media content, source codes, spreadsheets, etc. Customers of all sizes are actively using this service as it provides support for multiple use cases such as data lakes, web applications, mobile applications, database backups, etc.

Read more: What is S3 Bucket?

Quick Outline

What is Event Notification in S3 Bucket?

Events are referred to as any change in the current state of application. In the S3 bucket, these events can be generated when an object is uploaded, deleted, or copied, etc. Event notification in the S3 bucket alerts the users whenever any change in the state of an object is observed. These notifications can be either received on the email address or can be monitored by the CloudWatch logs.

Why Should Event Notifications be Used in S3 Bucket?

The S3 bucket provides the capability of storing an unlimited number of objects (data). The maximum size limit for the objects in the S3 bucket is 5TB of memory. As the number of these objects grows in the bucket, it becomes quite a challenge for the administrators and developers to monitor the state of objects in the S3 bucket. For example, an application that is running in association with the data uploaded to the S3 bucket will collapse if any object is removed or modified.

This functionality can lead to a drastic turn of events when there are huge volumes of data (objects) to be monitored. This is where Event notification comes into play. With this functionality, a user will be notified of every change in the state of the bucket; be it uploading, deleting, creating, or modifying the object. By implementing this feature, a user can save time, effort, and cost of manually monitoring the state of the S3 bucket.

Which Events Can Publish Notifications in S3 Buckets?

Events that are supported by the Event Notification Functionality are listed below:

  • Create Object.
  • Delete an Object.
  • Restore an Object.
  • Replication events.
  • S3 Lifecycle expiration and transition events.
  • Object ACL PUT events.
  • RRS object lost events.

How to Work With Amazon S3 Event Notification?

As the S3 bucket supports multiple services of AWS such as Lambda, SNS topic, RDS, etc., event notification can be implemented by using the following methods:

Method 1: S3 Event Notification Using Amazon EventBridge

In this section of the blog, we will implement Event notification in the S3 bucket using Amazon EventBridge. Below are the steps mentioned in which we can implement this functionality:

Step 1: Create the S3 Bucket

The first step in implementing the Event Notification by using the EventBridge service is to create the S3 bucket. Visit the AWS Management Console and search the “S3” service in the search bar. Click on the service name from the displayed results:

Learn more: Getting Started With Buckets: Overview of Amazon Simple Storage Service

On the S3 Dashboard, click the “Create bucket” button:

In the General Configuration section, provide a globally unique identifier for the S3 bucket in the “Bucket name” text field. Users can also select the region for the deployment of the S3 bucket. At the moment, we are not changing the default AWS region of the bucket:

By keeping the rest of the settings as default, click on the “Create bucket” button at the bottom of the interface:

The bucket has been created successfully:

Step 2: Enable EventBridge

Choose the name of the S3 bucket from the Dashboard:

From the bucket’s console, click the “Properties” tab to implement the S3 event notification:

Scroll down to the “Event notifications” section and click the “Edit” button:

On the next interface, select the “On” option and hit the “Save changes” option:

The event notifications are successfully edited:

Step 3: Create the SNS topic

To create the SNS topic, search for the “SNS” service from the AWS Management console. Click on the “Simple Notification Service” from the displayed results:

Read more: What is Amazon SNS and How to Use it?

On the SNS topic dashboard, click on the “Create topic” button:

In the “Details” section, there are two types of SNS topics i.e., FIFO and Standard.

  • Standard: The “Standard” type allows multiple subscription protocols such as SQS, Lambda, etc.
  • FIFO: This type of SNS topic only allows one subscription protocol such as SQS.

For the demo, select the “Standard” option:

Provide a globally unique and meaningful name for the SNS topic in the highlighted text field:

Scroll down to the “Access policy – optional” section and select the following highlighted option. By specifying these options, only the topic owner can publish the message. Similarly, the accounts that are associated with the topic owner’s account and the topic owner itself can subscribe to the topic:

By keeping the rest of the settings as default, click on the “Create topic” button at the bottom of the interface:

The topic has been created successfully:

Step 4: Create Subscription

Scroll down the SNS topic console and tap the “Subscription” tab from the various options displayed. Tap the “Create subscription” button:

Learn more: How to Use Amazon Simple Notification Service?

From the displayed interface, select “Email” as the protocol. Provide the email in the “Endpoint” text field. This email will receive all the alerts for the S3 bucket:

By keeping the defaults, click on the “Create subscription” button:

The subscription has been created successfully:

In the “Details” section of the Subscription interface, the status remains “Pending confirmation”:

To update the status to “Confirmed”, open the Email account that you have provided for the SNS topic. Choose the “AWS Notifications” email and tap the link provided by AWS for confirmation:

After clicking the subscription link provided by AWS, the following interface will be displayed to you:

Revisit the Subscription dashboard to view the updated status of the subscription. For this purpose, tap the “Subscription” option from the left navigation pane of the SNS topic. The status of the subscription is now updated to “Confirmed”:

Step 5: Create EventBridge Rules

In this section, we will now create the EventBridge Rules. To implement EventBridge rules, visit the Service’s console by searching the “EventBridge” in the search bar of the AWS Management Console. Click on the service’s name from the displayed results:

Read more: What is Amazon EventBridge and How to Use it?

Click the “Rules” option under the “Buses” section from the left navigation pane of Amazon EventBridge:

Scroll down to the “Rules” section. Click the “Create rule” button:

Within the Rule detail section, provide an identifier for the rule in the Name text field. The description is the optional text field. For the Event bus, pick the “default” option:

Select the “Rule with an event pattern” option as S3 will create an event notification in this method. Tap the “Next” button situated at the bottom of the interface:

On the next interface in the Event source section, pick the highlighted option from the “Event source” section:

Scroll down to the “Sample event- optional” section and choose the “AWS events” option. From the “Sample events” text field, select the “Object Created” option. This will send the notification to the specified email when an object is uploaded to the S3 bucket:

Navigate to the “Creation method” section and pick the “Use pattern form” option:

Scroll down to the “Event pattern” section. In the Event source text field, select the “AWS services” option as the events will be created by the S3 bucket. Similarly, specify the “Simple Storage Service (S3)” option in the AWS service text field. For the “Event type”, select the “Amazon S3 Event Notification” option:

Within the “Event pattern” section, specify the “Specific event(s)” option in the “Event Type Specification 1” block. This is because we only want to send the user notification when any object is uploaded (created) to the S3 bucket. Similarly, for the “Event Type Specification 2”, specify the bucket’s name in the “Specific buckets(s) by name” text field:

After these modifications, tap the “Next” button situated at the bottom of the interface:

Next comes the “Target 1” section. Specify the “AWS service” option in the Target types. Similarly, select the “SNS topic” as the target and specify the name of the SNS topic. After these configurations, click the “Next” button to proceed further:

Users can specify tags for the EventBridge rule. At the moment, click the “Next” button to proceed further:

After carefully reviewing the information for the EventBridge rule, tap the “Create rule” button:

The rule has been created successfully:

Step 6: Verification

For verification, navigate to the S3 dashboard and click the name of the S3 bucket:

On the S3 console, click the “Upload” button to create an event:

To upload the file to the S3 bucket, click the “Add files” option. Select a file from your device and upload it to the S3 bucket:

Tap the “Upload” button loaded at the bottom of the interface:

The object has been successfully uploaded to the S3 bucket:

Open the email account provided at the time of creating the subscription for the SNS topic. Within this account, there will be an email from AWS alerting the user about the object uploaded to the S3 bucket:

Bonus Tip: Event Notification for Encrypted SNS topic Using EventBridge

Within large enterprises, the SNS topic is also encrypted for security purposes. Therefore, to implement Event Notifications for the encrypted SNS topic, follow the below-mentioned steps carefully:

  • Step 1: Create the S3 Bucket
  • Step 2: Create Encryption Key
  • Step 3: Create the SNS Topic
  • Step 4: Verification

Step 1: Create an S3 bucket.

To create the S3 bucket, follow the steps mentioned in Method 1 for creating the S3 bucket.

Learn more: How to Create a Bucket in AWS S3

Step 2: Create Encryption key

To create the Encryption key, search and select the “KMS” service from the AWS Management Console:

Read more: How to Encrypt Data at Rest in S3 Using KMS

From the KMS console, tap the “Create a key” option:

Within the next interface, select the “Symmetric” option for the Key type. In the “Key usage” section, pick the “Encrypt and decrypt” option:

Tap the “Next” option to proceed to the next step for creating the KMS key:

Provide the name for the key in the “Alias” text field:

Tap the “Next” option from this interface:

For the Key administrator, select the user from the displayed list. The user can create a specific role for it too or can also select the root user. At the moment, we are selecting the user that has been previously created:

In the Key deletion section, check the following option. This will allow the specified user to delete the encryption key at any moment. This option is enabled as default. For this demo, keeping the default, click the “Next” button:

On the next interface, specify the user that can use this key with another service for cryptographic operation. For this demo, select the user that was previously specified for the “Key administrator”:

Scroll down to the end of the interface and click the “Next” button:

In the Review section, scroll down to the “Key policy” section. Paste the following policy to allow “decrypt” and “GenerateDatakey” permissions. The permission is to be added before any new statement:

{
            "Sid": "Allow CWE to use the key",
            "Effect": "Allow",
            "Principal": {
                "Service": "events.amazonaws.com"
            },
            "Action": [
                "kms:Decrypt",
                "kms:GenerateDataKey"
            ],
            "Resource": "*"
        },

After carefully reviweing the information provided for the KMS, click the “Finish” button:

The key has been successfully created:

Step 3: Create the SNS topic

For creating the SNS topic, follow the steps mentioned in Method 1. After creating the SNS topic, click the Edit button:

Read more: What is Amazon SNS and How to Use it?

Locate the “Encryption” option and then specify the key in the “AWS KMS key” text field:

After the required modifications, click the “Save changes” option located at the bottom of the interface:

The modifications are successfully configured:

Step 4: Verification

To verify this, upload a file to the S3 bucket. For uploading the file to the S3 bucket, follow the steps mentioned in Step 6 of this Method Here, we have uploaded the file to the S3 bucket:

Open your Email account which was provided at the time of creating the “SNS” topic. Here, an email by the name of “AWS notification” will be sent by the AWS:

That is all from this section of the blog.

Method 2: S3 Event Notification Using AWS Lambda

The S3 bucket can be used with AWS Lambda for event notifications. Following are the steps for implementing Event notification in the S3 bucket using AWS Lambda:

  • Step 1: Create the S3 Bucket
  • Step 2: Create a Lambda Function
  • Step 3: Implement Event Notifications
  • Step 4: Verification

Step 1: Create the S3 Bucket

To create the S3 bucket, follow the steps mentioned in Method 1 for creating the S3 bucket.

Step 2: Create a Lambda Function

Search the “Lambda” service in the search bar of the AWS Management Console. Select the service name from the displayed results:

Read more: How to Use AWS Lambda

From the Lambda interface, click the “Create function” button:

Choose the “Author from scratch” option and then provide a name for the Lambda Function in the “Function name” text field:

For the Runtime, select the “Python 3.11” option from the drop-down list:

Read more: How to Use AWS Lambda Function in Python

Click the “Create function” button at the end of the interface:

The Lambda Function has been created successfully:

Scroll down to the “Code source” section. Modify the Lambda function by adding the following line of code. After modification to the Lambda Function, click the “Deploy” button to save and apply changes:

Step 3: Implement Event Notification

Head back to the S3 dashboard and choose the bucket by clicking on the bucket’s name:

Tap the “Properties” tab from the S3 dashboard:

Scroll down the “Properties” interface and locate the “Event notification” section. Tap the “Create event notification” button:

In the General configuration section, give a unique name of the event in the “Event name” text field. Users can also specify prefixes and suffixes if any:

In the “Event types”, select “All object create events” in the Object creation section. By enabling this feature, events will generated whenever an object is uploaded, copied, etc:

Navigate to the “Destination” section on the Event notification interface. From this interface, select the “Lambda function” option from the Destination:

Within the Destination section, select the highlighted option under the “Specify Lambda function” block. Choose the Lambda function from the drop-down list and hit the “Save changes” button to apply changes:

Here, the event notification has been configured successfully:

Step 4: Verification

To verify if the events are created, click on the bucket’s name to upload a file. Click on the “Upload” button. Here, the file has been uploaded to the bucket successfully:

Visit the Lambda function interface and click on the function’s name:

Within the function’s interface, click the “Monitor” tab:

From this interface, tap the “View CloudWatch logs” button:

Read more: What are CloudWatch Logs in AWS?

Scroll down the interface to the “Log streams” section and click on the stream to view the events:

Here, we can see that the event has been printed along with the details of the bucket and the object uploaded:

That is all from this section.

Method 3: S3 Event Notification Using SNS

Event notifications in the S3 bucket can also be implemented using the SNS topic. For this purpose, create the S3 bucket for storing objects and the SNS topics for a subscription. The SNS topic will contain a subscription while the event notification feature in the S3 bucket will be enabled. This feature of the S3 bucket will subscribe to the SNS topic. So that every change in the state of the bucket will be detected and determined by the SNS topic. The user will be alerted via the provided email in the SNS topic. For the practical implementation, refer to this article: “How to Use Amazon S3 Event Notification”

Method 4: S3 Event Notifications Using SQS

The Last method in the list of configuring the Event notification in the S3 bucket is using the SQS. For this purpose, the S3 bucket will be required for storing the object upon which the event notifications will occur. Similarly, two queues will be created using SQS. These two queues will be attached to the S3 bucket using the Event Notification feature. So whenever the user uploads a file to the S3 bucket, the event notifications will send the message to the user via queues. For a practical demonstration of it, refer to this article: “ How to Configure Event Notification on S3 Bucket”.

Conclusion

To enable event notifications, there are four ways to configure the S3 bucket with event notifications i.e., EventBridge rules, Lambda Functions, SNS, and SQS. The Event notification will alert the user either on the provided emails or by invoking the Lambda Functions whenever a change is observed in the state of Bucket. This change is referred to as events and it can be uploading a file to the buckets, deleting or modifying the bucket, etc. This article presents a comprehensive step-by-step demonstration for implementing the S3 event notifications in AWS.

About the author

Shameen Shahid

I am a self-motivated technical content writer. I hold a bachelor’s degree in computer science an have expertise in AWS and want to share my knowledge with the world.