AWS : S3 (Simple Storage Service) VII - Cross-Region Replication, how to Copy or Move Objects from one region to another
In this tutorial, we'll learn how to copy or move S3 Objects from one region to another. We'll use Cross-Region Replication for the case when the source and destination buckets are owned by the same AWS account.
We'll assume we already owned one bucket (source). So, we need to create another bucket (destination) in a different AWS region, enable versioning on both the buckets, and then configure cross-region replication on the source bucket.
Once enabled, new objects uploaded to a particular S3 bucket are automatically replicated to a designated destination bucket located in a different AWS region. The replication process also copies any metadata and ACLs (Access Control Lists) associated with the object and can be enabled and managed through the S3 API.
Existing objects will not be replicated. Cross-Region Replication replicates every future upload of every object to another bucket.
Cross-region replication is the automatic, asynchronous copying of objects across buckets in different AWS regions. By activating cross-region replication, Amazon S3 will replicate newly created objects, object updates, and object deletions from a source bucket into a destination bucket in a different region. Cross-region replication has specific requirements that define what can and cannot be replicated across regions based on how the object is created and how it is encrypted.
How to configure Cross-Region Replication (Source and Destination Buckets Owned by the Same AWS Account)?
Create a destination bucket in another AWS region, say, Tokyo:
Enable versioning on the bucket, we can get the menu by clicking on the Properties icon on the left of the bucket name:
Enable cross-region replication on the source bucket. We can decide if we want to replicate all objects or only objects with a specific prefix.
In the Properties pane, click Cross-Region Replication.
- Choose the Source-either the entire bucket or a prefix within the bucket.
- Choose the Destination Region from the drop-down list.
- Choose the Destination Bucket if we do not see our desired destination bucket in the list, confirm that the bucket exists in the region we selected above, and that we have enabled versioning on that bucket. If no buckets exist in that region, and we click Create a new bucket from the dropdown list, we'll be prompted to create a new bucket with versioning enabled in that destination region.
- In order to perform cross-region replication of objects on our behalf, Amazon S3 will need to use an IAM role that we have created. Click Create/Select IAM Role and a new browser tab will open up within the AWS Identity and Access Management (IAM) console. On that page, we'll select an existing IAM role or create a new one that will allow Amazon S3 to replicate objects from the source bucket to the destination bucket on our behalf. By default, Amazon S3 will generate a policy document for the IAM role that matches the source and destination buckets we've chosen. To continue, click Allow to return to the Amazon S3 console.
- Click Save.
Now we want to verify that Amazon S3 replicated the objects in the destination bucket.
Time it takes for Amazon S3 to replicate an object depends on the object size.
Also, note that "Existing objects will not be replicated. Cross-Region Replication replicates every future upload of every object to another bucket."
AWS (Amazon Web Services)
- AWS : EKS (Elastic Container Service for Kubernetes)
- AWS : Creating a snapshot (cloning an image)
- AWS : Attaching Amazon EBS volume to an instance
- AWS : Adding swap space to an attached volume via mkswap and swapon
- AWS : Creating an EC2 instance and attaching Amazon EBS volume to the instance using Python boto module with User data
- AWS : Creating an instance to a new region by copying an AMI
- AWS : S3 (Simple Storage Service) 1
- AWS : S3 (Simple Storage Service) 2 - Creating and Deleting a Bucket
- AWS : S3 (Simple Storage Service) 3 - Bucket Versioning
- AWS : S3 (Simple Storage Service) 4 - Uploading a large file
- AWS : S3 (Simple Storage Service) 5 - Uploading folders/files recursively
- AWS : S3 (Simple Storage Service) 6 - Bucket Policy for File/Folder View/Download
- AWS : S3 (Simple Storage Service) 7 - How to Copy or Move Objects from one region to another
- AWS : S3 (Simple Storage Service) 8 - Archiving S3 Data to Glacier
- AWS : Creating a CloudFront distribution with an Amazon S3 origin
- AWS : Creating VPC with CloudFormation
- AWS : WAF (Web Application Firewall) with preconfigured CloudFormation template and Web ACL for CloudFront distribution
- AWS : CloudWatch & Logs with Lambda Function / S3
- AWS : Lambda Serverless Computing with EC2, CloudWatch Alarm, SNS
- AWS : CLI (Command Line Interface)
- AWS : CLI (ECS with ALB & autoscaling)
- AWS : ECS with cloudformation and json task definition
- AWS Application Load Balancer (ALB) and ECS with Flask app
- AWS : Load Balancing with HAProxy (High Availability Proxy)
- AWS : VirtualBox on EC2
- AWS : NTP setup on EC2
- AWS & OpenSSL : Creating / Installing a Server SSL Certificate
- AWS : OpenVPN Access Server 2 Install
- AWS : VPC (Virtual Private Cloud) 1 - netmask, subnets, default gateway, and CIDR
- AWS : VPC (Virtual Private Cloud) 2 - VPC Wizard
- AWS : VPC (Virtual Private Cloud) 3 - VPC Wizard with NAT
- DevOps / Sys Admin Q & A (VI) - AWS VPC setup (public/private subnets with NAT)
- AWS - OpenVPN Protocols : PPTP, L2TP/IPsec, and OpenVPN
- AWS : Autoscaling group (ASG)
- AWS : Setting up Autoscaling Alarms and Notifications via CLI and Cloudformation
- AWS : Adding a SSH User Account on Linux Instance
- AWS : Windows Servers - Remote Desktop Connections using RDP
- AWS : Scheduled stopping and starting an instance - python & cron
- AWS : Detecting stopped instance and sending an alert email using Mandrill smtp
- AWS : Elastic Beanstalk with NodeJS
- AWS : Elastic Beanstalk Inplace/Rolling Blue/Green Deploy
- AWS : Identity and Access Management (IAM) Roles for Amazon EC2
- AWS : Identity and Access Management (IAM) Policies
- AWS : Creating IAM Roles and associating them with EC2 Instances in CloudFormation
- AWS Identity and Access Management (IAM) Roles, SSO(Single Sign On), SAML(Security Assertion Markup Language), IdP(identity provider), STS(Security Token Service), and ADFS(Active Directory Federation Services)
- AWS : Amazon Route 53
- AWS : Amazon Route 53 - DNS (Domain Name Server) setup
- AWS : Amazon Route 53 - subdomain setup and virtual host on Nginx
- AWS Amazon Route 53 : Private Hosted Zone
- AWS : SNS (Simple Notification Service) example with ELB and CloudWatch
- AWS : Lambda with AWS CloudTrail
- AWS : SQS (Simple Queue Service) with NodeJS and AWS SDK
- AWS : Redshift data warehouse
- AWS : CloudFormation
- AWS : CloudFormation Bootstrap UserData/Metadata
- AWS : CloudFormation - Creating an ASG with rolling update
- AWS : Cloudformation Cross-stack reference
- AWS : OpsWorks
- AWS CodeDeploy : Deploy an Application from GitHub
- AWS EC2 Container Service (ECS)
- AWS EC2 Container Service (ECS) II
- AWS Hello World Lambda Function
- AWS Node.js Lambda Function & API Gateway
- Amazon Kinesis Streams
- AWS: Kinesis Data Firehose with Lambda and ElasticSearch
- Amazon DynamoDB
- Amazon ML (Machine Learning)
- Simple Systems Manager (SSM)
- AWS : RDS Connecting to a DB Instance Running the SQL Server Database Engine
- AWS : RDS Importing and Exporting SQL Server Data
- AWS : RDS PostgreSQL & pgAdmin III
- AWS : RDS PostgreSQL 2 - Creating/Deleting a Table
- AWS : MySQL Replication : Master-slave
- AWS : MySQL backup & restore
- AWS RDS : Cross-Region Read Replicas for MySQL and Snapshots for PostgreSQL
- AWS : Restoring Postgres on EC2 instance from S3 backup
- AWS : Scaling-Up
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization