dcsimg
September 26, 2018
Hot Topics:

Migrating MongoDB to DynamoDB, Part 1

  • August 31, 2018
  • By Deepak Vohra
  • Send Email »
  • More Articles »

The AWS Database Migration Service (DMS) is designed to migrate databases on AWS reliably, with zero downtime. Initially, DMS supported only relational databases, including AWS Redshift. In April 2017, DMS added two NoSQL databases: MongoDB as a source database and AWS DynamoDB as a target database. In this tutorial of two articles, we shall discuss migrating a MongoDB database to DynamoDB on DMS. One of the requirements for using MongoDB as a DMS source is that MongoDB must be running as a replica set, which we shall create using a Docker image in the first of these two articles.

This article has the following sections:

Setting the Environment

The only pre-requisite is an AWS account, which may be created at https://aws.amazon.com/resources/create-account/. We shall run both the source and target databases on AWS. For the MongoDB source, we shall use Docker, for which we will launch an EC2 instance with AMI Container Linux by CoreOS (Stable) selected from the AWS Marketplace, as shown in Figure 1. CoreOS is chosen as the Linux platform because it has Docker pre-installed on it.

Selecting the CoreOS AMI to launch an EC2 Instance
Figure 1: Selecting the CoreOS AMI to launch an EC2 Instance

The Security Group used by the CoreOS EC2 instance must have Inbound/Outbound rules set to accept all traffic. This implies traffic of all protocol on all ports between all sources and destinations (0.0.0.0/0,::/0).

Creating an IAM User for Database Migration Service

In this section, we shall create an IAM user to access the different AWS services used in creating a migration, including DMS, EC2, DynamoDB, KMS, IAM, and CloudWatch. First, we need to create a Policy with the required permissions. Subsequently, we shall create a user and assign the policy to the user. To create an IAM policy, select Policies in the IAM console and click Create policy. In Create Policy, select Create Your Own Policy. In Review Policy, specify a Policy Name (DMS as an example) and copy and paste the following policy document in the Policy Document field.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": "dms:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "dynamodb:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "kms:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "iam:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "ec2:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "cloudwatch:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "aws-marketplace:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": "logs:*",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": [
            "redshift:Describe*",
            "redshift:ModifyClusterIamRoles"
         ],
         "Resource": "*"
      }
   ]
}

Click Validate Policy. If the output is "This policy is valid," click Create Policy, as shown in Figure 2.

Creating an IAM Policy
Figure 2: Creating an IAM Policy

A new IAM Policy gets created, as shown in Figure 3.

IAM Policy "DMS"
Figure 3: IAM Policy "DMS"

Next, create an IAM user. Select Users and click Add user, as shown in Figure 4.

Add user
Figure 4: Add user

In Add user, specify a User name, as shown in Figure 5. For Access type, select Programmatic access and AWS Management Console access.

Adding User
Figure 5: Adding User

For Console password, select Custom password and specify a password (see Figure 6). Click Next.

Select AWS Access Type>Next
Figure 6: Select AWS Access Type>Next

In Set permissions, click Attach existing policies directly, as shown in Figure 7.

Setting Permissions
Figure 7: Setting Permissions

Select the DMS policy created earlier, and click Next, as shown in Figure 8.

Selecting DMS Policy
Figure 8: Selecting DMS Policy

In Review, click Create user, as shown in Figure 9.

Review>Create user
Figure 9: Review>Create user

An IAM user gets created. Copy the URL shown in Figure 10 to log in to AWS Management Console as the user created.

IAM User URL
Figure 10: IAM User URL

A new user gets listed in Users (see Figure 11).

IAM User URL
Figure 11: IAM User URL

Creating an Encryption Key

Next, create an encryption key to be used for the DMS migration. Log in as the IAM user created and use the URL copied in Figure 10. Select the IAM service in the AWS management console and select Encryption Keys. Click Create key to start a wizard to create an encryption key. Use the wizard to create an encryption key (dms), as shown in Figure 12.

New Encryption Key
Figure 12: New Encryption Key

Creating a MongoDB Database

In this section, we shall create a MongoDB database which we shall migrate subsequently to DynamoDB. We shall be using Docker to run a MongoDB instance, for which a CoreOS instance was launched. To log in to a CoreOS instance, obtain the Public IP Address of the CoreOS instance, as shown in Figure 13.

Public IP Address of CoreOS Instance
Figure 13: Public IP Address of CoreOS Instance

SSH log in to the CoreOS instance using the key pair and the Public IP.

ssh -i "docker.pem" core@54.152.203.62

The CoreOS instance command-line prompt gets displayed, as shown in Figure 14.

CoreOS Instance
Figure 14: CoreOS Instance

Next, run the following command to start a Docker container for MongoDB using the MongoDB image "mongo". The Docker container port 27017 is exposed on the host also as 27017 using the -p option to docker run. The container name is set to "mongo1" and the command mongod --replSet repl0 is run in the container created to start a MongoDB replica set called "repl0". As mentioned before, to use MongoDB as a DMS source a MongoDB replica set is required and a standalone MongoDB is not usable as a source.

docker run \
-p 27017:27017 \
mongo mongod --replSet repl0

The Docker image mongo gets pulled and as indicated by the message "MongoDB starting" in Figure 15 MongoDB begins to gets started.

Downloading Docker Image docker
Figure 15: Downloading Docker Image docker

A MongoDB instance gets started on port 27017 (see Figure 16). A replica set has not yet been created and we shall initialize a replica set next.

Mongo Instance Started
Figure 16: Mongo Instance Started

A Docker container gets listed with the docker ps command, as shown in Figure 17.

Listing Docker Container for Mongo
Figure 17: Listing Docker Container for Mongo

Use the following command to start a command shell for the Mongo command line interface (CLI).

docker exec -it mongo1 mongo

MongoDB shell version 3.4.4 gets connected to at URL mongodb://127.0.0.1:27017, as shown in Figure 18.

Connecting MongoDB Shell
Figure 18: Connecting MongoDB Shell

The Mongo CLI command prompt gets displayed, as shown in Figure 19.

Mongo Shell Command Prompt
Figure 19: Mongo Shell Command Prompt

Set the MongoDB database to use as test with the use test command, as shown in Figure 20.

Setting Database as test
Figure 20: Setting Database as test

Next, we shall initialize a replica set for which we need to define the replica set members or instances. Obtain the Private IP of the CoreOS EC2 instance on which the Docker container for MongoDB is running (see Figure 21).

Private IP Of CoreOS Instance
Figure 21: Private IP Of CoreOS Instance

In the Mongo CLI, specify the following configuration for the replica set configuration.

config = {
   "_id" : "repl0",
   "members" : [
      {
         "_id" : 0,
         "host" : "172.30.2.20:27017"
      }
   ]
}

The replica set configuration gets set, as shown in Figure 22.

Setting Replica set Configuration
Figure 22: Setting Replica set Configuration

Initiate the replica set configuration using the configuration.

rs.initiate(config)

The replica set gets initialized, as shown in Figure 23.

Replica set Initialized
Figure 23: Replica set Initialized

Output the replica set configuration.

rs.conf()

The repl0:PRIMARY command prompt indicates that the replica set has been initialized and the replica set Primary member has been set to run Mongo CLI commands. The Primary is the only member in a replica set for write operations. Create a MongoDB collection called wlslog with the db.createCollection(<collection name>) command.

db.createCollection("wlslog")

A MongoDB collection gets created, as shown in Figure 24. A MongoDB collection is a collection of documents. The documents are in BSON (binary JSON) format.

Creating Collection
Figure 24: Creating Collection

Run the following statements that define JSON documents in the Mongo CLI.

doc1 = {"timestamp":"Apr 8, 2014 7:06:16 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000365",
   "msg":"Server state changed to STANDBY"}
doc2 = {"timestamp":"Apr 8, 2014 7:06:17 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000365",
   "msg":"Server state changed to STARTING"}
doc3 = {"timestamp":"Apr 8, 2014 7:06:18 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000365",
   "msg":"Server state changed to ADMIN"}
doc4 = {"timestamp":"Apr 8, 2014 7:06:19 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000365",
   "msg":"Server state changed to RESUMING"}
doc5 = {"timestamp":"Apr 8, 2014 7:06:20 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000331",
   "msg":"Started WebLogic Admin Server"}
doc6 = {"timestamp":"Apr 8, 2014 7:06:21 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000365",
   "msg":"Server state changed to RUNNING"}
doc7 = {"timestamp":"Apr 8, 2014 7:06:22 PM PDT",
   "category":"Notice","type":"WebLogicServer",
   "servername":"AdminServer","code":"BEA-000360",
   "msg":"Server started in RUNNING mode"}

The JSON documents' variables get defined, as shown in Figure 25.

Defining Variables for JSON Documents
Figure 25: Defining Variables for JSON Documents

Add the JSON documents to the wlslog collection.

db.wlslog.insert([doc1,doc2,doc3,doc4,doc5,doc6,doc7])

As indicated by the output in Figure 26, seven documents get added to the wlslog collection.

JSON Documents Added to Collection
Figure 26: JSON Documents Added to Collection

List the documents added to the wlslog collection.

db.wlslog.find()

The seven documents added get listed, as shown in Figure 27.

Finding or Getting Documents from a Mongo Collection
Figure 27: Finding or Getting Documents from a Mongo Collection

Creating a DynamoDB Table

Having created a MongoDB replica set for DMS source, next we shall create a DynamoDB table for the DMS target. Log in as the IAM user (dvohra) created earlier and assigned a policy. Select the DynamoDB service in the AW Management Console and select Create table, as shown in Figure 28.

DynamoDB>Create table
Figure 28: DynamoDB>Create table

In the Create DynamoDB table, specify a Table name and specify the Primary key, which is also the Partition key, as _id, as shown in Figure 29. Although the Table name is arbitrary and is set to wlslog, which is the same as the MongoDB collection created in MongoDB replica set, the Primary key must be set to _id because each MongoDB document gets assigned the primary key field _id.

Creating a DynamoDB Table
Figure 29: Creating a DynamoDB Table

The DynamoDB table wlslog gets created, as shown in Figure 30.

DynamoDB table wlslog Created
Figure 30: DynamoDB table wlslog Created

Click the DynamoDB table wlslog in the Dashboard and the table details, including the primary key _id, are displayed (see Figure 31).

DynamoDB table wlslog Detail
Figure 31: DynamoDB table wlslog Detail

When a DMS migration is created, an IAM role dms-vpc-role with managed policy AmazonDMSVPCManagementRole gets created automatically. For the DMS service to access the DynamoDB service, we need to modify the service access role dms-vpc-role to add the following policy document, which provides access to DynamoDB from DMS.

{
   "Version": "2012-10-17",
   "Statement": [{
      "Effect": "Allow",
      "Action": [
         "dynamodb:*"
      ],
      "Resource": ["*"]
   }]
}

Using the same procedure used for creating the DMS policy, create a policy DynamoDB and specify the preceding policy document in the Policy Document field-box, as shown in Figure 32. Click Create Policy.

Review Policy>Create Policy
Figure 32: Review Policy>Create Policy

The DynamoDB policy gets created, as shown in Figure 33.

IAM Policy DynamoDB Created
Figure 33: IAM Policy DynamoDB Created

The dms-vpc-role to which the DynamoDB policy is to be added is shown in Figure 34.

DMS VPC Role
Figure 34: DMS VPC Role

Click dms-vpc-role and add the DynamoDB policy using Attach Policy. The AmazonDMSVPCManagementRole and DynamoDB policies should be listed as Managed Policies, as shown in Figure 35.

Permissions Policies in DMS VPC Role
Figure 35: Permissions Policies in DMS VPC Role

Conclusion

In this article, we introduced using AWS Database Migration Service (DMS) for migrating MongoDB to Amazon DynamoDB. We started by creating a MongoDB replica set as the data source to migrate and also created a DynamoDB table as destination table. In a subsequent article, we shall discuss creating and running a DMS migration to migrate data.






Comment and Contribute

 


(Maximum characters: 1200). You have characters left.

 

 


Enterprise Development Update

Don't miss an article. Subscribe to our newsletter below.

By submitting your information, you agree that developer.com may send you developer offers via email, phone and text message, as well as email offers about other products and services that developer believes may be of interest to you. developer will process your information in accordance with the Quinstreet Privacy Policy.

Sitemap

Thanks for your registration, follow us on our social networks to keep up-to-date