Dynamodb Bulk Insert: an Easy Tutorial

In this article, we’ll show how to do bulk inserts in DynamoDB. If you’re new to Amazon DynamoDB, start with these resources:

• Introduction to Amazon DynamoDB
• How To Add Data to Amazon DynamoDB
• How To Query Amazon DynamoDB

(This tutorial is part of our DynamoDB Guide. Use the right-hand menu to navigate.)

Bulk inserts and deletes

DynamoDB can handle bulk inserts and bulk deletes. We use the CLI since it’s language agnostic. The file can be up to 16 MB but cannot have more than 25 request operations in one file.

Request operations can be:

• PutRequest
• DeleteRequest

The bulk request does not handle updates.

Data from IMDB

To illustrate, we have pulled 24 items from the IMDB (Internet Movie Database) and put them into JSON format. You can download that data from here.

The format for the bulk operation is:

{ "table name: [
"request operation": {
"item: {
(put your item here in Attribute value format)
}
}
}]
}

Here is an example:

{
"title": [{
"PutRequest": {
"Item": {
"tconst": {
"S": "tt0276132"
},
"titleType": {
"S": "movie"
},
"primaryTitle": {
"S": "The Fetishist"
},
"originalTitle": {
"S": "The Fetishist"
},
"isAdult": {
"S": "0"
},
"startYear": {
"S": "2019"
},
"endYear": {
"S": "\\N"
},
"runtimeMinutes": {
"S": "\\N"
},
"genres": {
"S": "Animation"
}
}
}
}]
}

If you are running DynamoDB locally then start it like this:

java -Djava.library.path=./DynamoDBLoc_lib -jar DynamoDBLocal.jar -sharedDb

Create a table like this:

aws dynamodb create-table \
--table-name title \
--attribute-definitions AttributeName=tconst,AttributeType=S \
--key-schema AttributeName=tconst,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \
--endpoint-url 

Then load the data like this, having saved the IMDB data in the file 100.basics.json.

aws dynamodb batch-write-item \
--endpoint-url  \
--request-items file:////Users/walkerrowe/Documents/imdb/100.basics.json \
--return-consumed-capacity  TOTAL \
--return-item-collection-metrics  SIZE          

It responds:

{
"UnprocessedItems": {}, 
"ConsumedCapacity": [
{
"CapacityUnits": 23.0, 
"TableName": "title"
}
]
}

It told you how many records it wrote. You can query that it worked like this:

aws dynamodb query  \
--endpoint-url  \
--table-name title    \
--key-condition-expression "tconst = :tconst"     \
--expression-attribute-values '{ ":tconst":{"S":"tt0276132"}}'