WTF is DynamoDB

It's originally built to handle the shopping cart on Amazon. It is a hybrid between a document database and a key-value store.

WTF is DynamoDB

So i have been working extensively with DynamoDB, and what is learned so far is that it didn't really match my use case. So i wanted to highlight some of the advantages, "nice to knows" and which use cases it would fit well.

First of it's kind of a hybrid

To give a bit of history DynamoDB, it's actually written to be a key value store back in the days (link to white paper). It's originally built to handle the shopping cart on Amazon, this is why it started out as a key-value store, later on, they figured out they needed some more features which makes it a hybrid between a document database and a key-value store. Out of the box you get almost unlimited autoscaling.

Note it only autoscale up, not down again which can cost you a lot in the long run.

What is it good for?

I seen it first hand it's insanely fast with simple CRUD operations, it has autoscaling which is nice for small startups who dont have a dedicated devOps team member.

Then it also have DAX which is a on site cache in front of the dynamo tables, or as they say "DAX is a fully managed, highly available, in-memory cache for DynamoDB" and should be able to 10x performance of the database which is out the box fast.

Then it has automated backups, also lovely when you're a small startup without a devOps team member. (You have to activate it manually on each table)

Worth mentioning is the feature of streaming data to almost any other service on the AWS cloud, which is neat, that way you can channel your data into a elastic search and make heavy text search on your data, but this is not instant it would take 1-10 sec to complete the stream, that way you could miss data if you do a query to elastic right after the create or update operation.

What is it not good for

So DynamoDB excel in CRUD's and very light query operations, but everything beyond that is "No man's land", doing stuff like aggregations, heavy query's like sorting more than one field, or anything remotely complex. So if you at any point are thinking of doing something like that then DynamoDB isn't for you. But let's say you're building a e-commerce website where you need something to store your shopping cart stuff in then DynamoDB would be a great choice.

DynamoDB has narrow use case

So as you might have found out it has a very narrow use case, i would definitely go with dynamo if i had a kind of "In memory but for eternal time data to store".

So you found out that it fits your use case

You figured out that you need to use DynamoDB here is some of the "Nice to knows" and what to look out for.

First of i would use a ORM there are multiple options out there, i'm a NodeJS user so if you are too i could recommend Dynogels so most libs for DynamoDB has low amount of activity and small communities, mostly i think it's because of the slim use case it has.

If you decide to go with DynamoDB, note that DynamoDB has one of the most cryptic documentations possible.

If you want to stream data to elastic search from your dynamo table i would suggest looking into this repo, they are pretty tight with the code and ensures that its written tidy.

Be aware of the limits in dynamo, so here is a whole post from AWS on just limitations.

Now you used DynamoDB and found out you need something in the limitations

Well this happened to me, i needed querying with sorting on multiple fields i needed to aggregate data i need to do text search, more or less everything Dynamo can't do i need to do. So first of i channeled the data in to elastic but quickly found it was not very useful because if the delay of first writing to Dynamo and then to Elastic which also feels wrong to store the same data two places.

So as many before me i found out MongoDB could handle almost all of these things in a good way.

So i decided to slowly move all my data into MongoDB, but this wasn't really possible from the initial point.

What i did was rewrite each table schema into a mongoose schema and rewriting all of the database interference with backend code, well i more or less rewrote our entire backend.

And to move the data i first tried to export that data in a neat manor but actually came up with a neat little script to move all my data to the new table, the script is based on you using Dynogels and moving to Mongodb.

Script to freedom