I’m writing DynamoDB. Well, re-writing it. Not that I wrote it the first time. But I’m creating a ‘weak reference’ clone based on the original. The idea is you could have the benefits of a multi-tenant key/value datastore with the features of DynamoDB including scalability, security and reliability.
Really, this started because I wanted to write a control plane that manages hash ranges for independent nodes. This provides the building blocks of a highly scalable system. Think what Cassandra does internally. But instead of a node providing all the things, I wanted the control plane external to the data plane. Enterprise like. I figure once I had this control plane, I could use it for a DDB clone, or SQS or any other highly scalable system. I figure it would be easier to start with DynamoDB. Technically, this is true.
I’ve got a proxy layer infront of the data nodes that make up the distributed data store I’m making. They authenticate/authorize the clients, find the correct data nodes from the control plane, and make the requests. I’m ‘trying’ to keep it simple but we’ll see. I’ve got the core database on each node locked down tight. Each node has their own AES256 bit key which is XOR’d with the control plane one. They have their own tenant’s key as does the control plane. So if one tier gets compromised, you cannot decrypt the data. That took a bit of time to use.