Dynamo: amazon's highly available key-value store

Dynamo

Dynamo is distributed storage systems developed by Amazon to support its online retail operation. Its main goals are reliability at massive scale, and scalability. It should be able to treat failure handling as a norm. Dynamo satisfies eventual consistency and target design space of an “always writable” data storage (highly available for write). It allows the flexibility of deciding who performs the conflict resolution, whether it is application layer or the data store layer. It is able to perform incremental scalability (scale out one storage host at a time). It maintains symmetryness in the network, which means there is no central server that manages everything. It should be decentralized and able to handle heterogeneity of the nodes. Dynamo uses consistent hashing to handle partitioning problem, it uses vector clocks for data reconciliation during reads. Sloppy quorum and hinted handoff are used to handle temporary failures. Merkle trees are used to provide data integrity during failure recovery. And it uses gossip-based membership protocol and failure detector.

Advantages

Optimized for storage system usage whereby we need highly availability for write (with consistency trade off of course). Flexibility on local persistence component which implies we can easily changes the storage engines implementation (Berkeley DB, MySQL, Berkeley DB Java edition and in-memory buffer with persistent backing store). Dynamo is highly configurable based on R, W, and N values (number of replicas needed). It has been used, deployed, tested and fine-tuned in Amazon’s production environment.

Disadvantages

It has no security feature implemented because it is run in Amazon’s internal non-hostile environment (no malicious node). Gossip protocol to update group membership status in Dynamo may not scalable when used in thousands of nodes due to overhead in maintaining routing table.

Volunteer Computing Suitability

Dynamo works well if the Volunteer Computing system needs high write availability. However, it has no security feature implemented since it is assumed to be used in non-hostile environment. Therefore, current Dynamo implementation is not suitable for Volunteer Computing system unless the security feature is implemented and tested well.

decentralized_storage_systems/dynamo.txt · Last modified: 2012/04/23 00:56 by julia
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki