etcd: A Distributed Key-Value Store for Reliable Coordination

Image Credit: etcd.io

History of etcd:

The name “etcd” originated from two ideas, the unix “/etc” folder and “d"istributed systems. The “/etc” folder is a place to store configuration data for a single system whereas etcd stores configuration information for large-scale distributed systems. Hence, a “d"istributed “/etc” is “etcd”.

Definition:

etcd is a distributed key-value store widely used in cloud-native and distributed systems for managing and storing configuration data and coordinating distributed applications. It is part of the CoreOS project, which is now maintained by the Cloud Native Computing Foundation (CNCF).

etcd provides a simple and reliable way to store and retrieve data in a distributed environment. It uses the Raft consensus algorithm to ensure strong consistency and fault tolerance, making it suitable for use cases that require high availability and reliability.

etcd stores metadata in a consistent and fault-tolerant way.

Features of etcd:

etcd is written in Go(lang) and uses the Raft consensus algorithm to manage a highly-available replicated log.

Image Credit: etcd.io

Distributed and highly available: etcd is designed to run on multiple nodes, ensuring data availability despite failures or network partitions.

Key-value data model: It stores data as key-value pairs, where both the keys and values are strings. It allows you to create, read, update, and delete key-value pairs.
Watch mechanism: etcd provides a watch mechanism that allows applications to monitor changes to specific keys. This enables real-time notifications and triggers actions based on the changes.
Simple API: etcd offers a straightforward HTTP/JSON-based API, making it easy to interact with and integrate into various programming languages and frameworks.
Security and authentication: Etcd supports TLS encryption for secure communication and provides authentication mechanisms for controlling access to the data.

Etcd is commonly used as a distributed configuration store for systems like Kubernetes, where it stores cluster state information, configuration parameters, and other metadata. It also finds applications in service discovery, coordination, and other distributed systems scenarios.

Use Cases:

Kubernetes: It stores configuration data into etcd for service discovery and cluster management; etcd’s consistency is crucial for correctly scheduling and operating services. The Kubernetes API server persists cluster state into etcd. It uses etcd’s watch API to monitor the cluster and roll out critical configuration changes.

Container Linux by CoreOS: Applications running on Container Linux get automatic, zero-downtime Linux kernel updates. Container Linux uses locksmith to coordinate updates. Locksmith implements a distributed semaphore over etcd to ensure only a subset of a cluster is rebooting at any given time.

Adopters:

All Kubernetes users, this means etcd's users include companies such as Niantic, Inc Pokemon Go, Box, CoreOS, Ticketmaster, Salesforce, and many many more.

Huawei: Application: System configuration for overlay network (Canal).

Yandex: System configuration for services, service discovery.

Tencent Games: Metadata and configuration data for service discovery, Kubernetes, etc.

Baidu Waimai: SkyDNS, Kubernetes, UDC, CMDB, and other distributed systems.

https://github.com/etcd-io/etcd/blob/main/ADOPTERS.md

Comparison with other tools

Consul:

Consul is an end-to-end service discovery framework. It provides built-in health checking, failure detection, and DNS services. In addition, Consul exposes a key-value store with RESTful HTTP APIs. The storage system does not scale as well as other systems like etcd or Zookeeper in key-value operations; systems requiring millions of keys will suffer from high latencies and memory pressure. The key value API is missing, most notably, multi-version keys, conditional transactions, and reliable streaming watches.

NewSQL (Cloud Spanner, CockroachDB, TiDB):

Both etcd and NewSQL databases (Cockroach, TiDB, Google Spanner) provide strong data consistency with high availability.

NewSQL databases are meant to horizontally scale across data centers. These systems typically partition data across multiple consistent replication groups (shards), potentially distant, storing data sets on the order of terabytes and above. This sort of scaling makes them poor candidates for distributed coordination as they have long latencies from waiting on clocks and expect updates with mostly localized dependency graphs. The data is organized into tables, including SQL-style query facilities with richer semantics than etcd, but at the cost of additional complexity for processing, planning, and optimizing queries.

In short, choose etcd for storing metadata or coordinating distributed applications. If storing more than a few GB of data or if full SQL queries are needed, choose a NewSQL database.

ZooKeeper:

ZooKeeper solves the same problem as etcd: distributed system coordination and metadata storage. However, etcd has the luxury of hindsight taken from engineering and operational experience with ZooKeeper’s design and implementation. The lessons learned from Zookeeper certainly informed etcd’s design, helping it support large-scale systems like Kubernetes. The improvements etcd made over Zookeeper include: