To ensure continuous service and minimize downtime, distributed systems rely on high availability (HA) mechanisms. These mechanisms are built upon replication and failover strategies, enabling systems to recover from failures without interrupting user operations.

In this lesson, we focus on how synchronous and asynchronous replication that support high availability and explore failover strategies for achieving resilience in distributed systems.

Replication for High Availability

Replication ensures that data is copied across multiple nodes to prevent single points of failure. While we’ve already discussed replication, here’s a quick comparison of how synchronous and asynchronous replication differ in supporting high availability:

We have covered the replication concept in the Distributed Database and Scalability chapter in-depth. However, we have given a quick overview below.

Synchronous Replication

1. How It Works:

   • Every write operation is simultaneously applied to all replicas before the transaction is considered complete.
   • Guarantees strong consistency across replicas.

2. Example Use Case:

   • A financial system where ensuring consistency in account balances is non-negotiable.

Asynchronous Replication

1. How It Works:

   • Write operations are first applied to the primary node and then propagated to replicas later.
   • There may be a lag between the primary node and replicas, leading to eventual consistency.

2. Example Use Case:

   • Content delivery networks (CDNs), where low latency is more critical than real-time consistency.

Failover Strategies

Failover is the process of automatically switching to a standby node or system when the primary one fails. It ensures high availability by minimizing service interruptions.

Types of Failover

1. Cold Failover

   • The standby node remains inactive until the primary node fails.
   • Requires manual or automated intervention to start the standby system.
   • Pros: Resource-efficient; no active usage of the standby system.
   • Cons: Slower recovery time due to the initialization process.

2. Hot Failover

   • The standby node is actively synchronized with the primary node and takes over immediately upon failure.
   • Pros: Minimal downtime; faster recovery.
   • Cons: Higher resource consumption as the standby is always running.

3. Warm Failover

   • The standby node is partially active, maintaining some synchronization with the primary.
   • Pros: Balances cost and recovery time.
   • Cons: May still have some downtime during failover.

Key Components of Failover

1. Heartbeat Mechanism:

   • Nodes send regular "heartbeat" messages to signal their health.
   • If the primary node’s heartbeat is not received, the system initiates failover.

2. Failover Decision Making:

   • Quorum Systems: Ensures a majority of nodes agree before a failover is initiated to avoid split-brain scenarios.
   • Priority Rules: Determines which standby node takes over if multiple are available.

3. Automated Failover:

   • Tools like ZooKeeper, Consul, or cloud-based services automate failover by monitoring nodes and switching traffic to healthy ones.

Designing High Availability (HA) Systems

Designing HA systems requires a combination of replication, failover, and architectural decisions tailored to your application’s requirements.

Best Practices for High Availability

1. Redundancy: Use multiple replicas across different geographical locations to ensure resilience against regional failures.
2. Load Balancing: Distribute traffic across multiple nodes to avoid overloading a single replica.
3. Monitor and Alert: Implement monitoring tools to detect failures proactively and initiate failover quickly.
4. Data Backup: Regularly back up data to protect against catastrophic failures.
5. Testing Failover Mechanisms: Regularly simulate failures to ensure your failover strategies work as intended.

Example HA System

Consider a distributed e-commerce application with:

1. Primary Node in New York:
   • Handles all transactions.
2. Replica Nodes in London and Tokyo:
   • Synchronized using asynchronous replication to reduce latency for users in Europe and Asia.
3. Failover Mechanism:
   • If the New York node fails, the London node (hot standby) immediately takes over.
   • The Tokyo node (cold standby) remains inactive unless both New York and London fail.