Managing the Message Format Versions

Imagine you and your friend have a secret code to pass notes. One day, you decide to add a new symbol to the code to mean something new. If you start using it without telling your friend, your new notes will confuse them. This is exactly what happens in event-driven systems.

For example, an OrderPlaced event might look like this:

Now imagine another service reads this event to send a confirmation email. Then, six months later, you add a new field: shippingAddress. You update the producer. The event becomes:

The problem is that other services, like the OrderConfirmationEmailService, might still be expecting the old version 1 format. When they receive this new message, they won't know what to do with the shippingAddress field. Worse, if a field they relied on was removed, they would simply crash.

This forces teams to carefully manage how schemas evolve. Common strategies include:

• Backward Compatibility: New schemas can be read by services expecting the old schema. This usually means you can only add new, optional fields. You can't rename or remove existing ones.

• Forward Compatibility: Services expecting a new schema can still read messages written in an old one. This is harder to achieve and often requires setting default values for missing fields.

• Schema Registry: This is like a central dictionary for all your event "secret codes." Before a service sends a message, it checks with the registry to make sure the format is valid and compatible. It prevents services from sending out "confusing notes."

Without strict rules for changing message formats, a simple update can cause a cascade of failures throughout a large system.

Observability and Debugging

In a traditional, non-event-driven system, when a user clicks a button, one piece of code calls another, which calls another, in a straight line. If something breaks, you can look at the error log and see the entire sequence of calls, like following a single piece of string from start to finish.

In an event-driven system, that single string is cut into dozens of tiny pieces. The OrderService publishes an OrderPlaced event. The PaymentService, ShippingService, and NotificationService all pick it up and do their own work independently. They might, in turn, publish their own events.

Now, imagine a customer calls saying they placed an order but never got a confirmation email. Where did it go wrong?

• Did the OrderService fail to publish the event?

• Did the NotificationService not receive it?

• Did it receive the event but fail to connect to the email server?

Debugging this can be difficult as you can't see the whole picture at once.

To solve this, we use distributed tracing. When the very first event is created, we attach a unique ID to it, called a Correlation ID. Every service that processes this event or creates a new event as a result must copy that same ID onto its own work.

When you need to investigate a problem, you can search for this one correlation ID across all the logs of all your services. This allows you to stitch the story back together and see the journey of that single request across the entire distributed system.

Handling Failures and Message Loss

Events can disappear. Not because of bugs — because of infrastructure issues like network failure, a service crashing, or the message broker itself having a problem.

The core promise of many event systems is at-least-once delivery. This means the system will do everything it can to make sure your event gets delivered. If a service that is supposed to receive an event is temporarily down, the message broker will hold onto the message and try again later.

But what if a service has a persistent bug and crashes every time it tries to process a specific message? The broker will keep trying to redeliver it, and the service will keep crashing, until the broker retry limit is reached. To handle this, we use a Dead-Letter Queue (DLQ). After a few failed delivery attempts, the message broker moves the crash causing message to the DLQ. This stops the cycle of crashing and allows the service to continue processing other, valid messages. Engineers can then inspect the DLQ later to debug the problematic message.

Idempotency

The guarantee of "at-least-once delivery" creates a new, tricky problem: what if a message is delivered more than once? This can happen if a service processes an event but crashes before it can tell the message broker, "I'm done!" The broker, thinking the message was never handled, will deliver it again when the service restarts.

If the event was IncreaseItemCountInCart, receiving it twice is a big problem. The customer who wanted one item now has two in their cart. If it was ChargeCreditCard, they get charged twice.

To prevent this, services must be idempotent. We can achieve idempotency by having the service keep a record of the event IDs it has already processed. When a new event comes in, the service first checks its records.

1. Has it seen this event ID before?

2. If yes, it simply ignores the duplicate and tells the broker, "Yep, I'm done."

3. If no, it processes the event and then saves the event ID to its records before telling the broker it's done.

This ensures that even if a message is delivered 100 times, the action is only performed once.

Eventual Consistency

In a simple application with one database, when you write data, it's there instantly. If you change your shipping address, the very next screen you load will show the new address. This is called strong consistency.

Event-driven systems give up this guarantee for the sake of scalability and resilience. They operate on a model of eventual consistency. For example, when a user updates their address, the CustomerService updates its own database and publishes an AddressUpdated event. The ShippingService and BillingService subscribe to this event. But it might take a few hundred milliseconds for them to receive the event and update their own data (This example is to provide some context, but ideally, the address should be stored at one place and the id of that record should be passed around in the events).

Designing for eventual consistency means the system must be built to handle this temporary state of disagreement. This might involve:

• Designing user interfaces that account for the delay.

• Adding logic to services to double-check critical data if needed.

• Accepting that for some non-critical data, a small delay is acceptable.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

Imagine you and your friend are in different cities. You both look at your watches and agree to clap at exactly 1:00 PM. You clap, but did you both clap at the exact same instant? Probably not. Your watch might be a few seconds faster than your friend's, or maybe it runs a tiny bit slower.

Computers in a large system face the same problem, but on a massive scale. A system like FAANG’s runs on thousands of computers spread across the globe. Each computer has its own internal clock, just like you have your own watch. These clocks are never perfectly in sync.

This creates a big problem: if two events happen on two different computers, it's hard to say with 100% certainty which one happened first just by looking at their timestamps.

Why Clock Synchronization is Hard

The obvious solution seems to be to just synchronize all the computer clocks. We have protocols like the Network Time Protocol (NTP) that try to do this by connecting to ultra-precise atomic clocks. However, even with NTP, achieving perfect synchronization is impossible for two main reasons:

1. Network Latency: Information doesn't travel instantly. When a computer asks a central time server, "What time is it?", the question takes time to travel to the server, and the answer takes time to travel back. This delay, called latency, is unpredictable. It can change based on network traffic, distance, and other factors. It's like asking your friend for the time over a walkie-talkie with a random delay. You can't be sure if the time you hear is still accurate.

2. Clock Drift: No two clocks are built exactly the same. The quartz crystals that regulate time in computers vibrate at slightly different frequencies. This causes some clocks to run a little faster and others a little slower. Over time, these small differences add up, and the clocks "drift" apart. Even if you sync them perfectly, they will start to disagree again almost immediately.

Because of these issues, we have to accept that we can never know the exact physical time across a distributed system. The time on one machine might be a few milliseconds ahead of another, and in high-speed large-scale systems, a millisecond is a very long time.

A Workaround: Logical Clocks

If we can't rely on wall-clock time, how do we solve the ordering problem? Computer scientists realized that we don't always need to know the exact time an event happened. Often, it's enough to know the causal order—that is, if one event caused another.

This is called the happened-before relationship. It's simple:

• If event A happens on a computer before event B on the same computer, then A happened before B.

• If event A is the sending of a message and event D is the receiving of that same message, then A happened before D.

• If A happened before B, and B happened before C, then A happened before C.

Events that don't have a happened-before relationship are called concurrent. We can't say which one happened first, and it might not matter. For example, if you send an email to Alice and your friend sends an email to Bob, these events are concurrent.

Logical Clocks

To track this "happened-before" relationship, we can use logical clocks. Imagine each computer in the system has a counter, like a ticket dispenser at a bakery.

1. Every time an event happens on a computer, it increments its own counter and assigns that number to the event. (e.g., counter = counter + 1).

2. When a computer sends a message, it includes its current counter value in the message.

3. When a computer receives a message, it compares its own counter to the counter in the message. It sets its own counter to the maximum of the two values, and then increments it by one.

An example with two computers, P1 and P2:

• P1 has an internal event. Its counter goes from 0 to 1.

• P1 sends a message to P2 with the value 1.

• P2 receives the message. Its counter is 0. It sees the message's value is 1. It updates its counter to max(0, 1) + 1 = 2.

• P2 has an internal event. Its counter goes from 2 to 3.

By looking at these logical timestamps, we know that the event at P1 (timestamp 1) happened before the event at P2 (timestamp 3). This method guarantees that if event A caused event B, the logical timestamp of A will be less than the timestamp of B.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

Graceful degradation is a design principle where a system maintains essential functionality even when parts of it fail. Instead of crashing completely, the system operates at a reduced capacity. For example, on a video streaming platform, if the recommendation service fails, the platform shouldn't crash. Instead, it might gracefully degrade by showing a generic list of popular videos rather than personalized suggestions. The core function—video playback—remains available.

Strategies for Graceful Degradation

Implementing graceful degradation requires a combination of strategies that control traffic, manage failures, and provide visibility into the system's health.

Rate Limiting or Request Throttling

Think of rate limiting or request throttling as a bouncer controlling the traffic to your servers. During a huge flash sale or a denial-of-service attack, your system can be flooded with requests. A rate limiter lets a certain number of requests in and tells the rest to come back later. This simple move stops a server from overloading or failing with many incoming requests.

Request Coalescing

If thousands of people hit submit on an application to get the same data, instead of sending thousands of identical queries to your database or datastore, you send just one. You get the answer and then distribute it to everyone who was waiting. It's efficient, saves resources, and keeps your system from getting bogged down with repetitive tasks.

Request Drops or Load Shedding

Sometimes, you just can't serve everyone. In those moments, load shedding becomes a necessity. This is the act of dropping non-critical requests to save the most important ones. For an e-commerce site, this means prioritizing a user's final purchase over an analytical request to log their click.

Jitter and Retry

When a service fails and then recovers, a huge list of waiting clients will try to connect all at once. This is a classic thundering herd problem, and it can immediately crash the service all over again. The solution is retry with jitter. Instead of all clients retrying at the same time, you add a small, random delay. This distributes the requests over a period of time instead of sending all requests at the same time, giving the service time to recover completely.

Circuit Breakers

This pattern is just like the electric circuit breaker in your home. If a service keeps failing, you open the circuit. All subsequent requests to that service are immediately blocked for a set period. This saves resources, provides an instant failure response to the user, and gives the broken service time to heal. After a timeout, it slowly lets a few requests through to see if the service is healthy again. If they succeed, the circuit closes, and normal operation resumes. If they fail, it opens again.

Consider an architecture where an Order Service needs to call a Payment Service. If the Payment Service becomes unresponsive, the circuit breaker in the Order Service will open. For the next 60 seconds, any attempt to process a new order will instantly fail without trying to contact the Payment Service, saving resources and providing a fast failure response to the user. After 60 seconds, it will let one request through to test the connection.

Request Timeouts

A client application should not wait indefinitely for a response from a service. If a response isn't received within a configured time frame, the request should be abandoned, and resources should be freed up. This prevents a slow or unresponsive downstream service from causing resource exhaustion (like holding onto threads or connections) in the upstream service.

Monitoring and Alerts

Monitoring and alerting are important for detecting failures before they escalate. This involves collecting metrics (e.g., error rates, latency, CPU utilization), logs, and traces from all system components. Alerting systems are configured on top of this data to notify engineers when key metrics cross dangerous thresholds. This allows for proactive intervention and quick diagnosis of problems.

A team running a large data processing pipeline would monitor the queue length of their message broker. Suppose the number of messages in the queue suddenly starts growing rapidly. In that case, an alert is triggered and sent to the on-call engineer. This indicates that a consumer service is failing or is too slow, allowing the engineer to investigate and resolve the issue before the system runs out of memory or data processing falls too far behind.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

When a request arrives for data that isn't in the cache (cache miss or cold cache), the system must retrieve it from the primary database or service. This process introduces latency, as fetching data from disk-based databases is significantly slower than from an in-memory cache.

Cache warming avoids this performance penalty by ensuring that high-demand data is already in the cache when the first client request arrives. It preloads frequently accessed data into a cache. This is especially important during system restarts, deployments, or traffic spikes.

Cache Warming Strategies

Several strategies can be employed to warm a cache.

Pre-warming on Deployment

Immediately after a new service version is deployed, its in-memory cache is typically empty. A script or process is initiated to populate the cache. The script identifies the most critical or frequently accessed data and loads it. Preloading the cache with frequently accessed data prevents an initial flood of cache misses and a subsequent thundering herd problem.

For example, in a global e-commerce platform, product catalog data is accessed millions of times per hour. Suppose a new application instance starts without a warm cache. In that case, the first thousand requests might hit the database directly, increasing response time and database load. The system avoids this bottleneck by warming the cache with top-selling product IDs before the instance accepts traffic.

Scheduled Warming

Scheduled warming involves running a batch job regularly (e.g., hourly or daily) to refresh and populate the cache. This is useful for data that doesn't need to be real-time but is computationally expensive.

A data analytics platform might run a nightly job to pre-compute and cache complex analytical queries for its customers. When users request their daily report, the pre-computed result is served directly from the cache, avoiding a resource-intensive database query.

Event-Driven Warming

This dynamic strategy triggers the cache warming process in response to specific events within the system.

For instance, an e-commerce site running a flash sale, warming the cache with relevant content or product data just before the event can prevent the system from crashing under the initial load or if an item in an e-commerce inventory drops below a certain threshold, it might trigger a process to cache related or alternative products, anticipating that users will be looking for substitutes.

Just-in-Time (JIT) Warming

When a cache miss occurs for an item, the system fetches it from the database. However, instead of just caching that single item, it also fetches and caches related items that will likely be requested next.

For example, if a user requests page 1 of a product listing, the system could proactively fetch and cache pages 2 and 3, assuming the user will likely navigate to them.

Cons of Cache Warming

Despite its benefits, cache warming has drawbacks and requires careful implementation.

• Increased System Complexity: Implementing cache warming adds another moving part to the system architecture. It requires writing, managing, and monitoring warming scripts or services. This introduces new potential points of failure that must be handled.

• Resource Consumption Spikes: Running a cache warming process can be resource-intensive. It can cause a sudden spike in CPU, memory, and network I/O on both the application servers and the backend databases. If not managed carefully, a warming process could negatively impact the performance of the live system. This is often mitigated by running warmers on dedicated instances or throttling the warming process.

• Stale Data: A challenge with any caching strategy is data staleness. A cache warmed with a data snapshot can quickly become outdated if the source data changes.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

What are the Tools in LLMs?

Tools are external functions or APIs a model can invoke to perform tasks beyond its inherent textual knowledge. These tasks include searching the web, performing calculations, accessing databases, sending emails, or interacting with other software services.

Example

Consider a large e-commerce platform's customer service chatbot powered by an LLM. When a user asks, "What is the shipping status of my order number 12345?" the LLM does not have direct access to the shipping information. A tool would be an API call to the platform's order-tracking system. The LLM, recognizing the user's intent, would call this tool, pass the order number (12345) as a parameter, and then receive the shipping status, which it can then communicate back to the user.

How Tool Calling Works

The process of tool calling involves a series of steps that begin with the user's prompt and end with the LLM generating a response that incorporates the tool's output.

How LLM Knows It Can Call a Tool

The LLM's ability to call a tool results from its training and the information provided during inference.

• During Training: When an LLM is trained for tool calling, it's exposed to vast amounts of data where natural language queries are paired with corresponding tool calls and their expected outputs. The LLM develops a deep understanding of language patterns, semantic relationships, and intent through its pre-training on massive text data.
  When it encounters tool definitions, it learns to associate specific patterns in user queries with the functionalities described by the tool. For example, if the tool getCurrentWeather has the description "Fetches the current weather conditions," the LLM forms an internal association between phrases like "What's the weather," "temperature in," "Is it raining," and this tool. This isn't explicit programming but rather an association learned from the training data.

• During Inference: These tool definitions are explicitly provided to the LLM as part of its input prompt or context window. These definitions specify the tool's name, a description of what it does, and the parameters it expects. The LLM then processes these definitions alongside the user's query. This is similar to giving a human a list of available functions and their documentation before asking them to solve a problem.

  

How LLM Decides When to Call a Tool

The decision to call a tool is based on the LLM's understanding of the user's intent and the information required to fulfill that intent. The LLM first processes the user's input and attempts to answer it using its internal knowledge base. Suppose the query can be answered directly from its training data (e.g., "What is the capital of France?"). In that case, it will proceed to generate a direct textual response.

However, if the query requires external, real-time, or proprietary information not present in its pre-trained weights (e.g., "What's the current stock price of Google?"), the LLM identifies this knowledge gap. Then, LLM analyzes the user's query to understand the intent (underlying goal or action). It then attempts to map this recognized intent to the available functionalities described by the tools. For instance:

• User: "What's the weather in Tokyo?"
  Intent: Get current weather information.

• User: "Book me a flight to New York for next Tuesday."
  Intent: Initiate a flight booking process.

How LLM Decides Which Tool to Call

When multiple tools are available, the LLM selects the most appropriate one based on the user's query and the descriptions of the available tools. This decision is driven by the LLM's ability to match the semantic meaning of the user's request with the functionalities described in the tool definitions. The LLM might rank potential tools based on relevance scores or confidence levels. For example, if both a "weather forecasting tool" and a "stock price lookup tool" are available, and the user asks, "What's the current temperature in New York?" the LLM will favor the weather forecasting tool due to semantic similarity.

How Results from a Tool are Used

Once a tool call is executed, the external system returns a result. This result, often in a structured format like JSON, is fed back to the LLM. The LLM then integrates this new information into its context. It uses it to formulate an informative response to the user. In the e-commerce example, if the order tracking tool returns {"status": "Shipped," "estimated_delivery": "2025-06-25"}, the LLM would then generate a response like: "Your order number 12345 has been shipped and is estimated to be delivered by June 25, 2025."

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

A feature flag is a mechanism that allows developers to turn certain functionalities on or off during runtime without deploying new code. Imagine a large e-commerce platform processing millions of transactions. If the platform wants to introduce a new, experimental recommendation algorithm, it's risky to release it to all users simultaneously. A feature flag acts like a light switch for this new algorithm. The code for the new algorithm is deployed as part of the application, but it's wrapped in a conditional block controlled by the feature flag. If the flag is "on," the new algorithm runs; if it's "off," the old, stable algorithm continues to operate.

How Feature Flags Help with Rollouts

Feature flags provide several advantages during the rollout of new features or changes in complex systems.

Partial and Controlled Rollouts

Instead of a "big bang" release where a new feature is pushed to 100% of users simultaneously, feature flags allow for a more controlled exposure. For instance, a video-sharing platform introducing a new video processing pipeline can initially enable it for just 1% of its user base or users in a specific geographic region. This limited exposure allows the team to monitor the new pipeline's performance, resource consumption (e.g., CPU, memory on data processing nodes), and error rates in a real-world environment but with a contained blast radius. If issues arise, only a small subset of users is affected. As confidence in the new feature grows, the percentage of users or servers exposed can gradually increase to 5%, 20%, 50%, and finally 100%.

Faster Development Iteration & Testing/Experimentation

Feature flags allow faster development iterations because developers can merge incomplete or experimental features into the main codebase, hiding them behind a flag. This reduces merge conflicts and allows for testing in the production environment. Teams can run A/B tests by enabling a feature for one group of users (Group A) and keeping it disabled or providing an alternative for another group (Group B).

Dynamic Configuration without Redeployment

Feature flags allow functionality to be turned on or off without redeploying the service. In a distributed system with hundreds or thousands of service instances, redeploying all of them to turn a feature on or off is a significant operational overhead. The running application instances pick up feature flag changes within seconds or minutes, allowing for near real-time control over system behavior.

Rapid Rollbacks

When a new feature starts causing unexpected problems, feature flags allow instant rollbacks. Instead of redeployment of the previous application version, which can be time-consuming and stressful, the feature flag can be turned"off." The problematic feature is immediately disabled, and the system reverts to its prior stable state. This is much faster and safer than a traditional rollback process involving code changes and deployments.

How Feature Flags Work

A feature flag system involves two main components:

1. Conditional Logic in Code: The application code contains if-else statements (or similar conditional logic) that check the state of a feature flag.

   

2. Flag Configuration Management: This is a system (often a dedicated service or a distributed configuration store like etcd or Consul) that stores the current state (on/off) and targeting rules (e.g., enabled for 10% of users, or for users in Canada) for all feature flags. Applications typically fetch the flag configurations from this system at startup and then periodically refresh them or subscribe to updates.

Downsides of Feature Flags

1. Technical Debt: Over time, the number of feature flags in a codebase can grow substantially. Each flag adds complexity to the code (e.g., more conditional branches). Old flags, especially those for features that are now fully rolled out or abandoned, become technical debt.

2. Testing Complexity: With each flag, the number of possible execution paths in an application increases, potentially making testing more complex.

3. Management Overhead: A feature flagging system requires setup and ongoing management. Rules need to be defined, statuses tracked, and the system must be reliable.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

Managing concurrent access to shared resources is a critical challenge in distributed systems. Conflicting updates can lead to data corruption and inconsistency. Optimistic locking operates on the principle of "assume no conflict, but verify." Instead of preventing conflicts by acquiring locks upfront, transactions can proceed. It only checks for conflicts at the point of committing changes. This process typically involves three distinct phases:

Read Phase

A transaction reads data from the data store without acquiring any locks on the resources. The system notes the data's version at the time of the read, which can be a timestamp, a version number, or a checksum. The transaction then computes and prepares any intended modifications based on this read data.

For example, consider a distributed product inventory system. When a user attempts to update a product's stock level, the system first reads the current stock count and its associated version number. Let's say Product A has a stock of 100 and a version number of 5.

Validation Phase

Before committing any changes, the transaction enters the validation phase. Here, the system checks if any other concurrent transaction has modified the data read in the first phase. This is done by comparing the version of the data noted during the read phase with the current version of the data in the data store. If the versions match, it implies that no other transaction has altered the data, and the current transaction can proceed. If the versions differ, another transaction has modified the data in the interim, leading to a conflict.

For example, before updating the stock of Product A to 90 (due to a sale), the system checks if the version number for Product A in the database is still 5.

Write Phase (or Rollback)

If the validation phase is successful (no conflict detected), the transaction proceeds to the write phase. The prepared changes are applied to the data store, and the version of the modified data is updated (e.g., the version number is incremented). This makes the changes visible to other transactions. If the validation phase detects a conflict, the transaction is aborted or rolled back. The application then needs to decide how to handle this conflict – it might retry the entire transaction, notify the user, or apply a specific conflict resolution strategy.

When to Use Optimistic Locking

Low Contention Environments

The core assumption of optimistic locking is that conflicts are rare. Therefore, it shines in systems where the probability of multiple transactions attempting to modify the same piece of data simultaneously is low.

Read-Heavy Workloads

Systems where data is read frequently but modified infrequently benefit greatly from optimistic locking. Read operations do not require locks, allowing for high concurrency and responsiveness.

High Concurrency Requirements

By not holding locks for extended periods (or at all during the read and computation phases), optimistic locking allows a greater number of transactions to proceed concurrently. This can significantly improve system throughput.

Systems Where Retries are Acceptable

Since conflicts lead to transaction rollbacks, the application must be designed to gracefully handle these retries. Suppose the cost of retrying a transaction is low and the conflict rate is minimal. In that case, optimistic locking is an efficient choice.

Problems with Optimistic Locking

Potential for Starvation

Suppose a particular piece of data is highly contended. In that case, some transactions attempting to update it might repeatedly fail validation and get rolled back, leading to starvation for those transactions. Frequent conflicts can also lead to significant performance degradation due to repeated rollbacks and retries.

Complexity in Handling Retries

The application logic must be robust enough to handle transaction failures and implement retry mechanisms. This can add complexity to the application development. Deciding on the appropriate retry strategy (e.g., number of retries, backoff period) is crucial.

Data Validation Complexity

Ensuring that the validation check is comprehensive and correct is critical. The validation logic can become intricate for complex transactions involving multiple data items.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, please 🔁 share this post.

Strategies

Several techniques can be employed, often in combination, to scale a database.

Indexing

Indexes speed up data retrieval by creating pointers to rows, similar to a book's index. Instead of scanning an entire table (full table scan) to find specific rows, the database can use an index to locate the data quickly. For example, consider a orders table with millions of rows. Searching for an order by the order_id without an index would require scanning every row. Creating an index on the order_id column allows the database to find the user's record almost instantly.

Trade-off

Indexes speed up read queries (SELECT) but slow down write operations (INSERT, UPDATE, DELETE) because the index must also be updated during writes. They also consume disk space. Choose indexes wisely based on common query patterns.

Materialized Views

Materialized views store the pre-computed result of a query. For complex queries involving joins or aggregations that run frequently, computing the result once and storing it can reduce query time. For example, an analytics dashboard displays daily total sales aggregated by product category. This query involves joining orders, order_items, and products tables and performing aggregations. Creating a materialized view, daily_sales_by_category, that stores these daily totals allows the dashboard to load much faster by querying this simple view.

Trade-off

The data in a materialized view can become stale. It needs a refresh strategy that consumes resources (e.g., nightly or triggered by data changes). Materialized views also consume storage space.

Denormalization

Normalization is organizing database tables to reduce data redundancy and improve data integrity, often involving splitting data into multiple related tables. Denormalization is the opposite process: intentionally adding redundant data to one or more tables to avoid costly joins when querying. For example, an e-commerce application must frequently display the product name and order details. A normalized design might require joining the orders table with the products table. To speed this up, you could denormalize by adding a product_name column directly to the orders table. This avoids the join during reads.

Trade-off

Denormalization increases storage requirements. More importantly, it complicates data updates and increases the risk of data inconsistency. If a product name changes, it must be updated in every order record referencing it, not just in the central products table.

Vertical Scaling

This involves increasing the resources of the existing database server, such as adding more CPU power, RAM, or faster disk storage (like SSDs).

Trade-off

Physical limits exist to how much you can scale up a single machine. High-end hardware is expensive. It does not significantly improve fault tolerance, as you still have a single point of failure. Upgrades often require downtime.

Database Caching

Caching involves storing frequently accessed data in a temporary, fast-access memory layer (like Redis or Memcached) between the application and the database. Subsequent requests for the same data can be served directly from the cache, bypassing the database entirely. For example, a news website caches popular articles. When a user requests an article, the application first checks the cache. If the article is cached, it's returned immediately. If not, the application queries the database, returns the article to the user, and stores a copy in the cache for future requests.

Trade-off

Cache invalidation is a complex problem. Caching adds another component to manage and introduces potential consistency issues (stale data). It primarily helps with read-heavy workloads.

Replication

Replication involves creating one or more copies (replicas) of the primary database server. Write operations (INSERT, UPDATE, DELETE) to the primary server. Read operations (SELECT) can be distributed across the replicas. This distributes the read load and improves read throughput. Replication also provides high availability; if the primary fails, a replica can be promoted to the new primary.

Trade-off

There is often a replication lag before changes on the primary appear on the replicas. This can lead to reading slightly stale data. It increases infrastructure costs and complexity. It does not solve bottlenecks related to high write volumes on the primary server.

Sharding

Sharding involves partitioning the database data horizontally across multiple independent database servers, called shards. Each shard contains a subset of the total data. The application logic or a proxy layer routes queries to the appropriate shard based on a partition key. This distributes both read and write loads across many machines.

Trade-off

Sharding increases complexity in both application development and database operations. Queries that need data from multiple shards become complex and slow. Rebalancing data is challenging. Choosing an appropriate shard key is critical to avoid uneven data distribution.

When to Consider?

Scaling introduces complexity and cost, so it should not be implemented prematurely. Start monitoring key database metrics early. Before implementing complex scaling solutions, always investigate if query optimization or schema improvements can resolve the bottlenecks. Rewriting inefficient queries or adding appropriate indexes sometimes provides relief without infrastructural changes.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

In a single-leader system, one node (the leader) handles all write operations. Followers replicate the leader’s data and serve read requests to distribute the load. The request is sent only to the leader when a client wants to write data. The leader processes the write, records it in its local log (a sequence of operations), and then sends this update to all its followers. Followers apply the updates in the same order as the leader, ensuring they eventually have the same data.

What is Failover?

Failover is the process of automatically switching to a backup system or node when the primary system or node fails or needs to be taken offline for maintenance. In single-leader systems, failover deals with two main scenarios: a follower node failing or the leader node failing. The goal is to minimize downtime and data loss.

Follower Failover

The followers are read-only replicas (or read-mostly) that do not accept direct writes from clients. However, their failure does not stop the system from accepting new data.

1. Detection: The leader or a monitoring system detects unresponsive followers (e.g., through missed heartbeats or failed connection attempts).

2. Removal: The failed follower might be temporarily or permanently removed from the list of active followers. Read requests previously directed to it need to be rerouted to other followers or the leader.

3. Replacement: A new follower node can be started. This new node needs to catch up with the leader's current state. It typically does this by copying a recent snapshot of the leader's data and then applying all the changes from the leader's log that occurred after the snapshot was taken. Once caught up, it starts receiving live updates like any other follower.

For example, if a follower crashes, a new virtual machine can be provisioned, installed with the necessary software, and configured to connect to the leader to start replication from the last known point or a full snapshot. Follower failover is relatively straightforward because the leader remains operational throughout the process. However, delays in detecting failures or slow synchronization can temporarily reduce redundancy, making the system more vulnerable.

Leader Failover

Leader failure is more critical because the leader is the only node that can accept writes. Without a leader, the system cannot process new data updates.

1. Detection: The first step is detecting that the leader has failed. This is done through health checks or heartbeats monitored by followers or an external coordination service. If the leader doesn't respond for a specific period, it's considered failed.

2. Leader Election: Once the failure is confirmed, the system must choose a new leader from the available followers. The most common requirement is to select the follower with the most up-to-date data log. This minimizes potential data loss. Various algorithms exist for leader election, often managed by consensus protocols (like Raft or Paxos) implemented directly by the nodes or external services.

3. Reconfiguration: Clients and remaining followers must be informed about the new leader. Clients need to redirect their write requests to the newly elected leader. Followers need to disconnect from the old leader and follow the new one.

There are two main types of leader failover:

Graceful Failover (Planned)

This happens during scheduled maintenance or the leader's planned shutdown. The current leader finishes processing pending writes, ensures one specific follower is fully up-to-date, and then transfers leadership to that follower before shutting down. This process typically involves no data loss because the handover is controlled.

Emergency Failover (Unplanned)

This occurs when the leader fails unexpectedly (e.g., crashes, network partition). The system must react automatically. A follower node (usually the one with the most recent data log) is promoted to be the new leader through the election process. If it recovers later, the old leader might rejoin the cluster as a follower to avoid having two leaders.

Network latency matters when choosing a new leader, especially in geographically distributed systems. Ideally, the new leader should be located "close" (in terms of network latency) to most clients or the previous leader's location to minimize the time it takes for requests to travel. However, the primary concern is data consistency (choosing the most up-to-date follower), so proximity is a secondary factor.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

Idempotency refers to the property of an operation where executing it multiple times has the same effect as executing it once. Distributed systems inherently deal with unreliability. Network partitions, timeouts, server unavailability, or client crashes can interrupt communication flows. When a client sends a request but doesn't receive a timely response, it cannot be certain whether the request was processed, lost before reaching the server, or processed successfully, but the response was lost. The natural reaction is to retry the request. Without idempotency, retrying operations can cause unintended side effects like duplicate payments or incorrect data.

For example, a customer submits an order on an e-commerce platform, but a network glitch causes the client to retry the request. Without idempotency, the system might create two identical orders, charging the customer twice. An idempotent API ensures only one order is created, regardless of retries. Similarly, in financial systems, a funds transfer API must guarantee a single transfer even if the request is sent multiple times due to timeouts.

Achieving Idempotency

Implementing idempotency requires the server to detect duplicate requests. This involves tracking the state of requests using a unique identifier provided by the client, known as an Idempotency Key. The client generates a unique key (e.g., a UUID) for each distinct operation and it must send the same idempotency key for retries of the same operation (in case of failures).

• When a request arrives with an idempotency key, the server checks to see if it has processed one with this key before.

• Suppose the key is recognized and associated with a completed operation. In that case, the server skips executing the operation again and returns the previously generated response.

• Suppose the key is recognized, but the original operation is still in progress (e.g., held by a lock). The server might wait or return a conflict/busy status in that case.

• If the key is new, the server processes the request, stores the result associated with the idempotency key, and then returns the response.

Idempotency in Stateful Services

Stateful services manage idempotency by storing information about processed requests.

Database Unique Constraints

Databases provide built-in mechanisms to enforce idempotency. For example, assigning unique primary keys to entities ensures duplicate entries cannot exist. Continuing with the e-commerce example, if each order has a globally unique identifier (GUID), attempting to insert an order with the same GUID will fail due to a primary key violation. This technique works well for write-heavy systems but requires proper error handling.

Deduplicating Duplicate Requests

In distributed systems, messages may be delivered multiple times due to network issues or retries. Before processing each request, the system checks if the request's idempotency key has been seen before. For example, a message queue processing user signups might store processed message IDs in a distributed cache. If a duplicate message arrives, the system immediately skips reprocessing and acknowledges receipt.

Idempotency in Stateless Services

Stateless services cannot easily implement idempotency, as they retain no memory of past requests. Idempotency for stateless services is often handled by stateful components upstream (like an API gateway) or downstream (like ensuring the ultimate data storage operation is idempotent).

Race Conditions in Idempotent Systems

Race conditions occur when multiple processes access shared resources concurrently, potentially causing inconsistent outcomes. In distributed systems, race conditions are common due to high concurrency and network latencies. Consider a scenario where two identical requests with the same idempotency key arrive simultaneously at different server nodes. Without synchronization, both nodes might check the idempotency store, find no key, and process the payment, resulting in duplicate transactions. Mitigation strategies include:

• Distributed Locking: Before processing, the service attempts to acquire a lock named after the idempotency key. Only the lock holder proceeds with processing.

• Request Sharding: Using consistent hashing on the idempotency key, requests concerning the same key can be routed to the same server, reducing the need for global locks.

  

• Database Transaction Isolation Levels: Transaction isolation levels (like Serializable) can lock the relevant resource (the potential row for the idempotency key) during the check-and-process phase. This forces concurrent requests for the same key to serialize.

• Atomic Operations: Leverage atomic operations/commands provided by the underlying storage. The first successful insert wins; subsequent attempts fail gracefully or update a status, allowing the application to know the request was already handled.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

Why Partition Data?

Data is partitioned primarily for scalability. A single server has limits on CPU power, memory, disk space, and network bandwidth. As a dataset or the number of requests grows, a single server eventually becomes a bottleneck. By partitioning the data across multiple servers, the storage requirement is distributed.

Scaling Reads and Writes

The read/write query load can also be spread out. If data is partitioned effectively, a query targeting a specific piece of data must only be processed by the node(s) holding that partition rather than overwhelming a single central server. Similarly, write operations can be directed to the appropriate partition, allowing multiple writes to occur concurrently across the cluster.

Fault Tolerance

Fault tolerance is another critical benefit. If data is stored on a single machine and that machine fails, the entire dataset becomes unavailable. With partitioning, if one node fails, only the partitions it hosts become unavailable (replication strategies mitigate this). The rest of the dataset remains accessible, improving the system's overall availability and resilience.

Hotspots

Achieving scalability through partitioning depends heavily on how the data is partitioned. A poorly designed partitioning strategy can lead to data skew, where some partitions store significantly more data than others. This imbalance causes hotspots.

Range-Based Partitioning

Range-based partitioning involves dividing the data based on contiguous ranges of a specific attribute, known as the partition key. The system defines key ranges, each assigned to a specific partition. The partition key should be an attribute whose values have a natural order, such as timestamps, numerical IDs, or alphabetical strings.

For example, consider a database storing user information partitioned by user_id.

• Partition 1: user_id from 1 to 1,000,000

• Partition 2: user_id from 1,000,001 to 2,000,000

• Partition 3: user_id from 2,000,001 to 3,000,000

Another example could use order dates:

• Partition A: Orders placed in January 2025

• Partition B: Orders placed in February 2025

• Partition C: Orders placed in March 2025

The primary advantage of range partitioning is its efficiency for range queries. Suppose an application needs to retrieve all users with IDs between 500,000 and 600,000. In that case, the system knows immediately that this data resides entirely within Partition 1. The query only needs to be routed to the node(s) holding Partition 1, significantly reducing the search scope compared to random assignment.

Susceptible to Hotspots

However, range partitioning is susceptible to creating hot spots. Specific partitions can become overloaded if access patterns are not uniformly distributed across the key ranges. In the user_id example, if newly registered users (with higher IDs) are accessed more frequently, the partitions holding the highest ID ranges will experience more load. In the order date example, if most queries target recent orders, the current month's data partition will become a hot spot for both reads and writes.

Key-Based Partitioning

Key-based partitioning aims to distribute data more evenly than range partitioning, mitigating the hot spot issues associated with predictable ranges. In this strategy, a hash function is applied to the partition key of each data record. The output of the hash function determines which partition the record belongs to. A common method is to use the modulo operator. If there are N partitions, the target partition for a record with key k might be calculated as partition = hash(k) % N.

For example, consider partitioning user data by user_id across three partitions using a hash function:

• User with user_id 123 —> hash(123) = 84362 —> 84362 % 3 = 2. The record goes to Partition 2.

• User with user_id 456 —> hash(456) = 19373 —> 19372 % 3 = 1. The record goes to Partition 1.

• User with user_id 789 —> hash(789) = 55219 —> 55221 % 3 = 0. The record goes to Partition 0.

A good hash function distributes keys pseudo-randomly and uniformly across its possible output values. This ensures that each partition receives a roughly equal amount of data and request load. This approach effectively spreads write load and avoids hot spots caused by access patterns concentrating on specific key ranges.

The main disadvantage of hash partitioning is that it makes range queries inefficient. Since keys that are close together in their natural order (e.g., user_id 1000 and 1001) are likely mapped to different partitions by the hash function, retrieving a range of keys (e.g., all users with IDs between 1000 and 2000) requires querying all partitions. The relationship between adjacent keys is lost in the hashing process.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

Managing and efficiently querying vast amounts of data in distributed systems is a big challenge. Partitioning (or sharding) is a common technique to handle this scale. It involves splitting a large database into smaller, more manageable pieces called partitions. Each partition holds a portion of the data and can live on a separate machine in a cluster. For example, consider a customer database with millions of records. The system can distribute the load across multiple machines by partitioning the data based on a key such as CustomerID.

Each partition acts like a smaller, independent database. A primary index is maintained within each partition, typically based on the table's primary key. If the table's primary key is also the partition key, query filtering can be done as follows:

1. The query router (or client) determines the target partition using the partition key value provided in the query (e.g., by hashing it).

2. The query is sent only to that specific partition.

3. The target partition uses its local primary index to locate the data quickly.

For example, SELECT * FROM orders WHERE customer_id = 54321 AND order_id = 9876; If partitioning is by customer_id, the system hashes 54321 to find the correct partition. That partition then uses its local index (likely on (customer_id, order_id) or order_id to find the specific order.

Secondary Indexes

Often, applications need to query data based on attributes other than the partition key, such as searching orders by product_id. Performing such queries without an appropriate index would require scanning every row in every partition (a full table scan), which is infeasible in large systems. To support such queries, secondary indexes are created on these non-partition key columns. In partitioned databases, secondary indexes can be local (tied to one partition) or global (covering all partitions).

Local Secondary Indexes

A local secondary index exists within each data partition and only references data stored on that specific partition. For example, the orders table is partitioned by customer_id. A local secondary index is created on product_id. Partition 1's product_id index only contains entries for orders belonging to customers whose customer_id hashes to Partition 1.

Despite their benefits, local secondary indexes have limitations. Since they are confined to individual partitions, querying them requires the system to perform a "scatter and gather" operation. This means the query is broadcast to all partitions. Each partition independently searches its local secondary index for the matching data, and the query coordinator aggregates the results. While effective for small datasets, this approach becomes inefficient at scale because it increases network overhead and query latency.

Global Secondary Indexes

Distributed systems use global secondary indexes to overcome the limitations of local secondary indexes. A global secondary index spans all or multiple partitions and is not tied to the table's primary/partition key. It maps the secondary key value(s) directly to the base table's primary/partition key(s). Queries targeting the secondary key first consult the global index. The index returns the primary/partition key(s) of the matching rows. The system then uses these keys to query only the necessary data partitions directly.

For example, the orders table is partitioned by customer_id. A global secondary index is created on product_id. This index stores entries like

• product_id_abc => [(order_id_1, customer_id_99), (order_id_39, customer_id_56)] and

• product_id_xyz => [(order_id_44, customer_id_20), (order_id_64, customer_id_32)]

Query SELECT * FROM orders WHERE product_id = 'product_id_xyz' returns [(order_id_44, customer_id_20), (order_id_64, customer_id_32)] from the global secondary index and then uses the retrieved customer_id values (the partition key) to determine the specific data partitions holding the relevant rows. Queries (eg. SELECT * FROM orders WHERE customer_id = 'customer_id_32' AND order_id = 'order_id_64') are then sent directly to those partitions using the full primary key.

Problems with Keeping Global Index

Global secondary indexes solve the problem of scattering and gathering by centralizing the indexed data. However, maintaining a global index on a single machine poses challenges. The node can become the single point of failure or a bottleneck during index reads and updates. As the dataset grows, the index itself can become a bottleneck, leading to slower query performance and potential downtime.

Splitting Global Indexes Across Multiple Partitions

The index itself is partitioned across multiple nodes, similar to how the base data is partitioned, to address the scalability issues of global secondary indexes. The partition key for the global index is typically the indexed column itself. Each index partition holds a slice of the global index entries based on the index's partition key.

For example, the global index on product_id (shared previously) is partitioned across 2 nodes based on hash(product_id) % 2. A query for product_id = 'product_id_xyz' calculates hash('product_id_xyz') % 10, say it results in 1. The query is sent to Index Partition 1. Index Partition 1 returns the relevant (customer_id, order_id) pairs. Then, the system fetches the actual order data from the data partitions indicated by the customer_id values.

Updating Global Secondary Indexes

Updates to a partitioned global secondary index require coordination because multiple, potentially remote partitions must be updated atomically or consistently. Whenever a record is inserted, updated, or deleted, the corresponding entry in the global index must also be modified. This process involves two steps:

1. Primary Data Update: The system first updates the primary data in its partition.

2. Index Update: The system then propagates the change to the appropriate partition of the global secondary index.

To ensure consistency, distributed databases often use techniques such as eventual consistency or distributed transactions.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

Latency refers to the time it takes for a request initiated by a client to travel across the network to one or more servers, be processed, and for the response to travel back to the client. It is a metric used for evaluating the responsiveness and efficiency of software systems. Several factors contribute to latency in distributed systems.

1) Geographic Distance

Data travels at a finite speed through network cables or wireless signals. The physical distance between components directly impacts this delay. For instance, a request from the US to a European server will inherently experience higher latency than one within the same city.

2) Network Congestion

When network links become overloaded with traffic, data packets may experience queuing delays at routers and switches. This congestion can significantly increase latency, especially during peak usage times.

3) Processing Delay

Servers require time to process requests. This includes parsing the request, accessing data, performing computations, and preparing the response. Complex operations or resource-intensive tasks will naturally lead to higher processing delays

4) I/O Delay

Many operations in distributed systems involve reading from or writing to storage systems (like databases or disk drives). Disk I/O operations are generally slower than in-memory operations, contributing to latency.

5) Data Handling

When data is transmitted across a network, it often needs to be converted into a transferable format (serialization) and then back into its original format upon arrival (deserialization). These processes add to the latency, particularly for large data objects or complex data structures.

How to Reduce Latency?

Various techniques can be used to reduce latency in large-scale software systems. The choice of strategy depends on the specific scenario and the primary sources of latency.

Network-Level Optimizations

1) Geo-Specific Routing

Directing user requests to the geographically closest data center reduces network propagation delay. For example, a user in London accessing a service hosted in the US and Europe should be routed to the European data center. DNS-based routing (like AWS Route 53 Geolocation Routing) or specialized traffic management tools can be used to achieve this.

2) Content Delivery Networks (CDNs)

CDNs cache static assets (images, videos, CSS) on servers distributed geographically closer to users. This reduces the need to fetch data from origin servers and minimizes network latency.

Data-Level Optimizations

1) Caching

Caching stores frequently accessed data in memory for faster retrieval, reducing the need for slower operations like database queries. This approach is effective for read-heavy workloads. For example, a frequently accessed user profile can be cached in memory to avoid hitting the database for every request.

2) Data Compression and Serialization

Reducing the size of data transmitted over the network directly reduces the time it takes to transfer. Besides that, efficient data serialization formats can also improve latency. Formats like Protocol Buffers or Apache Avro are often more compact and faster to serialize/deserialize compared to JSON or XML, especially for large data volumes.

System-Level Optimizations

1) Asynchronous Systems

In many scenarios, a user doesn't need an immediate response. Using asynchronous communication patterns, such as message queues (like Kafka, RabbitMQ, or Amazon SQS), allows the system to process requests in the background. The user can receive an acknowledgment quickly, and the actual processing can happen later without blocking the user's interaction. This is useful for tasks like image processing or sending emails.

2) Sharding, Replication, and Load Balancing

Horizontally partitioning the data across multiple nodes can allow queries to be executed in parallel across different shards, reducing the amount of data each server needs to process. Read-heavy workloads can be distributed across multiple replicas, reducing the load on the primary node. Moreover, incoming requests can be distributed evenly (load balancing) across multiple instances of an application server to prevent any single server from becoming overloaded.

3) Connection Pooling and Database Indexing

Establishing and tearing down network connections for every request can be expensive. Connection pooling allows applications to reuse existing connections, reducing the overhead associated with connection establishment. In addition to this, indexing frequently queried database columns and avoiding full-table scans can help with quick data lookups.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

What is Cache Invalidation?

Cache invalidation ensures that the data stored in the cache is up-to-date and consistent with the original data source. When the original data changes, the corresponding entry in the cache must be either removed or updated to reflect the change. This is crucial because if the cache contains stale data (data that is no longer accurate), users might receive incorrect information, leading to inconsistencies and potential errors in the system.

Why is Cache Invalidation Hard?

Cache Invalidation presents a significant challenge in large-scale distributed systems processing high volumes of data. The inherent complexities arise from various factors:

Distributed Systems and Network Latency

In distributed systems, data can be cached across numerous nodes. When the original data is updated, the invalidation signal must reach all the nodes holding the cached copy. This process is inherently susceptible to network latency and potential failures. Sending invalidation messages across a large network takes time, and there's no guarantee that all messages will be delivered on time. This delay can lead to temporary inconsistencies where some users see the updated data. In contrast, others still see the stale version.

Concurrency and Ordering of Operations

In high-throughput systems, multiple read and write operations can occur concurrently. Ensuring that invalidation messages are processed in the correct order relative to data updates can be challenging. Suppose an invalidation message arrives before the update is fully committed to the primary data store. In that case, the cache might prematurely remove a non-stale entry. On the other hand, the cache will serve stale data if the update is complete before the invalidation.

Balancing Freshness and Performance

There's an inherent trade-off between data freshness and performance. Aggressive invalidation strategies immediately removing cached data upon any update ensure high freshness but can lead to increased load on the primary data store as caches are frequently repopulated. On the other hand, less aggressive strategies that rely on time-based expiration might improve performance by reducing the frequency of invalidation but risk serving stale data for longer periods.

Cache Size Limitations and Eviction Policies

Caches have finite storage capacity. When a cache reaches its limit, it needs to evict some entries to make space for new ones. The eviction policy (e.g., Least Recently Used, Least Frequently Used) can inadvertently remove data that is still valid and might be requested again soon, leading to cache misses and increased load on the primary data store.

False Sharing

False sharing is a performance issue in multi-core processors where two or more data items reside within the same cache line. When one core modifies a data item, the entire cache line is invalidated for other cores, even if they accessed different data items within the same line, which degrades performance. For example, two variables stored contiguously in memory might share a cache line. Updating one variable invalidates the entire line, even if the other remains unchanged.

Resource Constraints and Cost Considerations

Implementing sophisticated cache invalidation mechanisms can be resource-intensive. Maintaining metadata about cached entries, tracking dependencies, and reliably propagating invalidation messages require computational resources, network bandwidth, and storage. The cost of implementing and maintaining these mechanisms must be carefully considered, especially in large-scale systems where even small inefficiencies can have a significant impact.

Predicting Future Access Patterns

An ideal invalidation strategy would proactively invalidate data that is likely to become stale or is unlikely to be accessed again soon. However, accurately predicting future access patterns is a complex problem. Relying on heuristics or historical data might not always be effective in dynamic environments where access patterns can change rapidly. For example, a video streaming service might prefetch content based on user behavior, but unexpected viral trends could render predictions obsolete, flooding caches with irrelevant data.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

What Are Faults and Why Do They Occur?

A fault is a defect in a system component that can lead to an error (incorrect state) and, eventually, a failure (deviation from expected behavior). Faults in distributed systems can be categorized into several types:

• Hardware Faults: Physical failures like disk crashes, RAM corruption, or power outages.

  • Example: A server in a cloud cluster loses network connectivity due to a faulty router.

• Software Faults: Bugs in code, race conditions, or memory leaks.

  • Example: A microservice crashes after encountering an unhandled null pointer exception.

• Human Errors: Misconfigurations, accidental data deletions, or flawed deployment scripts.

  • Example: A developer deploys a buggy update that corrupts a distributed cache.

• Network Faults: Packet loss, latency spikes, or partitions (split-brain scenarios).

  • Example: A database replica in Europe becomes unreachable due to an undersea cable cut.

• Environmental Faults: Natural disasters, power grid failures, or overheating data centers.

• Byzantine Faults: These are the faults where a component behaves arbitrarily, potentially sending conflicting or malicious information to different parts of the system. This can be due to hardware errors, software bugs, or even malicious attacks.

Impact of Faults

When faults occur in a distributed system, they can impact the overall system functionality and user experience.

• Data Loss: A critical impact of hardware failures, especially disk drive failures, can be the permanent loss of valuable data if proper replication mechanisms are not in place. Software bugs leading to incorrect data updates or deletions can also result in data loss.

• Service Unavailability: If a critical component of the system fails, its service might become unavailable to users.

• Performance Degradation: Faults can lead to a decrease in performance. For example, network delays can slow communication between components, and failed nodes might need to be bypassed or their workload redistributed, adding overhead to the remaining nodes.

• Inconsistent State: Faults, particularly network partitions, can lead to inconsistent views of the data or the system's state among different parts of the system.

• Security Breaches: In some cases, faults can be exploited by malicious actors to compromise the system's security.

How to Achieve Fault Tolerance?

Replication

This involves creating multiple copies of critical data or components across different nodes. If one replica fails, others can take over, ensuring continued availability and data durability.

• Active Replication: All replicas process incoming requests concurrently. This provides low latency failover but requires careful coordination to ensure consistency. For example, a load balancer distributes incoming user requests to multiple identical application servers. If one server fails, the load balancer simply directs traffic to the remaining healthy servers.
  

  

• Passive Replication (Standby): One replica is designated as the primary, handling all requests. Secondary replicas remain idle and are activated only when the primary fails. This may have a higher failover latency. For example, a primary database server handles all write operations, while a secondary server continuously receives updates. If the primary fails, the secondary is promoted to become the new primary.

  

Failure Detection and Recovery

This includes having mechanisms to detect when a component has failed.

• Heartbeats: Periodic messages sent by a component to indicate its health. The absence of a heartbeat within a certain timeout period indicates a failure.

  

• Checkpointing: Periodically saving the state of a system or component (checkpointing) allows for recovery from failures by restoring the last saved state.

  

Retries and Timeouts

• Exponential Backoff: Retry failed requests with increasing delays (jitter) to avoid overwhelming the system.

  

• Circuit Breakers: Temporarily block requests to a failing service or component until it recovers and is ready to handle traffic again.

  

• Timeouts: Setting time limits for expected responses. The sender assumes a failure if a response is not received within the timeout.

Idempotency and Quorum

• Idempotency: Designing operations that can be executed multiple times without unintended side effects. This is particularly important when dealing with potential message loss or retries in distributed systems.

• Quorum Systems: Require a majority of nodes to agree before committing changes. This ensures that the system can tolerate a certain number of failures while still maintaining consistency.

  

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

• Increased Latency: The extra hop through the intermediary service adds latency to the data retrieval.

• Resource Consumption: The intermediary service consumes compute, memory, and network bandwidth to handle data requests.

• Scalability Issues: As the number of clients and data requests increase, the intermediary service may become overloaded, leading to performance degradation.

What is the Valet Key Pattern?

The Valet Key pattern addresses the challenge of securely providing temporary, direct access to data stored in a system without requiring the data to pass through an intermediary service. It uses short-lived tokens (valet keys ) to grant clients temporary, direct access to specific resources (e.g., cloud storage buckets or files). Here's how it works:

• Client Requests Access: The client asks the application for permission to read/write data.

• Authorization: The intermediary service (the "key issuer") authorizes the client to access specific data.

• Key Generation: The key issuer generates a "valet key" (a security token) that grants the client temporary, restricted access to the data. This key includes information such as:

  • The location of the data.

  • The permissions granted to the client (e.g., read, write).

  • The expiration time of the key.

• Key Delivery: The key issuer securely delivers the valet key to the client.

• Direct Access: The client uses the valet key to access the data storage system directly, bypassing the intermediary service.

The Valet Key Pattern also allows precise control over data access by enabling granular scoping and revocation mechanisms. Valet keys can be configured to restrict access to specific operations (e.g., read-only, write), files, directories, or even time windows (e.g., valid for 10 minutes). For example, a key might grant temporary upload access to a single folder in cloud storage but block deletions or modifications. Cloud providers like AWS S3 or Azure Blob Storage support these features through pre-signed URLs or SAS tokens, which enforce these limits at the storage layer.

Security Implications

The Valet Key pattern also introduces some security considerations:

• Key Security: The entire system's security relies on the valet keys' security. Keys must be generated and delivered securely to prevent unauthorized access.

• Key Expiration: It is crucial to enforce key expiration to limit the window of opportunity for attackers.

• Permissions Management: Carefully manage the permissions granted to each valet key to prevent excessive access.

• Key Revocation: The system must provide a mechanism to revoke valet keys in case of compromise.

• Secure Key Issuing: The service responsible for generating the valet keys must be very well secured, as it would be a valuable target for attackers.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

What is the Claim-Check Pattern?

The Claim-Check Pattern is a messaging design pattern that separates the transmission of large data payloads from the actual message. Instead of sending the entire payload within the message, the pattern stores the payload in an external storage system (e.g., a database, blob storage, or file system) and sends only a reference to the stored data in the message. This reference is often called a "claim check," akin to a ticket you receive at a coat check. This is achieved through a multi-step process:

• Store the Payload: When a service needs to send a large message, it first uploads the payload to an external storage system. This could be a cloud storage service like Amazon S3, Azure Blob Storage, or a distributed file system.

• Generate a Reference: Once the payload is stored, the service or database generates a unique reference (e.g., a URL, file path, or database key) that points to the stored data.

• Send the Claim Check: Instead of sending the entire payload, the sender service sends a lightweight message containing only the reference (claim check) and any necessary metadata.

• Retrieve the Payload: The receiving service uses the claim check to fetch the payload from the external storage system when needed.

Claim-Check with Event Sourcing for Privacy

Event Sourcing is a pattern in which an application's state is determined by a sequence of events. However, storing all information directly in events can raise privacy concerns when dealing with sensitive data. Claim-Check improves privacy within Event Sourcing by ensuring sensitive data is not directly stored within the event stream.

• Storing Sensitive Data Separately: Instead of embedding sensitive data (e.g., personally identifiable information or PII) directly in events, store it in a secure external storage system.

• Generating Claim Check for Sensitive Data: Include only a claim check (reference) in the event log. This ensures that sensitive data is not exposed in the event stream.

• Access Control for Payload Retrieval: Implement strict access controls on the external storage system to ensure that only authorized services or users can retrieve the sensitive data.

• Audit Trail: Event sourcing naturally provides an audit trail of events. Combined with access logs for the storage location, a complete audit trail of data access is maintained.

• Data Retention Policies: Apply data retention policies to the external storage system to automatically delete or anonymize sensitive data after a specified period, ensuring compliance with privacy regulations like GDPR.

• Data Encryption: Sensitive data can be encrypted before storage, providing an additional layer of security.

Pros and Cons

Pros

• Reduced Message Size: By storing large payloads externally, messages become lightweight, improving performance and reducing bandwidth usage.

• Enhanced Security: Sensitive data can be stored separately and accessed with controlled authorization. Separate retention policies can be applied to data and metadata.

• Increased Scalability: Smaller messages reduce the load on message brokers, enabling the system to handle higher message volumes.

• Cost Efficiency: Storing large payloads in cost-effective storage systems (e.g., object storage) can be cheaper than transmitting them through a message broker.

Cons

• Increased Complexity: The pattern introduces additional components (e.g., external storage) and steps (e.g., uploading and retrieving payloads), which can complicate the system.

• Latency Overhead: Fetching payloads from external storage adds latency, which may not be acceptable in low-latency systems.

• Dependency on External Storage: The system becomes dependent on the availability/performance of the external storage system. If the storage system becomes unavailable, messages cannot be fully processed.

• Potential for Orphaned Data: If the claim check is lost or not processed, the stored data may become orphaned, consuming storage space without purpose.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

What is the CQRS Pattern?

The Command Query Responsibility Segregation (CQRS) pattern is a design approach that separates a data store's read and write operations into distinct models. The core principle is to use separate interfaces for querying data (reads) and commanding data (writes). The command model handles create, update, and delete operations, focusing on data integrity and business logic. The query model handles data retrieval and is optimized for fast reads. This separation allows for

1. Optimized Performance: Read and write operations can be tuned separately, improving overall performance.

2. Independent Scaling: Read and write databases can be scaled independently based on their specific load requirements.

CQRS on Single Database Systems

In a single database system, the command model may utilize a normalized database schema optimized for writes, and the query model may use denormalized data, or materialized views for performance enhancements. Data synchronization between the command and query models can be done synchronously or asynchronously.

Consider a micro-blogging application with a single "Posts" table.

Traditional Model:

• When a user creates a post, the system writes to the "Posts" table.

• When a user views a post, the system reads from the same "Posts" table.

• If the site becomes very popular, both read and write operations compete for resources, potentially slowing down the system.

CQRS Model:

• Write Model: The "Posts" table remains optimized for writes. It ensures data consistency when creating or updating posts.

• Read Model: A separate "PublishedPostsView" can be created. This view can use the "Posts" table to pre-calculate summaries or format the content for display. When a user views a post, the system reads from this optimized view. This view might not contain all the details of the original post, but only the data needed for display.

• Another read model might be created for search functionality, containing indexed terms extracted from the post content.

• When a new post is created, the write model updates the "Posts" table and then asynchronously updates the "PublishedPostsView" and the search index.

Both operations utilize the same underlying database. However, their access patterns and data structures are logically separated, improving system responsiveness.

CQRS on Multi-Database Systems

Implementing CQRS on multiple databases involves using separate databases for read and write operations. This approach is suitable for applications that require high scalability and can tolerate eventual consistency. The command model might use a relational database for transactional integrity, while the query model might use a NoSQL database for scalability and flexibility. This separation allows each database to be selected and configured based on its specific workload requirements, addressing the limitations of a single database handling diverse operations. Data synchronization between databases is typically asynchronous.

For example, an e-commerce platform might use a relational database for order processing (commands) and a NoSQL database for product catalog queries. When a customer places an order, the relational database handles the transaction. Product details for display are retrieved from the NoSQL database, which is optimized for fast, scalable reads. Changes to product information in the relational database are propagated to the NoSQL database asynchronously.

CQRS and Event Sourcing

Event Sourcing is often used in conjunction with CQRS. The command model stores events in an event store, while the query model builds read models by replaying events from the event store. Eventual consistency is inherent in this approach.

Pros and Cons

Pros

Apart from the ones already stated above,

• Increased Flexibility: Each model can use different database technologies and data models, providing flexibility and adaptability.

• Improved Maintainability: Separating concerns makes the codebase easier to understand and maintain.

Cons

• Eventual Consistency: Asynchronous synchronization can lead to eventual consistency, which may not be suitable for all applications.

• Data Duplication: Read models often involve data duplication, which can increase storage requirements.

• Infrastructure Overhead: Multiple databases and messaging systems may increase infrastructure costs.

• Synchronization Complexity: Ensuring data consistency between the read and write sides can be difficult.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

What is the Saga Pattern?

The Saga pattern is a design pattern that helps manage transaction updates across multiple services by breaking them down into a sequence of small local transactional updates, called "saga steps" or "subtransactions." Each step represents a unit of work that interacts with a single service. Once a step is completed, it triggers the next step in the sequence. If any step fails, the saga executes compensating updates to undo the changes made by the previous steps, ensuring that the system returns to its initial state.

Types of Saga Approaches

There are two main approaches to implementing the Saga Pattern: Orchestration and Choreography.

Orchestration

In this approach, a central orchestrator service coordinates the saga steps. The orchestrator tells each service when to execute its local transaction. It maintains the state of the saga and handles any failures by invoking compensating transactions. The orchestrator knows the entire saga flow.

How it works

The client initiates the saga by communicating with the orchestrator. The orchestrator then invokes the first service. Upon successful completion, the orchestrator moves to the next step, invoking the corresponding service. If a service fails, the orchestrator triggers compensating transactions in reverse order.

Choreography

In the Choreography approach, there is no central coordinator. Instead, each service involved in the saga knows its role and communicates with the other services through events or messages. Each service listens for specific events and performs local transactions when the appropriate event is received. The saga flow is distributed across the services.

How it works

The client initiates the saga by communicating with the first service. This service performs its transaction and publishes an event. Other services, listening for this event, perform their respective transactions and publish their events. This chain reaction continues until the saga is complete. If a service fails, it publishes a compensating event, triggering other services to execute compensating transactions.

Orchestration v/s Choreography

• Choreography has no single point of failure, as each service manages its part of the saga.

• Orchestration provides simplified error handling and monitoring with centralized control. In contrast, each service needs to handle its errors in Choreography, which can lead to complex error-handling logic.

• In Orchestration, the coordinator needs to know about all the services involved in the saga, which can lead to tight coupling. In contrast, in Choreography, services need to agree on the events and the order of transactions, which can lead to overhead in coordination.

Pros and Cons of Saga Pattern

Pros

• Data consistency: The Saga Pattern ensures data consistency across multiple services, even in the face of failures.

• Scalability: It allows for scalable and loosely coupled services, as each service can operate independently.

• Flexibility: The Saga Pattern can handle complex business transactions that involve multiple services and steps.

Cons

• Complexity: Implementing the Saga Pattern can be complex, especially for large-scale systems with many services.

• Error handling: Managing errors and compensating transactions can be challenging, especially in the choreography approach.

• Performance overhead: The need to track the saga's state and handle compensating transactions can introduce performance overhead.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.

Traditional data storage typically focuses on the current state of an entity. For example, in an e-commerce system, you might store the current state of a customer's order: items, quantities, shipping address, etc. Event sourcing takes a different approach. Instead of storing the current state directly, it stores the events that led to that state. Each event represents a fact that happened in the past. Think of it as a detailed log of transactions on your bank statement. These events are immutable and stored in an append-only event store. The core idea is that an application's state can be derived by replaying events in the order they occurred, just like you can get your current bank balance by replaying all the transactions from the beginning. This makes Event Sourcing particularly useful for applications that require a high degree of audibility and traceability.

How Event Sourcing Works

Every change to the application state is captured as an event object in an Event Sourcing system. These events are then stored in an event store, a database optimized for handling event data. Here's a step-by-step breakdown of how Event Sourcing works:

1. Event Creation: When a user performs an action (e.g., placing an order), the application creates an event object that describes the action.

2. Event Storage: The event is stored in the event store. This store is append-only, meaning events can only be added and never modified or deleted.

3. State Rebuilding: To determine the current state of an entity (e.g., an order), the application replays all the events related to that entity from the event store.

Imagine an e-commerce application. Instead of storing the current state of a shopping cart (items, quantities, etc.), we store events like "ItemAddedToCart," "ItemRemovedFromCart," "QuantityUpdated," etc. Each event represents a specific change to the cart. The current state of the cart can be reconstructed at any time by replaying these events in order.

Rebuilding State in Event Sourcing

Reconstructing the state from events involves reading all the events related to an entity from the event store and applying them in sequence to reconstruct the current state. It's like simulating all the changes that have occurred to construct the current state. For example, consider an e-commerce application where an order goes through various states like "Created," "Paid," and "Shipped." To determine the current state of an order, you would:

1. Retrieve all events related to the order from the event store.

2. Initialize an empty order object.

3. Apply each event to the order object in the order in which they were stored.

By the end of this process, the order object will reflect the current state of the order.

Use of Snapshots in Rebuilding State

As the number of events grows, replaying the entire event stream to reconstruct the state can become slow and inefficient. This is where snapshots come in. A snapshot is a saved state of an entity at a specific point in time. Instead of replaying all events from the beginning, the application can load the latest snapshot and then replay only the events that occurred after the snapshot was taken.

Pros and Cons

Pros

• Auditability: Event Sourcing provides a complete history of changes, making it easy to audit and trace an entity's state over time.

• Debugging: The ability to replay events makes it easier to debug issues by reproducing the exact state of an entity at any point in time.

• Flexibility: Since the state is derived from events, new features can be easily added, or existing features can be changed without affecting historical data.

Cons

• Complexity: Event sourcing adds complexity to your application. You must design your system around events, implement state rebuilding, and manage snapshots.

• Storage Requirements: Storing all events can lead to significant storage requirements, especially for long-running systems with many events.

• Data Privacy: Storing all events can raise privacy concerns, as historical data may contain sensitive information that must be managed carefully.

If you enjoyed this article, please hit the ❤️ like button.

If you think someone else will benefit from this, then please 🔁 share this post.