What is Function Sharding in Serverless Computing?
Understanding How Data Computation Can be Divided-and-Conquered in Serverless Architecture
What is Function Sharding?
Serverless computing has changed how we build and deploy applications. It offers good scalability and cost savings. However, serverless architectures can face challenges as applications become more complex and get more traffic.
Function sharding is a divide-and-conquer concept similar to MapReduce. It spreads out the work of a single function across multiple function instances based on specific criteria, usually part of the input data. It’s like splitting a big task among several workers, each handling a particular section. This differs from the standard serverless model, where each function run is managed by one instance alone. Some criteria to split input across multiple instances of a function:
Data Locality: If the function works with data, function calls can be divided based on where the data is located. For instance, if user data is spread across different regions, user-related functions can be sharded accordingly.
Hashing: Each input value can be hashed, and the resulting hash value can be used to decide which function instance should process the call, distributing the load evenly.
Range-based: If the input parameter has a numerical range, the range can be split among the function instances.
Why Use Function Sharding?
Some of the reasons to use Function Sharding:
Higher Concurrency and Throughput
Function sharding spreads out the workload, allowing more tasks to run simultaneously. Multiple instances can work on different parts of the data together, cutting the total processing time and allowing to process data within function timeouts.
Better Scalability
Sharding allows horizontal scaling by adding more function instances to handle larger loads. This helps the application manage traffic spikes more effectively.
Fault Isolation
If one shard encounters an issue, it only impacts a small portion of the data, making the application more resilient and reliable.
High-Level Implementation
Choose a Sharding Key: Select a property from the input data to determine which function shard will process the data. For example, userId may be used when working with user data.
Map Data to Shards (Map Operation): Connect the sharding key to a specific function shard. This could be as simple as using a hash function or more complex routing rules.
Distribute and Execute: Based on the mapping rules, send the input data to the right function shards. Each function shard executes the assigned input data independently.
Shuffle & Aggregate Results (Reduce Operation): Combine the results from each function shard into a final output.
If you enjoyed this article, please hit the ❤️ like button.
If you think someone else will benefit from this, then please 🔁 share this post.