I have question on the topic of Queueing requests. Can you give a general idea how can we implement this? More specifically, how the other requests will know that there was already a request which misses the cache and trying to fill it and now we have to wait till that happen?
Usually the auto-scaling or spinning of new instances happens with triggers which occur before the existing system reaches full capacity which allows the news instances to take their time (or cold start) to warm up.
Also, p99 is usually calculated over a period of time and cold start might be insignificant when compared to that period of time.
I have question on the topic of Queueing requests. Can you give a general idea how can we implement this? More specifically, how the other requests will know that there was already a request which misses the cache and trying to fill it and now we have to wait till that happen?
Hey Piyush, thanks for taking time to read the article. This week's article explains how you can avoid a stampede https://newsletter.scalablethread.com/p/how-to-avoid-cache-stampede-or-dogpile
Hope that answers your question.
I have question but bit out of topic,
How do you manage burst traffic without impacted to p99 latency ?
from my experienece, spin off new instance tooks time due to "cold start"
Andy, thanks for reading the article.
Usually the auto-scaling or spinning of new instances happens with triggers which occur before the existing system reaches full capacity which allows the news instances to take their time (or cold start) to warm up.
Also, p99 is usually calculated over a period of time and cold start might be insignificant when compared to that period of time.