Server-Side Load Balancer:
Let’s imagine that you have a distributed system that consists of many services (applications) running on different computers. However, when the number of users is large, a service (application) is usually created multiple replicas. Each replica runs on a different computer. At this time, " Load Balancer " appears. It helps to distribute incoming traffic equally among servers.
Traditionally, Load Balancers are components placed at the Server Side. When the requests come from the Client , they will go to the load balancer, and the load balancer will designate a Server for the request. The simplest algorithm used by the load balancer is random designation, a.k.a Round Robin.
Client-Side Load Balancer:
When the load balancer is located at Client side, it will actively decide which server it will send requests to based on some criteria.
Actually, servers are different. They differ from each other in the following criteria:
- Availability: No server also works at all times.
- Performance: The speed of servers is different.
- Geography: Servers are located in different locations, for instance, they are located in different countries. They may be close to this Client but far from other Client.
Client-side load balancers usually send requests to servers in the same zone, or with a fast response.
Spring Cloud - Ribbon:
Ribbon is part of Netflix Open Source Software (Netflix OSS) family. It is a library that provides a Client- side load balancer.
Spring Cloud creates APIs to help you to easily use Ribbon libraries.
Below are the main concepts related to the Ribbon :
- List of servers.
- Filtered list of servers.
- Load Balancer
- List Of Servers :
A list of servers that can respond to a particular service for 1 Client . For example, 1 Client needs information about the weather. There will be a list of servers being able to provide this information. This list consists of servers configured directly in the Client application, and servers discovered by the Client.
- Filtered List of Servers:
We continue the above example. One Client needs weather information, and a list of servers can provide such information. However, not all of those servers operate, or those servers are too far from the Client; therefore, they responds very slowly. The Client will remove these servers from the list, and eventually it will have a more suitable list of servers (a filtered list).
- Load Balancer (Ribbon) :
Ribbon is a load balancer. It is a Client -side component. It decides which server will be called (of the filtered list of servers).
There are some strategies to make a decision. But they usually rely on a " Rule Component " to make a true decision. By default, the Spring Cloud Ribbon uses the ZoneAwareLoadBalancer strategy (Servers in the same zone as Client ).
Rule Component is a smart module. It creates a " Call or uncall " decision.
Smart Load Balancing:
For load balancing, there are some trickiness and hence need to be smart about it:
-
Suppose that Server A has a high error rate while Server B has low error rate. However, Server B maybe be busy or even overwhelmed. It does not make sense that all traffic goes to Server B.
-
When Server is newly started and it has no historical data to analyze.
-
When Server has high error rate and hence load balancing may route no traffic to it. It does not make sense to route no traffic to it forever since it may recover.
From above points, the problem is really (1) the criteria of determining bad servers which can be ruled out as routing candidates (2) rate limiting so that not to throttle servers.
Spring Cloud Ribbon has two more rules beside Round Robin rule:
- AvailabilityFilteringRule:
By default, an instance is circuit tripped if the RestClient fails to make a connection to it for the last three times. Once an instance is circuit tripped, it will remain in this state for 30 seconds before the circuit is deemed as closed again. However, if it continues to fail connections, it will become “circuit tripped” again and the wait time for it to become “circuit closed” will increase exponentially to the number of consecutive failures.
This rule will skip servers that are deemed “circuit tripped” or with high concurrent connection count.
- WeightedResponseTimeRule
For this rule, each server is given a weight according to its average response time. The longer the response time, the less weight it will get. The rule randomly picks a server where the possibility is determined by server’s weight.
Smart load balancers can provide advanced features like
- Establishing affinity between clients and servers by dividing them into zones (like racks in a data center) and favor servers in the same zone to reduce latency
- Keeping statistics of servers and avoid servers with high latency or frequent failures
- Keeping statistics of zones and avoid zones that might be in outage
Utilizing the advanced features requires using one of the client provided in Ribbon as it has integration with load balancer and provides input to the load balancer statistics.
References: