Load balancing

RPC load balancing

You can use load balancers to scale your application and/or perform zero-disconnect deployments (by doing a rolling update of RPC servers without restarting WebSocket servers).

Using Linkerd

Check out this blog post: Scaling Rails web sockets in Kubernetes with AnyCable.

Using Envoy

Envoy is a modern proxy server which supports HTTP2 and gRPC.

See the example configuration in the anycable-go repo.

Using NGINX

You can use NGINX gRPC module to distribute traffic across multiple RPC servers.

The minimalist configuration looks like this (credits goes to avlazarov):

upstream grpcservers {
    server 0.0.0.0:50051;
    server 0.0.0.0:50052;
}

server {
    listen 50050 http2;
    server_name localhost;

    access_log /var/log/nginx/grpc_log.json;
    error_log /var/log/nginx/grpc_error_log.json debug;

    location / {
        grpc_pass grpc://grpcservers;
    }
}

gRPC clients (more precisely, grpc-go used by anycable-go) provide client-level load balancing via DNS resolving. If the provided hostname resolves to multiple A records, a client connect to all of them and use round-robin strategy to distribute the requests.

To activate this mechanism, you MUST provide use the following schema to build an URI: dns://[authority]/host[:port].

For example, when using Docker, you can rely on its internal DNS server and omit the authority part altogether: ANYCABLE_RPC_HOST=dns:///rpc:50051 (three slashes!). See the docs.

Since gRPC clients performs the DNS resolution only during the connection initialization, newly added servers (in case of auto-scaling) are not picked up. To resolve this issue, you can configure a max connection lifetime at the server side, so, connections are recreated periodically (that also triggers re-resolution).

You can control gRPC connection lifetimes via the rpc_max_connection_age configuration option for AnyCable RPC server (could be also configured via the ANYCABLE_RPC_MAX_CONNECTION_AGE env variable). It's set to 300 (seconds, thus, 5 minutes) by default, so you're likely don't want to change it.

You can also monitor the current number of gRPC connections by looking at the AnyCable-Go's grpc_active_conn_num metrics value.

Using a fixed list of RPC addresses

You can also provide a static list of gRPC servers to spread out the calls using the special grpc-list:// scheme:

$ anycable-go --rpc_host=grpc-list://grpc-list://rpc1.example.com:50051,rpc2.example.com:50051

...
RPC controller initialized: grpc-list://grpc-list://rpc1.example.com:50051,rpc2.example.com:50051 (concurrency: 28, impl: grpc, enable_tls: false, proto_versions: v1, proxy_headers: cookie, proxy_cookies: <all>) context=rpc

This is useful when you run AnyCable in environments without service discovery and a known list of server addresses (e.g., when using Kamal).

WebSocket load balancing

There is nothing specific in load balancing AnyCable WebSocket server comparing to other WebSocket applications. See, for example, NGINX documentation.

NOTE: We recommend to use a least connected strategy for WebSockets to have more uniform clients distribution (see, for example, NGINX).

AWS ECS 🔗

Load Testing