NGINX DNS load balancer – worker_connections

I’m running a Dnsmasq server that points different DNS zones to various different DNS servers (home and over VPN). For my home networks I have three Windows DNS servers handling local zones. I also run a NGINX UDP load balancer that distributes local zones queries across those three Windows DNS servers.

For a while I’ve been having various issues.

2018/12/22 15:31:31 [alert] 2002#2002: *21123 10240 worker_connections are not enough while connecting to upstream, udp client: xxx.xxx.xxx.xxx, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.xxx:53", bytes from/to client:0/0, bytes from/to upstream:0/0
2018/12/22 15:31:31 [alert] 2002#2002: *21124 10240 worker_connections are not enough while connecting to upstream, udp client: xxx.xxx.xxx.xxx, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.xxx:53", bytes from/to client:0/0, bytes from/to upstream:0/0
2018/12/22 15:31:31 [alert] 2002#2002: *21125 10240 worker_connections are not enough while connecting to upstream, udp client: xxx.xxx.xxx.xxx, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.xxx:53", bytes from/to client:0/0, bytes from/to upstream:0/0

Even if I have worker_connections setting set to 10240… it’s still not enough.

2018/12/22 15:18:05 [crit] 19142#19142: accept4() failed (24: Too many open files)
2018/12/22 15:19:20 [crit] 19140#19140: accept4() failed (24: Too many open files)
2018/12/22 15:20:20 [crit] 19141#19141: accept4() failed (24: Too many open files)
2018/12/22 15:21:20 [crit] 19143#19143: accept4() failed (24: Too many open files)
2018/12/22 15:20:17 [alert] 19141#19141: *6559 socket() failed (24: Too many open files) while connecting to upstream, udp client: xxx.xxx.xxx.xxx, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.xxx:53", bytes from/to client:0/0, bytes from/to upstream:0/0
2018/12/22 15:20:17 [alert] 19141#19141: *6560 socket() failed (24: Too many open files) while connecting to upstream, udp client: xxx.xxx.xxx.xxx, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.xxx:53", bytes from/to client:0/0, bytes from/to upstream:0/0
2018/12/22 15:20:17 [alert] 19141#19141: *6561 socket() failed (24: Too many open files) while connecting to upstream, udp client: xxx.xxx.xxx.xxx, server: 0.0.0.0:53, upstream: "xxx.xxx.xxx.xxx:53", bytes from/to client:0/0, bytes from/to upstream:0/0

Or I get too many open files.

I kept playing with worker_processes, worker_rlimit_nofile, and worker_connections options. No matter how much I increase nothing helps. Above errors will eventually come back.

Solution turns out to be a very simple. All I had to add are proxy_timeout and proxy_responses options to the server{} directive.

server {
    listen 53 udp;
    proxy_timeout 1s;
    proxy_responses 1;
    proxy_pass dns_backend;
}

proxy_timeout determines how long to wait for a response. Since it’s all locally it shouldn’t take more than one second.

proxy_responses determines how many responses it expects from request.

Once those two options are in place those errors never show back up! I reduce worker_connections back down from 10240 to 1024!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.