Add global flag for enabling asynchronous DNS resolution

[laurivosandi]

What is the problem this feature will solve?

Most Node libraries end up using dns.lookup which in turn uses blocking getaddrinfo from libc, this is discussed in detail here

This problem is particularly pronounced when making lots of outbound HTTP requests, it seems Node maxes out at around 20 requests per second being busy performing DNS lookups most of the time.

Patching applications to use asynchronous resolver requires in some cases extensive work so it would be good idea to add option to substitute dns.lookup with asynchronous one globally.

What is the feature you are proposing to solve the problem?

For example in Go this is achieved with environment variables:

export GODEBUG=netdns=go    # force pure Go resolver
export GODEBUG=netdns=cgo   # force native resolver (cgo, win32)

What alternatives have you considered?

No response

[tniessen]

cc @nodejs/dns @bnoordhuis

[bnoordhuis]

A flag is probably a bad idea. In the deep past node used c-ares (dns.resolve and friends) exclusively. dns.lookup was added explicitly because lots of things only work with the system resolver. Ex. mDNS.

At the moment dns.lookup goes through libuv's thread pool and that's a finite resource (look up UV_THREADPOOL_SIZE) that's shared with other operations but there's no reason uv_getaddrinfo and uv_getnameinfo couldn't use a bespoke thread pool that scales as needed.

[bnoordhuis]

There's a misunderstanding in the linked article that hasn't been true since node 10:

DNS requests in node appear asynchronous, but they're actually internally implemented as synchronous calls within node's internal libuv threadpool (which by default has only 4 threads). That means if you do >4 DNS lookups in parallel then you're going to block the libuv threadpool, even though they look like async IO. This will block every other DNS lookup, and also unrelated file IO and various crypto APIs

Libuv's thread pool reserves a fraction of the thread pool for "fast" operations (file I/O, cryptography, etc.). Slow-running DNS lookups therefore never completely block other operations.

[bnoordhuis]

I've thought about this for a bit and I think there is two things we can do:

1. Change the thread pool strategy in libuv. getaddrinfo(3) and getnameinfo(3) only need minimal stack space so it should be okay to spawn dozens or even hundreds of dns threads. Looks ugly in htop though. :-)

2. Cache DNS results very lightly in node. Strawman: cache for 1 second and refresh in the background so repeated lookups are fast without going stale. No knobs, no overrides, just a completely transparent implementation detail.

I don't plan on working on it myself but anyone who wants to exchange money for code is welcome to get in touch with me.

[kshitjj]

Hey @bnoordhuis,

I would like to work on this issue, the libuv thread pool is single-threaded for networking I am looking for ways to change network I/O to multithreaded.

As for Caching the DNS, I have started working on that. And I will put out a PR tomorrow.

[github-actions]

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[github-actions]

There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.

For more information on how the project manages feature requests, please consult the feature request management document.