Do not wait on clients of terminated event loops in shutdownNow()

[codeconsole]

Problem

shutdownNow() never returns when it is called again on a server that has already been shut down, if a client connected just as the first shutdown began. Seen with 1.47.0 and on main, with Netty 4.2.

shutdown() stops the event loop groups with no quiet period. A client accepted just before that can have its registration run on its worker event loop after the loop has closed its channels, so it becomes active and is added to channelGroup while the loop terminates. A later shutdownNow() then blocks in closeClients():

channelGroup.close().syncUninterruptibly();

Closing that client has to go through its terminated event loop, which rejects the work, so neither the client's close future nor the group's ever completes.

We hit this in Grails, where the application context stops the embedded server and a JVM shutdown hook stops it again as the JVM exits. That second shutdownNow() kept the JVM from exiting.

Fix

Drop the channel group once the event loops have terminated, so a later shutdownNow() has no clients left to wait for:

if (workerGroup != null) {
    workerGroup.terminationFuture().syncUninterruptibly();
}

// A client that connected while the event loops were shutting down can still be in the group, but its event
// loop has terminated and would never complete closing it. A later shutdownNow() must not wait for that.
channelGroup = null;

bind() creates a new group, so a server that is shut down and bound again still works.

This does not close such a client's socket, which would also need its event loop; it only stops shutdownNow() from waiting on it.

Tests

MongoServerShutdownTest reproduces the hang deterministically. The server is bound with a single worker thread, and a mock backend's handleClose holds that thread until a second client's registration is queued behind it and the server has started to shut down. Without the fix the second shutdownNow() times out every time; with it, it returns at once.