Repository navigation
Potential issue with virtual threads #172
Description
Describe the bug
Sorry, this will be a bit vague as I'm having hard time understanding as well.
Related: #157
Version: 2.10.1
We are using org.newsclub.net.unix.AFUNIXSocketFactory. I have been investigating an issue with jdbc connection pools where the application would just hang for unclear reasons and jdbc pool would start throwing errors about pool being busy even though there was nothing going on. It looked like some sort of connection leak but I saw nothing on the thread dumps whenever this happened.
The problem solved itself as soon as I disabled Jetty virtual threads. I don't have a clear indication it was junixsocket that caused the problem but it felt like a good suspect. Unfortunately, it happens only on a specific environment and I can't reproduce locally. And on this specific environment, it is hard to change it so the application can use postgresql TCP socket instead of unix socket to rule it out.
So I've asked AI to create a minimal test. The result is not clear cut.
@Test
public void blockingRead_onVirtualThreads_completesForAllClients() throws Exception {
int clients = Integer.getInteger("benchmark.junixsocket.clients", 64);
int serverDelayMillis = Integer.getInteger("benchmark.junixsocket.server-delay-ms", 20);
int timeoutSeconds = Integer.getInteger("benchmark.junixsocket.timeout-seconds", 60);
byte payload = 0x2A;
Path socketPath = Files.createTempFile("junixsocket-vt-", ".sock");
Files.deleteIfExists(socketPath);
var failures = new AtomicInteger(0);
var completed = new CountDownLatch(clients);
var address = AFUNIXSocketAddress.of(socketPath.toFile());
try (var server = AFUNIXServerSocket.newInstance()) {
server.bind(address);
var acceptor = Executors.newSingleThreadExecutor();
var handlers = Executors.newCachedThreadPool();
try {
acceptor.submit(() -> {
for (int i = 0; i < clients; i++) {
try {
var clientSocket = server.accept();
handlers.submit(() -> {
try (clientSocket) {
Thread.sleep(serverDelayMillis);
clientSocket.getOutputStream().write(payload);
clientSocket.getOutputStream().flush();
} catch (Throwable t) {
failures.incrementAndGet();
}
});
} catch (IOException e) {
failures.incrementAndGet();
}
}
});
long startNanos = System.nanoTime();
try (var vexec = Executors.newVirtualThreadPerTaskExecutor()) {
for (int i = 0; i < clients; i++) {
vexec.submit(() -> {
try (var socket = AFUNIXSocket.newInstance()) {
socket.connect(address);
int result = socket.getInputStream().read();
if (result != payload) {
failures.incrementAndGet();
}
} catch (Throwable t) {
failures.incrementAndGet();
} finally {
completed.countDown();
}
});
}
}
boolean doneInTime = completed.await(timeoutSeconds, TimeUnit.SECONDS);
long elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
System.out.printf("%njunixsocket VT test: clients=%d, delayMs=%d, elapsedMs=%d, doneInTime=%s, failures=%d%n%n",
clients, serverDelayMillis, elapsedMs, doneInTime, failures.get());
assertThat(doneInTime).isTrue();
assertThat(failures.get()).isZero();
} finally {
acceptor.shutdownNow();
handlers.shutdownNow();
acceptor.awaitTermination(5, TimeUnit.SECONDS);
handlers.awaitTermination(5, TimeUnit.SECONDS);
}
} finally {
Files.deleteIfExists(socketPath);
}
}
This test hangs for 5 seconds (awaitTermination) and then gets a few failures.
If I change newVirtualThreadPerTaskExecutor to newCachedThreadPool, the test finishes immediately without any failure.
Can't say what that means exactly either but I was hoping perhaps you can? If this isn't helpful, feel free to close this issue as I lack the time to provide more right now and will go ahead with disabling virtual threads anyway.
Activity
kohlschuetter commentedon Mar 29, 2026
Hi @natan-abolafya, thanks for reporting!
It looks like we're hitting a race between
acceptandconnect:connectmay fail with a "connection refused" error well within the specified connection timeout.A reasonable fix will be to keep retrying to connect if running from a virtual thread is detected (fix is coming for 2.11.0).
In the meantime: either don't use virtual threads or simply catch the SocketException when calling
connect, and keep retrying.common: AFSocketImpl:connect Permit spurious ConnectionRefused
common: test: Add VirtualThreadConnectTest
natan-abolafya commentedon Mar 30, 2026
Thanks @kohlschuetter.
I am not actually using it directly, as it's tomcat's jdbc pool using the AFUNIXSocketFactory, so for now I have disabled virtual threads. That said, maybe this issue is not the same issue I'm hitting but I'll give it a try when there is a new version.
kohlschuetter commentedon Mar 30, 2026
Please verify by using the latest 2.11.0-SNAPSHOT build, and report back here.
Alternatively, add this selftest jar snapshot to your project, which should contain everything in one piece.
natan-abolafya commentedon Apr 14, 2026
Hi. Sorry for the late response.
I've finally tried it and unfortunately it did not resolve my original issue with jdbc connection leak. That said my hypothesis that it is caused by junixsocket is a weak one as I'm having difficulty finding a decent evidence. The challenge is that I can only reproduce it on a specific deployment, not in an isolated way.
Azahe commentedon Apr 14, 2026
I just wanted to chip in with some context from our side - we are using Hikari connection pool with virtual threads and it looks like we might be hitting similar problem.
We are using junixsocket as a socket factory in a postgres driver. In a test that relies on heavy parallelism with limited system resources (that is in a github action runner - does not happen on my laptop) we sometimes see multiple connections being timed out waiting. Initially I thought it just might be insufficient resources thing - but the test takes ~80s when failing and 15s when everything works so it does not make sense. It definitely looks like something is hanging. Postgres is not seeing any deadlocks.
Switching from unix sockets to regular tcp connections (otherwise same stack) solves the problem.
Sadly the selftest snapshot jar does not seem to fix the issue - although it looked very promising.
Will also try to come up with some actual evidence as we continue to debug the issue.
Also I noticed that I cannot profile that test locally as it always causes it to hang (vs no profiler run) - but I am not sure that its related to the issue.
natan-abolafya commentedon Apr 14, 2026
I don't know if any of this is relevant or not but I figured I'd share some more info.
We've used virtual threads in the past but not as heavily to trigger the issue and never came across this issue. Only on the latest release of our application we've hit this issue when we enabled virtual threads much more widely. Now we went back to original use of virtual threads and we've hit the issue again the other day after 10 days of uptime where it looked like there was a slow build-up.
There could be various changes that may have caused it but one thing I could think of that it could be relevant is that we've gone from Java 21 to Java 25.
Azahe commentedon May 27, 2026
Hello!
It seems have found the source of our issue.
In a case where
ForkJoinPool.commonPool()has a size of 1 (which happens when a machine has 2 cpus) when there are at least 2 db connections that depend on each other it is possible to deadlock.On virtual threads with java 25. Unix sockets used to connect to postgres (but its not limited to databases - its just simpler to explain this way).
In our case it was quite simple dependency:
Assuming that first transaction manages to update shared resource and lock it for its transaction - then it yields its execution to second transaction.
Then second db connection insert something (or whatever) and does not yield but progresses to updating the shared resource - which is locked by first transaction - and so it yields.
Normally what would be needed to progress is to just let first transaction finish its work, however if on
ForkJoinPool.commonPool()(that has size one) the running thread becomes the one fromVirtualThreadPollerNaivewaiting in infinite loop for the second locked connection it becomes impossible - everything happens onForkJoinPool.commonPool()which has size 1 so anything happening in parallel is impossible so unlocking is impossible and so application-level deadlock is created.I think that if
VirtualThreadPollerNaivewould run its waiting threads on separate pool rather thanForkJoinPool.commonPool()then such a deadlock could be made impossible - if that other pool had size scaled with the count of possible waiting jobs. But there might be reasons for using common pool that I do not see.Tldr: on 2 cpus VirtualThreadPollerNaive waiting can starve interdependent work happening on virtual threads.
I am attaching a commit with a test reproducing the situation - written with help from AI, seems quite verbose but roughly seems to capture what I described, also the test passes after using dedicated pool for polling (EDIT: only because my laptop has
availableProcessorslarger than 1 - using dedicated pool is not enough it has to be able to wait concurrently for connection dependence to not be an issue)0001-Test-for-deadlock-presence-with-possible-fix-comment.patch
I run it like:
Azahe commentedon Jun 12, 2026
@kohlschuetter sorry to ping but would love to hear some feedback on the comment above.
8 remaining items
kohlschuetter commentedon Jul 2, 2026
Hi @natan-abolafya, hi @Azahe,
Please verify with the latest code on main. You also try the latest SNAPSHOT on
maven-snapshots, particulary https://central.sonatype.com/repository/maven-snapshots/com/kohlschutter/junixsocket/junixsocket-selftest/2.11.0-SNAPSHOT/junixsocket-selftest-2.11.0-20260702.181540-2-jar-with-dependencies.jarThanks for reporting!
Azahe commentedon Jul 10, 2026
Hi so I tested the snapshot against our particular suite and I can confirm it works now - no deadlocks... but 😅
I peeked at commit with the fix and from what I see it introduced
ThreadUtil.newWorkStealingPool()which creates a new pool the size ofRuntime.getRuntime().availableProcessors()- this fixes my particular case but isn't there a broader issue still present?So lets imagine a setup where there are constant
Pavailable processors andNwating threads on the pool. Additionaly lets assume that all the threads are interdependent on each other in some way (which is quite pessimistic but still).Won't the issue be possible in the case where
N > P? Any time there areP+1co-dependent threads trying to poll - andPare polling and hanging - it seems possible for one critical thread to not be polling (and waiting in the queue to the pool causing other to hang). So like my case but generalized.My ideas:
not use ForkJoinPool (which has target parallelism) and to use regular
ThreadPoolExecutorand allow it to scale within some sane range (where hopefully it would only neeed defaultRuntime#availableProcessorsduring normal operation) - which seems less efficient and it feels weird to scale virtual threads with platform ones - like aren't the benefits lost then?size of
ForkJoinPoolcould be expected to be passed by the user config (withRuntime#availableProcessorsas a very reasonable default) - this would make sense in a setup like ours where we have a concrete size of hikari thread pool and keeping it synced with VirtualThreadPollerNaive pool should guarantee corectness... but in general case and given the nature of virtual threads (you can create million of them) it seems awkward.resign from one-thread-one-poller? - currently the
do...while(true)is kinda the problem because it busily waits for polling to finish while (I think - I am focusing on one class here) it could check other threads waiting status? So I imagine creatingRuntime#availableProcessorsForkJoinPoolfor waiting - yes, but then havePollJobtake a list/array/collection of file descriptors to poll? I am not sure how efficient something like that could be but if file descriptors were distributed among threads evenly shouldn't it be ok? On the other hand the poller hasNaivein its name so maybe that would be an overcomplication?I guess it is sort of my fault that it was harder to notice this - after all my test focused on my exact setup, sorry could have thought of this earlier.
Azahe commentedon Jul 10, 2026
I tried to verify my comment above with AI, and it seems my 3rd idea was the closest to standard practice but in general it could look like:
which is largely what already happens but suggests having one polling thread that polls all file descriptors (which
PollFdseems to already support but it would also need a wakeup fd for dynamically adding fds - new concept for me btw but seems very nice) - there could be multiple threads used for this polling but its (?)usually unnecessary (idk, this should probably be performance tested if anything)It also linked: https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/nio/ch/Poller.java - how they wait for fds in openjdk, and it seems that in
VThreadsPollerGroupthese do delegate to a (single?) system-wide "master poller"Azahe commentedon Oct 7, 2026
@kohlschuetter I went ahead and tried the shared-poller idea from my comments above: #180
Virtual threads register their fd and park, and a single platform thread polls all registered fds (plus a wakeup pipe) and unparks whichever are ready. That way the number of waiting virtual threads is no longer bounded by pool size, which covers the general
N > Pcase and not just the 1–2 CPU one. The naive poller is kept as a fallback (-Dorg.newsclub.net.unix.VirtualThreadPoller.naive=true) and now usesForkJoinPool.managedBlock. The PR addsVirtualThreadPollerTestwith more interdependent waiters than processors.Feel free to reshape it or take only parts of it, whatever fits best.
common: Multiplex virtual-thread polling onto a shared poller thread
common: Multiplex virtual-thread polling onto a shared poller thread
kohlschuetter commentedon Oct 15, 2026
@Azahe @natan-abolafya Please verify with the latest
2.11.2-SNAPSHOTversion, or directly viahttps://central.sonatype.com/repository/maven-snapshots/com/kohlschutter/junixsocket/junixsocket-selftest/2.11.2-SNAPSHOT/junixsocket-selftest-2.11.2-20261015.151511-1-jar-with-dependencies.jar. Thanks for your help!