I am having a problem where my application threads all suddenly die to IO errors as the same time, eg.
Exception in thread "Thread-24" java.io.IOException: Connection reset by peer
The application is a server that listens to GooglePubSub, picks up jobs, and invokes them against a remote API.
It uses Channels to fan-in and fan-out the tasks, and ensuring that only the central Dispatcher component is updating the local datastore.
When the Worker jobs, which are a small fixed number (to avoid overloading the remote API), have had a certain number of timeouts, all of a sudden the entire threadpool appears to go down at the same time.
The Pubsub Subscribers are using a default Threadpool executor, not the Couroutines dispatcher. The Supervisor Job and Worker Jobs are using Dispatchers.Default and
Dispatchers.IO respectively, which should be the same threadpool.