Parallel tests hang forever when a worker process dies #59037

The actionpack and actionview jobs of the Rails nightly build on main hang until the 30 minute timeout:

https://buildkite.com/rails/rails-nightly/builds/4818#01a12cc4-4544-4ccc-a528-a6519f42bf97

A parallel test worker crashes with [BUG] rb_objspace_reachable_objects_from_local: set is owned by another Ractor (https://bugs.ruby-lang.org/issues/22419), and the test run never finishes. The job then runs until the timeout instead of failing with the crash. The script below is a minimal reproduction of this, and it reproduces on main.

Steps to reproduce

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "activesupport", github: "rails/rails", branch: "main"
end

require "active_support"
require "active_support/test_case"
require "minitest/autorun"

class DeadWorkerTest < ActiveSupport::TestCase
  parallelize(workers: 2, threshold: 0)

  10.times do |i|
    test "test #{i}" do
      # Simulate a worker crash on the first test worker 0 runs
      Process.kill(:KILL, Process.pid) if parallel_worker_id == 0
      assert true
    end
  end
end

Expected behavior

The run finishes, and the tests that the dead worker did not finish are reported as errors.

Actual behavior

The run hangs forever after printing # Running:.

ActiveSupport::Testing::Parallelization::Server#shutdown waits in while @distributor.pending?. With the default RoundRobinDistributor, the tests queued for the dead worker are never taken, so the loop never ends. With work_stealing: true, the other worker takes them and the run finishes with a result not reported error.

This loop was introduced in #56175, which assigns tests to workers in round-robin order. #57053 fixed a hang when workers die during Server#shutdown by reaping dead workers, but only in wait_for_active_workers, which runs after this loop.

System configuration

Rails version: main (a3890ed)

Ruby version: 4.0.7