Add support for timeouts to individual trial runs #423

We should avoid problematic individual trial runs causing the whole run to stall. As a result, we should introduce configurable timeouts to individual trials. This along with #377 should allow for fast, repeatable runs that hopefully don't get stuck or lead to lost work / progress.