SDK

Python

Description

We already have a permutation, which acts as a basic map dataset. However, we should create an elastic dataset, for users that need elastic determinism. This will extend what is provided by permutation:

• The ability to shuffle per-epoch
• Wrap the concept of splits in a way to enable MDS-style elastic determinism
• Ability to concurrently consume a single split with multiple workers

Some of these features may filter down into the base permutation while others will go into a new dataset class that sits on top of permutation and provides a torch iterable dataset.