Add pagination for experiment runs
What the change will do and why it's important
Instead of running the entire dataset each time for the evaluation, we can allow for pagination against the dataset to include only specific examples if the following user wants to test on a fraction of the experiements.
Problem
Currently, our experiment runner processes the entire dataset for each run. This approach can be:
- Time-consuming and resource-intensive for large datasets
- Inefficient for quick tests or partial evaluations
- Challenging when we need to focus on specific subsets of data
Edited by Nathan Weinshenker