Skip to content

Log pipeline progress with tqdm

Bruno Cardoso requested to merge bc/add-progress-info into main

What does this merge request do and why?

First draft of showing a progress bar for a DoFn pipeline step.

Example on code-suggestions eval:

BatchRequestCompletions: 413it [08:02,  1.51s/it]

Example for duo-chat make-dataset-2:

ApplyQuestionVariations: 18it [00:53,  3.46s/it]

Ref https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/prompt-library/-/issues/101.

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
Edited by Bruno Cardoso

Merge request reports

Loading