Framework: Enable the declarative model of workflow definition
Summary
The current workflow definition model follows a procedural approach, where tasks are explicitly defined with input parameters and return values. This limits flexibility and makes it challenging to orchestrate complex workflows. We need to transition to a declarative model, similar to a Domain-Specific Language (DSL), where we define "what" is needed for a workflow rather than "how" to run it. This new approach will allow for more flexible task orchestration, easier composition of tasks, and a more intuitive way to define workflows.
The proposed solution involves introducing a new task framework with a runtime context for shared state, implementing a declarative model for workflow definitions, and refactoring existing cluster-related functions to work within this new paradigm.
Success criteria
-
Implement a new task framework that uses a shared context for passing information between tasks, replacing the current model of explicit input parameters and return values. -
Develop a runtime context that encapsulates request-scoped information and provides a shared state mechanism for tasks. -
Create a declarative interface or DSL for defining workflows, allowing users to specify "what" is needed rather than "how" to execute the workflow.