Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Sequential execution for local pipeline #579

Open
gabrielmbmb opened this issue Apr 25, 2024 · 1 comment
Open

[FEATURE] Sequential execution for local pipeline #579

gabrielmbmb opened this issue Apr 25, 2024 · 1 comment
Assignees
@gabrielmbmb
Labels
enhancement New feature or request
Milestone
1.3.0

Comments

@gabrielmbmb
Copy link
Member

Description

As mentioned by @alvarobartt and Ellamind team, it would be nice to have a sequential model for executing the pipeline, in which no multiprocessing & batching is used.

The idea would be to load each step, process all the data, unload the step, load the next step, ...
@gabrielmbmb gabrielmbmb self-assigned this Apr 25, 2024
@gabrielmbmb gabrielmbmb added this to the 1.2.0 milestone Apr 25, 2024
@gabrielmbmb gabrielmbmb added the enhancement New feature or request label Apr 25, 2024
@bjoernpl
Copy link
Contributor

Just adding here that a goal of this would be to enable proper debugging within steps/tasks in the pipeline. Thanks for picking this up! :)

@gabrielmbmb gabrielmbmb modified the milestones: 1.2.0, 1.3.0 May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gabrielmbmb gabrielmbmb

Labels
enhancement New feature or request
Projects
⚗️ distilabel
Status: No status
Milestone
1.3.0
Development

No branches or pull requests
2 participants
@gabrielmbmb @bjoernpl