Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate zipping preprocessed files to help save inodes #472

Open
roedoejet opened this issue Jun 17, 2024 · 1 comment
Open

Investigate zipping preprocessed files to help save inodes #472

roedoejet opened this issue Jun 17, 2024 · 1 comment
Labels
enhancement New feature or request
Milestone
beta

Comments

@roedoejet
Copy link
Member

roedoejet commented Jun 17, 2024 •
edited by SamuelLarkin
Loading

@SamuelLarkin suggests fuse-mounting a zip archive or a new dataloader that preprocesses the audio on-the-fly (if it doesn't slowdown training aka makes the trainer wait on preprocessing) to help circumvent the perennial inode problem for large datasets.
@roedoejet roedoejet added the enhancement New feature or request label Jun 17, 2024
@roedoejet roedoejet added this to the beta milestone Jun 17, 2024
@joanise
Copy link
Member

joanise commented Sep 9, 2024

@roedoejet This feels like a 1.0 milestone to me...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
beta
Development

No branches or pull requests
2 participants
@roedoejet @joanise