Questions Tagged [Pytorch-Dataloader]
Ask Question the Pytorch-Dataloader Tag Has No Usage Guidance. 23 Questions More Bountied 0 Unanswered Frequent Score Unanswered (My Tags) Apply Filter 61...
![Questions Tagged [Pytorch-Dataloader]](https://www.alledaags.nl/uploads/questions-tagged-pytorch-dataloader_1200.jpg)
The pytorch-dataloader tag has no usage guidance.
23
questions
61
votes
1
answer
50k
views
Pytorch. How does pin_memory work in Dataloader?
I want to understand how the pin_memory parameter in Dataloader works.
According to the documentation:
pin_memory (bool, optional) – If True, the data loader will copy tensors into CUDA pinned memory ...
9
votes
4
answers
23k
views
pytorch torchvision.datasets.ImageFolder FileNotFoundError: Found no valid file for the classes .ipynb_checkpoints
Tried to load training data with pytorch torch.datasets.ImageFolder in Colab.
transform = transforms.Compose([transforms.Resize(400),
transforms.ToTensor()])
...
2
votes
3
answers
4k
views
Pytorch Problem: My jupyter stuck when num_workers > 0
This is a snippet of my code in PyTorch, my jupiter notebook stuck when I used num_workers > 0, I spent a lot on this problem without any answer. I do not have a GPU and I work only with a CPU.
...
0
votes
1
answer
242
views
How does one create a pytoch data loader using an interleaved hugging face dataset?
When I interleave data sets, get a tokenized batch, feed the batch to the pytorch data loader, I get errors:
# -*- coding: utf-8 -*-
"""issues with dataloader and custom data sets
...
26
votes
3
answers
41k
views
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'
I am trying to load the dataset using Torch Dataset and DataLoader, but I got the following error:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'
the code I use is:
...
6
votes
1
answer
11k
views
pytorch dataloader - RuntimeError: stack expects each tensor to be equal size, but got [157] at entry 0 and [154] at entry 1
I am a beginner with pytorch. I am trying to do an aspect based sentiment analysis. I am facing the error mentioned in the subject. My code is as follows: I request help to resolve this error. Thanks ...
5
votes
1
answer
5k
views
Customizing the batch with specific elements
I am a fresh starter with PyTorch. Strangely I cannot find anything related to this, although it seems rather simple.
I want to structure my batch with specific examples, like all examples per batch ...
5
votes
2
answers
13k
views
How do I load the CelebA dataset on Google Colab, using torch vision, without running out of memory?
I am following a tutorial on DCGAN. Whenever I try to load the CelebA dataset, torchvision uses up all my run-time's memory(12GB) and the runtime crashes. Am looking for ways on how I can load and ...
5
votes
1
answer
4k
views
torch.nn.CrossEntropyLoss over Multiple Batches
I am currently working with torch.nn.CrossEntropyLoss. As far as I know, it is common to compute the loss batch-wise. However, is there a possibility to compute the loss over multiple batches?
More ...
4
votes
1
answer
6k
views
Strange Cuda out of Memory behavior in Pytorch
Edit: SOLVED- Problem relied on the number of workers, lowered them, problem solved
I am using a 24GB Titan RTX and I am using it for an image segmentation Unet with Pytorch,
it is always throwing ...
4
votes
2
answers
2k
views
Pytorch dataloaders : Bad file descriptor and EOF for workers>0
Description of the problem
I am encountering a strange behavior during a neural network training with Pytorch dataloaders made from a custom dataset. The dataloaders are set with workers=4, pin_memory=...
2
votes
0
answers
1k
views
Setting Hugging Face dataloader_num_workers for multi-GPU training
Should the HuggingFace transformers TrainingArguments dataloader_num_workers argument be set per GPU? Or total across GPUs? And does this answer change depending whether the training is running in ...
1
vote
2
answers
912
views
What I missing here, using ImageFolder to get the full folder name as labels for MNIST-double dataset images?
I would like to use dataset.ImageFolder to create an Image Dataset.
My current image directory structure looks like this:
1: In train images, I have subfolders which are my labels contain 00, 01, and ...
1
vote
1
answer
1k
views
How does the queue in Pytorch DataLoader work with num_workers >= 2?
Do I understand the following correctly?
When num_workers >=1, the main process pre-loads prefetch_factor * num_workers batches. When the training loop consumes one batch, the corresponding worker ...
1
vote
1
answer
2k
views
OSError: [Errno 22] Invalid argument | _pickle.UnpicklingError: pickle data was truncated
When trying to train a ResNet I get this error. Any help as to why this happens would be appreciated.
This happens when I try to iterate through the Dataloader:
File "C:\Users\JCout\AppData\Local\...
Elena Rostova holds a Master's degree in Public Health Journalism. She covers groundbreaking medical research, holistic wellness trends, mental health awareness, and nutritional science.
