[Problem Solved] Copying large files problem

[JohnJ]

Hello all. My PC is Linux mint 21.3 Virginia and Cinnamon 6.0.4.
Having problems copying large files. Have tried Copy and Paste in Nemo, mirror FreeFilesync, etc. Initially I thought that copying the file onto my USB was the problem - Nemo Starts at 155mb/sec then rapidly slows up to 75% and 20mb/sec. Then all stops and error code says 'Error splicing file: File is too large'. In FreeFilesync there is no error code but the mirror copy gets to 75% ish and then just locks up. Note. This is a good file and runs perfectly. Thoughts?

 

[dos2unix]

USB drives that are formatted with Fat32 have a size limit.
If you reformat the drive with exFat, then you can copy large files.

 

[JohnJ]

USB drives that are formatted with Fat32 have a size limit.
If you reformat the drive with exFat, then you can copy large files.

Oh, I see. Although When I try copying on my hard drive (it's formatted as NTFS) same problem.

 

[JohnJ]

USB drives that are formatted with Fat32 have a size limit.
If you reformat the drive with exFat, then you can copy large files.

Just formatted my USB to ext4. Sadly exactly the same result.

 

[osprey]

Hello all. My PC is Linux mint 21.3 Virginia and Cinnamon 6.0.4.
Having problems copying large files. Have tried Copy and Paste in Nemo, mirror FreeFilesync, etc. Initially I thought that copying the file onto my USB was the problem - Nemo Starts at 155mb/sec then rapidly slows up to 75% and 20mb/sec. Then all stops and error code says 'Error splicing file: File is too large'. In FreeFilesync there is no error code but the mirror copy gets to 75% ish and then just locks up. Note. This is a good file and runs perfectly. Thoughts?

View attachment 20869

It's possible to split the file using the split command which will break it into sizes which you can determine. The file could be broken into sizes that you know the filesystem has no trouble with. Then reconstitute the file from the parts with the cat command.

 

[JohnJ]

It's possible to split the file using the split command which will break it into sizes which you can determine. The file could be broken into sizes that you know the filesystem has no trouble with. Then reconstitute the file from the parts with the cat command.

Whew. Split and cat. Sounds complicated and not sure how to do this. I am hoping for a simpler way. Cheers

 

[bob466]

Does the same problem occur when transferring the same file to an external HDD or SSD ?

I have an External 2TB HDD that came formatted to NTFS...I recently transferred my VM which is 50GB to it with no problems.

I have another External 2TB HDD which is formatted to EXT4...that I use for Timeshift snapshots which are about 158GB and no problems. My Ventoy 32GB Flash Drive is FAT32 which most are now and I can copy Mint Cinnamon 21.1 which is 2.9GB with no problems...maybe try FAT32.

 

[CaffeineAddict]

@JohnJ
How large is your USB?

Please post output of lsblk when your USB is attached.

 

[osprey]

Whew. Split and cat. Sounds complicated and not sure how to do this. I am hoping for a simpler way. Cheers

Here's a reasonable site that has the details if you'd like to have a go down this route:

Split Command in Linux: Usage Guide with Examples - Linux Dedicated Server Blog

Are you finding it challenging to split large files in Linux? You're not alone. Many users find themselves grappling with this task, but there's a tool that

ioflood.com

Having the same problem on both the usb and the ntfs filesystem could actually be an omen of an issue apart from the size of the file, but the split and cat method would test that out.

 

[dos2unix]

FAT32:
Maximum File Size: FAT32 supports files up to 4 GB in size.
Maximum Partition Size: The maximum partition size for FAT32 is 2 TB.
NTFS:
Maximum File Size: NTFS has theoretically huge file size limits, so you won’t run into issues.
Maximum Partition Size: NTFS can handle partitions of practically any size, making it suitable for large drives and system partitions.
exFAT:
Maximum File Size: exFAT can handle files up to 128 petabytes (that’s 128,000 terabytes!).
Maximum Partition Size: It supports partitions up to the same massive size of 128 petabytes

ext4:
Maximum File Size: Ext4 supports files up to 16 TiB.
Maximum Partition Size: It can handle partitions up to 1 EiB (exabyte).
Performance: While ext4 performs well with smaller file transfers, it lacks the parallel I/O capabilities of XFS.
Security: Ext4 is chosen for its advanced directory and file system security, including security labels for specific user permissions.
XFS:
Maximum File Size: XFS supports larger files and can handle files up to 8 EiB.
Maximum Partition Size: It can manage partitions up to 8 EiB as well.
Performance: XFS excels in large file input/output and parallel operations, making it ideal for scenarios like media file storage and databases.
Security: While not as security-focused as ext4, XFS still offers robust performance and scalability.

Edit: I found this for NTFS.
For NTFS, here are the limits:
Maximum Disk Size: NTFS supports up to 256 terabytes of disk space.
Maximum File Size: You can create files up to 256 terabytes in size.
Maximum Number of Files on Disk: NTFS allows 4,294,967,295 files on a single disk.
Maximum Number of Files in a Single Folder: You can have up to 4,294,967,295 files in a folder.

Let’s break down the sizes:

Megabyte (MB): 1 MB is equal to 1024 kilobytes (KB) or 1,048,576 bytes1.
Gigabyte (GB): 1 GB is 1024 megabytes (MB) or approximately 1 billion bytes.
Terabyte (TB): 1 TB is 1024 gigabytes (GB) or roughly 1 trillion bytes.
Petabyte (PB): 1 PB is 1024 terabytes (TB) or around 1 quadrillion bytes.
Exabyte (EB): Finally, 1 EB is equal to 1024 petabytes (PB) or an astonishing 1 quintillion bytes.

For cross platform Windows/Mac/Linux Fat32 and ExFat are the best options.
Windows doesn't see Ext4 and XFS drives by default, I am told there are some utilities that
will let you mount XFS and EXT4 drives in Windows, but so far I haven't gotten them to work.

 

[dos2unix]

The largest single platter hard drive I have seen on the market so far is...

Exos Mozaic 3+ | Seagate US

Discover Exos Mozaic 3+, where mass capacity, unprecedented areal density, and proven performance combine for storage that’s hyper-efficient and more sustainable than ever.

www.seagate.com

32TB for one drive.

 

[JohnJ]

Does the same problem occur when transferring the same file to an external HDD or SSD ?

I have an External 2TB HDD that came formatted to NTFS...I recently transferred my VM which is 50GB to it with no problems.

I have another External 2TB HDD which is formatted to EXT4...that I use for Timeshift snapshots which are about 158GB and no problems. My Ventoy 32GB Flash Drive is FAT32 which most are now and I can copy Mint Cinnamon 21.1 which is 2.9GB with no problems...maybe try FAT32. View attachment 20873

Hi Bob. Yes. Internal or external HDD and SSD. Have tried them all. Very strange. Some are NTFS, some are Ext4. my USB is 15gb in size and have tried Fat32 and formatted it to EXT4. No Probs up to 4gbish. Once I try 5gb then definitely grinds to a halt.

 

[JohnJ]

FAT32:
Maximum File Size: FAT32 supports files up to 4 GB in size.
Maximum Partition Size: The maximum partition size for FAT32 is 2 TB.
NTFS:
Maximum File Size: NTFS has theoretically huge file size limits, so you won’t run into issues.
Maximum Partition Size: NTFS can handle partitions of practically any size, making it suitable for large drives and system partitions.
exFAT:
Maximum File Size: exFAT can handle files up to 128 petabytes (that’s 128,000 terabytes!).
Maximum Partition Size: It supports partitions up to the same massive size of 128 petabytes

ext4:
Maximum File Size: Ext4 supports files up to 16 TiB.
Maximum Partition Size: It can handle partitions up to 1 EiB (exabyte).
Performance: While ext4 performs well with smaller file transfers, it lacks the parallel I/O capabilities of XFS.
Security: Ext4 is chosen for its advanced directory and file system security, including security labels for specific user permissions.
XFS:
Maximum File Size: XFS supports larger files and can handle files up to 8 EiB.
Maximum Partition Size: It can manage partitions up to 8 EiB as well.
Performance: XFS excels in large file input/output and parallel operations, making it ideal for scenarios like media file storage and databases.
Security: While not as security-focused as ext4, XFS still offers robust performance and scalability.

Edit: I found this for NTFS.
For NTFS, here are the limits:
Maximum Disk Size: NTFS supports up to 256 terabytes of disk space.
Maximum File Size: You can create files up to 256 terabytes in size.
Maximum Number of Files on Disk: NTFS allows 4,294,967,295 files on a single disk.
Maximum Number of Files in a Single Folder: You can have up to 4,294,967,295 files in a folder.

Let’s break down the sizes:

Megabyte (MB): 1 MB is equal to 1024 kilobytes (KB) or 1,048,576 bytes1.
Gigabyte (GB): 1 GB is 1024 megabytes (MB) or approximately 1 billion bytes.
Terabyte (TB): 1 TB is 1024 gigabytes (GB) or roughly 1 trillion bytes.
Petabyte (PB): 1 PB is 1024 terabytes (TB) or around 1 quadrillion bytes.
Exabyte (EB): Finally, 1 EB is equal to 1024 petabytes (PB) or an astonishing 1 quintillion bytes.

For cross platform Windows/Mac/Linux Fat32 and ExFat are the best options.
Windows doesn't see Ext4 and XFS drives by default, I am told there are some utilities that
will let you mount XFS and EXT4 drives in Windows, but so far I haven't gotten them to work.

Thanks Dos2. So it should work but...

 

[JohnJ]

Here's a reasonable site that has the details if you'd like to have a go down this route:

Split Command in Linux: Usage Guide with Examples - Linux Dedicated Server Blog

Are you finding it challenging to split large files in Linux? You're not alone. Many users find themselves grappling with this task, but there's a tool that

ioflood.com

Having the same problem on both the usb and the ntfs filesystem could actually be an omen of an issue apart from the size of the file, but the split and cat method would test that out.

Thanks Osprey. It's a clear explanation but I am reluctant to go down this path. Might have to if all else fails.

 

[JohnJ]

@JohnJ
How large is your USB?

Please post output of lsblk when your USB is attached.

Thanks Caffeine. USB is 15gb and have tried different formats - Fat32, Ext4. Umm lsblk? Do I past this into the terminal? Cheers

 

[CaffeineAddict]

Thanks Caffeine. USB is 15gb and have tried different formats - Fat32, Ext4. Umm lsblk? Do I past this into the terminal? Cheers

OK, yes it's run in the terminal.

in addition when your USB is attached share output of this command:

cat /etc/fstab

 

[JohnJ]

Hello All. I hope that all you helpful ones can see this post regarding my large file (5gb) transfer problem. I fired up my old Windows SSD and tried to transfer this particular 5gb file. Same problem but this time an error code came up saying that my 5gb file is too big to fit onto my 15gb USB. Ergo...the 5gb file is somehow corrupt. I reckon that this is the problem. Whew. Sorry for the confusion. I should have thought about this. i.e. how can only one 5gb file cause problems when no others even bigger ones transfer no problems. My old brain is quite a bit slower these days. Back to the moto - think of the simple things first and investigate them. Thanks everyone.