Allocation Unit Size USD/HDD and HDD Partitioning
Sgt-Jackson
Posted messages
129
Status
Membre
-
Sgt-Jackson Posted messages 129 Status Membre -
Sgt-Jackson Posted messages 129 Status Membre -
Hello,
I have read a bunch of pages online discussing allocation unit size and the three main file systems, but I still have some questions that remain unanswered. Maybe you can help me?
For example, I have a 32 GB USB stick and I have put about 2,000 lightweight files (.txt, Word, images, MP3...), none exceeding 500,000 bytes. The most space-consuming files are ISO files (around 14 GB in total). The whole thing takes up 19.8 GB, but the properties of the USB show an occupied size of 24.4 GB. From what I've read, this is normal and comes from the allocation unit size.
What I don’t understand is that in the formatting window of the USB, it is formatted as exFAT and the allocation unit size is 32 kilobytes. If I understand correctly, 32 KB (which is 32,000 bytes?) is the maximum size of a single file that I can place in a cluster on the drive, so on the USB? The issue is that my ISO files range from 1.5 GB to 5 GB (which is millions of bytes). Am I wrong (most likely yes) and thus misunderstood something, or did I forget something?
----
The second question I have is: can I partition a 1 TB external hard drive and apply an allocation unit size suitable for small files (less than 4 GB) on the first partition and another suited for larger files (over 4 GB)? This method would allow me not to lose too much space when I format the hard drive to transfer files larger than 4 GB?
Thank you for reading and for your help.
Configuration: Windows / Chrome 73.0.3683.103
I have read a bunch of pages online discussing allocation unit size and the three main file systems, but I still have some questions that remain unanswered. Maybe you can help me?
For example, I have a 32 GB USB stick and I have put about 2,000 lightweight files (.txt, Word, images, MP3...), none exceeding 500,000 bytes. The most space-consuming files are ISO files (around 14 GB in total). The whole thing takes up 19.8 GB, but the properties of the USB show an occupied size of 24.4 GB. From what I've read, this is normal and comes from the allocation unit size.
What I don’t understand is that in the formatting window of the USB, it is formatted as exFAT and the allocation unit size is 32 kilobytes. If I understand correctly, 32 KB (which is 32,000 bytes?) is the maximum size of a single file that I can place in a cluster on the drive, so on the USB? The issue is that my ISO files range from 1.5 GB to 5 GB (which is millions of bytes). Am I wrong (most likely yes) and thus misunderstood something, or did I forget something?
----
The second question I have is: can I partition a 1 TB external hard drive and apply an allocation unit size suitable for small files (less than 4 GB) on the first partition and another suited for larger files (over 4 GB)? This method would allow me not to lose too much space when I format the hard drive to transfer files larger than 4 GB?
Thank you for reading and for your help.
Configuration: Windows / Chrome 73.0.3683.103
2 réponses
Hello @Sergeant Jackson,
'just a little clarification
A little clarification to start.
In computing, the multiplier for measurement units is 1024 (2 to the power of 10).
So, while a 1-liter bottle of fruit juice indeed contains 1000 milliliters, a USB drive of 1 GB actually contains 1024 MB. I'll close this parenthesis.
Come closer to see more clearly
Let's be rough: a hard drive (non-SSD) is divided:
- into concentric circles = tracks
- into pie slices = clusters
At the intersection of tracks and clusters, we find = sectors
Those very sectors that you are trying to adjust as accurately as possible.
Each sector has a maximum size that allows it to hold a maximum amount of information.
The fruit juice theory or how to understand the use of a hard drive
Let's revisit the previously mentioned fruit juice.
Here, the sectors will be glasses, and the file will be the fruit juice.
The bottle is full and contains 1500 ml of good juice... Organic, of course!
The glasses can hold 330 ml each.
Let's try to pour (store) our juice (file) into our glasses (sectors).
We fill the 1st glass with 330 ml => 1170 ml remain
We fill the 2nd glass with 330 ml => 840 ml remain
We fill the 3rd glass with 330 ml => 510 ml remain
We fill the 4th glass with 330 ml => 180 ml remain
We fill the 5th glass with the rest of the bottle, 180 ml => 0 remain
The bottle is empty.
So the glasses must be full, right?
Well, no!
Only glasses 1 to 4 are filled with 330 ml of juice.
Glass 5 is filled with only 180 ml of juice out of 330 ml of available space.
Result:
> the fruit juice (file) is 1500 ml in size
> the occupied glasses (sectors) represent a total volume of 5x330 ml = 1650 ml
The storage units used are therefore larger than the file that occupies them.
And in this case:
> 1650 - 1500 = 150 ml of empty space in a sector that can no longer be used.
Why?
Because if you pour chicken juice into the remaining space, what do you get in the sector?
A horrible mix of fruit juice/chicken juice that your stomach won't tolerate.
Theory vs. Practice
It is theoretically possible to optimize the sector sizes to better adapt them to the types of files you want to store.
The calculation would involve, for example, calculating the total sum of the bytes of the files, then dividing it by the number of files to get an average size in bytes per file.
This average size would be the size assigned to the sectors.
But since it's an average... many sectors would still be partially used.
I’ll let you imagine how much time this would require for each transfer!
Probably longer than the transfer itself since you'd also have to factor in the time for formatting the disk or drive.
In your case
Your 2000 files total barely 1 GB due to their maximum size of less than 500 KB. To say the least, they don't represent much compared to the 32 GB storage capacity of your drive.
Even large files won't generate more partially used sectors.
As for the difference between the size of your files (19.8 GB) and the space used on your drive (24.4 GB), it doesn't just come from incomplete sectors. Let's not forget that a drive is never fully available for file storage. A drive also contains other stored information like the File Allocation Table. A sort of invisible tree structure of files, so that Windows knows where to find what on the sectors.
So what do we do now?
The file system in Windows is such that incomplete sectors are and will always be generated by successive reads/writes on a medium. Just like fragmentation, which is its corollary.
The standard size applied to keys and disks commercially sold pre-formatted is generally 512 KB. And it seems to me that in a domestic use, that's an effective size across the board.
You would need a particularly specific (professional?) activity to regularly store a large number of files under 500 KB.
It would also need to be economically significant since a mid-range 32 GB (USB 3) drive costs less than 20 euros today.
I would therefore advise you not to worry about reallocating a specific size to the sectors of your drive(s). The efficiency gained would be minimal.
And the time spent managing all this would be proportionately immense!
To know regarding small files
Due to fragmentation on one hand, and the need to keep the addresses of the sectors in the File Allocation Table updated on the other hand, the read/write time for small files is less than for large files.
What I mean is that writing 1024 files of 1 KB takes longer than writing 1 file of 1024 KB.
In conclusion
Whether regarding your storage needs on a drive or a disk, you should be aware of where managing these media will lead you.
Formatting, partitioning, allocating sectors... all operations that can lead to:
- increased usage of the medium, hence accelerated wear
- data loss in case of disorganized operations
- time-consuming management
Especially since the least mechanical disk today is 1 TB for about 60 euros. And you need to put in some effort to fill a 1 TB external drive in a year. Especially with 500 KB files!
My analysis and viewpoint apply to your questions regarding the drive as well as those about your hard disk. By the way, aside from the fact that I’ve never thought of doing a double allocation of sectors on the same partitioned disk, I have serious doubts about the feasibility of that.
Why not ask Santa Claus for an SSD disk for December?
That's it! Tchô
'just a little clarification
A little clarification to start.
In computing, the multiplier for measurement units is 1024 (2 to the power of 10).
So, while a 1-liter bottle of fruit juice indeed contains 1000 milliliters, a USB drive of 1 GB actually contains 1024 MB. I'll close this parenthesis.
Come closer to see more clearly
Let's be rough: a hard drive (non-SSD) is divided:
- into concentric circles = tracks
- into pie slices = clusters
At the intersection of tracks and clusters, we find = sectors
Those very sectors that you are trying to adjust as accurately as possible.
Each sector has a maximum size that allows it to hold a maximum amount of information.
The fruit juice theory or how to understand the use of a hard drive
Let's revisit the previously mentioned fruit juice.
Here, the sectors will be glasses, and the file will be the fruit juice.
The bottle is full and contains 1500 ml of good juice... Organic, of course!
The glasses can hold 330 ml each.
Let's try to pour (store) our juice (file) into our glasses (sectors).
We fill the 1st glass with 330 ml => 1170 ml remain
We fill the 2nd glass with 330 ml => 840 ml remain
We fill the 3rd glass with 330 ml => 510 ml remain
We fill the 4th glass with 330 ml => 180 ml remain
We fill the 5th glass with the rest of the bottle, 180 ml => 0 remain
The bottle is empty.
So the glasses must be full, right?
Well, no!
Only glasses 1 to 4 are filled with 330 ml of juice.
Glass 5 is filled with only 180 ml of juice out of 330 ml of available space.
Result:
> the fruit juice (file) is 1500 ml in size
> the occupied glasses (sectors) represent a total volume of 5x330 ml = 1650 ml
The storage units used are therefore larger than the file that occupies them.
And in this case:
> 1650 - 1500 = 150 ml of empty space in a sector that can no longer be used.
Why?
Because if you pour chicken juice into the remaining space, what do you get in the sector?
A horrible mix of fruit juice/chicken juice that your stomach won't tolerate.
Theory vs. Practice
It is theoretically possible to optimize the sector sizes to better adapt them to the types of files you want to store.
The calculation would involve, for example, calculating the total sum of the bytes of the files, then dividing it by the number of files to get an average size in bytes per file.
This average size would be the size assigned to the sectors.
But since it's an average... many sectors would still be partially used.
I’ll let you imagine how much time this would require for each transfer!
Probably longer than the transfer itself since you'd also have to factor in the time for formatting the disk or drive.
In your case
Your 2000 files total barely 1 GB due to their maximum size of less than 500 KB. To say the least, they don't represent much compared to the 32 GB storage capacity of your drive.
Even large files won't generate more partially used sectors.
As for the difference between the size of your files (19.8 GB) and the space used on your drive (24.4 GB), it doesn't just come from incomplete sectors. Let's not forget that a drive is never fully available for file storage. A drive also contains other stored information like the File Allocation Table. A sort of invisible tree structure of files, so that Windows knows where to find what on the sectors.
So what do we do now?
The file system in Windows is such that incomplete sectors are and will always be generated by successive reads/writes on a medium. Just like fragmentation, which is its corollary.
The standard size applied to keys and disks commercially sold pre-formatted is generally 512 KB. And it seems to me that in a domestic use, that's an effective size across the board.
You would need a particularly specific (professional?) activity to regularly store a large number of files under 500 KB.
It would also need to be economically significant since a mid-range 32 GB (USB 3) drive costs less than 20 euros today.
I would therefore advise you not to worry about reallocating a specific size to the sectors of your drive(s). The efficiency gained would be minimal.
And the time spent managing all this would be proportionately immense!
To know regarding small files
Due to fragmentation on one hand, and the need to keep the addresses of the sectors in the File Allocation Table updated on the other hand, the read/write time for small files is less than for large files.
What I mean is that writing 1024 files of 1 KB takes longer than writing 1 file of 1024 KB.
In conclusion
Whether regarding your storage needs on a drive or a disk, you should be aware of where managing these media will lead you.
Formatting, partitioning, allocating sectors... all operations that can lead to:
- increased usage of the medium, hence accelerated wear
- data loss in case of disorganized operations
- time-consuming management
Especially since the least mechanical disk today is 1 TB for about 60 euros. And you need to put in some effort to fill a 1 TB external drive in a year. Especially with 500 KB files!
My analysis and viewpoint apply to your questions regarding the drive as well as those about your hard disk. By the way, aside from the fact that I’ve never thought of doing a double allocation of sectors on the same partitioned disk, I have serious doubts about the feasibility of that.
Why not ask Santa Claus for an SSD disk for December?
That's it! Tchô
Hello
The allocation unit size defines the minimum size that will be used.
Each cluster will be 32 KB.
If you create a file of 1 byte, the apparent size will be 32 KB.
Yes, this size is defined by partition.
You need to find a proper balance between this size, the size of the files to copy, and the size of the partition.
But you are worrying for nothing, in my opinion.
Most of the time, the default size works just fine.
--
There are 3 kinds of people: those who can count and those who cannot.
There are two kinds of people: those who divide everything into two categories and the others.
What I didn't understand is that in the formatting window of the drive, it is set to exFAT and the allocation unit size is 32 Kilobytes. If I understood correctly, 32 KB (or 32,000 bytes?) is the maximum size of a single file that I can place in a cluster of the drive, so on the USB stick? The problem is that my ISO files range from 1.5 GB to 5 GB (so millions of bytes). Am I wrong (logically, yes) and therefore misunderstood, or have I forgotten something?
The allocation unit size defines the minimum size that will be used.
Each cluster will be 32 KB.
If you create a file of 1 byte, the apparent size will be 32 KB.
The second question I have is: can I partition a 1 TB external hard drive and apply an allocation unit size suitable for small files (less than 4 GB) on the first partition and another suitable for larger files (greater than 4 GB)? This method would allow me not to waste too much space when formatting the hard drive to transfer files larger than 4 GB?
Yes, this size is defined by partition.
You need to find a proper balance between this size, the size of the files to copy, and the size of the partition.
But you are worrying for nothing, in my opinion.
Most of the time, the default size works just fine.
--
There are 3 kinds of people: those who can count and those who cannot.
There are two kinds of people: those who divide everything into two categories and the others.
Thank you very much for your detailed explanation, I see things more clearly now. I will follow your advice.
Thanks again!