==================================================================<

** Original area : "/grc/techtalk"
** Original message from : 0x6c976bb3@lookup.openpgp.key.invalid (Andrew Skretvedt)
** Original message to :
** Original date/time : 25 May 26, 02:43 >==================================================================<

Last month, there was a nice thread started by Peabody asking a question
about disk metadata.

Message-ID: <10ru7ci$k8m$1@GRC>
https://grc.com/groups/techtalk:300610

Tonight, I just ran across a pretty awesome looking space-usage analysis
tool for folks running one of the more advanced filesystems with
features like Copy-on-Write reflinks, snapshots, embedded data
compression, and the like.

These advanced features tend to make the actual storage required much
less than the apparent size of a file set.

This tends to make the process of accounting for space usage on such a filesystem a little harder and more abstract.

The tool is 'btdu', for Linux. It sports a TUI for terminal awesomeness.
It also has a straight cli-mode.

https://github.com/CyberShadow/btdu

It's most unique property is that it is a /sampling/ disk usage
analyzer. Rather than starting at the root directory or some other
directory and recursively enumerating all the objects therein to
generate space usage stats, it treats the filesystem rather like a dart
board and starts throwing darts into the on-disk address space. As they explain in their readme, for each dart location, they ask the filesystem
what file(s) point to the data there; then they collect metadata about
those files.

Like sampling a population, sampling a filesystem in this way builds an estimate of the usage state of the disk, which improves in its "margin
of error" rapidly as the number of samples builds. They say that as
little as 100 samples can estimate the space usage of a filesystem to
~1%. So you can get an idea of what's hogging all your space /very/ quickly.

(if you think about it, if you have space hogging files, one of those
initial 100 darts are likely to land on a block connected with one of
them, since their "target" size is bigger; their data is more likely to
be the earliest accounted for in a sampling run)

Typically, the sampling proceeds until you stop it. So the estimate
starts inaccurate, improves to reasonable in a short time, and after
this gradually converges toward an /exact/ report once /every/ block of
the filesystem has been sampled.

If you run Linux and use BTRFS filesystems especially, then this is
worth a look.

(I haven't studied-up yet enough to know if it can also work with ZFS,
XFS, and other filesystems with similar advanced features. Nor if you
could use it on BSD or macOS.)

My Mint 21.3 and 22.2 systems don't have this tool in their default
package manager; the tool seems relatively new (still showing version
0.x.y). So, I intend to setup the required build dependencies (written
in D) and build a copy from source.

I'll report back with some experiences once I have a binary to try out!


--
OpenPGP 0xC6901B2A6C976BB3 (https://keys.openpgp.org)



--- OpenXP 5.0.64
* Origin: (618:400/23.10)