Discussion:
Time Machine thinning
(too old to reply)
Stefano Mori
2013-03-29 14:36:40 UTC
Permalink
backupd's log shows "thinning..." messages, but they seem to appear very quickly in succession,

far faster than I'd imagine an

rm -fR timestamped_dir

could manage.

How does Time Machine thin/delete directory trees so fast? Is there some special magic??


Stefano
LuKreme
2013-03-30 00:25:28 UTC
Permalink
Post by Stefano Mori
backupd's log shows "thinning..." messages, but they seem to appear very quickly in succession,
far faster than I'd imagine an
rm -fR timestamped_dir
could manage.
How does Time Machine thin/delete directory trees so fast? Is there some special magic??
Thin != delete. Thinning is removing the entries from the database, as I recall.
--
'But look,' said Ponder, 'the graveyards are full of people who rushed
in bravely but unwisely.' 'Ook.' 'What did he say?' said the Bursar. 'I
think he said, "Sooner or later the graveyards are full of everybody".'
Stefano Mori
2013-03-30 12:58:43 UTC
Permalink
Post by LuKreme
Thin != delete. Thinning is removing the entries from the database, as I recall.
Ah!

Reason I ask is I do backups with rsync's --link-dest option, which I like, but there's the problem of deleting the oldest timestamps, as it takes a while...

Any way to bend the laws of physics?


Stefano
Arno Hautala
2013-03-30 18:23:39 UTC
Permalink
Post by LuKreme
Thin != delete. Thinning is removing the entries from the database, as I recall.
In this case, thin does equal delete.
Mar 30 13:20:50 shindig com.apple.backupd[54602]: Starting post-backup thinning
Mar 30 13:23:01 shindig com.apple.backupd[54602]: Deleted /Volumes/Backup of shindig/Backups.backupdb/shindig/2013-02-28-002627 (156.7 MB)
Mar 30 13:23:39 shindig com.apple.backupd[54602]: Deleted /Volumes/Backup of shindig/Backups.backupdb/shindig/2013-03-28-233100 (68.5 MB)
Mar 30 13:24:11 shindig com.apple.backupd[54602]: Deleted /Volumes/Backup of shindig/Backups.backupdb/shindig/2013-03-28-223255 (64.1 MB)
Mar 30 13:24:56 shindig com.apple.backupd[54602]: Deleted /Volumes/Backup of shindig/Backups.backupdb/shindig/2013-03-28-221436 (69.4 MB)
Mar 30 13:25:36 shindig com.apple.backupd[54602]: Deleted /Volumes/Backup of shindig/Backups.backupdb/shindig/2013-03-28-204147 (29 MB)
Mar 30 13:25:36 shindig com.apple.backupd[54602]: Post-back up thinning complete: 5 expired backups removed
Mar 30 13:25:44 shindig com.apple.backupd[54602]: Backup completed successfully.
Further, TimeMachine does need to delete the actual files to free up
more space. My guess is that it's quick because many of the items to
delete will be hard linked directories, which should dramatically
bring down the number of files to delete. rsync can only hard link
files, so there's going to be much more to remove.
Reason I ask is I do backups with rsync's --link-dest option, which I like, but there's the problem of deleting the oldest timestamps, as it takes a while...
Any way to bend the laws of physics?
My email backup (from offlineimap) contains several tens of thousands
of files and really slowed down TimeMachine (every backup had to check
every file to determine which were new in the directory). So I
switched to rsnapshot, which handled the many files much faster, but
still wasn't exactly speedy. Now I just rsync directly to the backup
location (running on FreeNAS) and rely on ZFS snapshots to quickly
make and prune old backups.

Apple really should have just executed on their ZFS project.

--
arno s hautala /-| ***@alum.wpi.edu

pgp b2c9d448
Stefano Mori
2013-03-31 10:18:50 UTC
Permalink
Post by Arno Hautala
Further, TimeMachine does need to delete the actual files to free up
more space. My guess is that it's quick because many of the items to
delete will be hard linked directories, which should dramatically
bring down the number of files to delete. rsync can only hard link
files, so there's going to be much more to remove.
I wondered about the hard linked directories, but couldn't imagine how that got past the need to unlink thousands of files. Is there some interesting algorithm in there?
Post by Arno Hautala
Now I just rsync directly to the backup
location (running on FreeNAS) and rely on ZFS snapshots to quickly
make and prune old backups.
Oh, cool.

Should I look at MacZFS?


Stefano
Arno Hautala
2013-03-31 14:13:02 UTC
Permalink
Post by Stefano Mori
I wondered about the hard linked directories, but couldn't imagine how that got past the need to unlink thousands of files. Is there some interesting algorithm in there?
By hard linking a directory, there isn't a need to link any of the
files contained in that directory. Think about your Applications
folder, which probably isn't changing every day or even week, let
alone every hour. TM can hard link /Applications and that one link
essentially covers all of the files and folders within it. When
pruning that backup it also just needs to delete one link instead of
visiting every file and folder.

Hand linking directories can be dangerous though (what if you make
A/B/C -> A), so it's a feature that isn't exposed to user tools.
Post by Stefano Mori
Post by Arno Hautala
Now I just rsync directly to the backup
location (running on FreeNAS) and rely on ZFS snapshots to quickly
make and prune old backups.
Oh, cool.
Should I look at MacZFS?
If you're not working with a lot of data? Probably. ZFS is notorious
for being RAM hungry. I think the recommendation is no fewer than 4 GB
(not entirely unreasonable these days I guess), but you very quickly
will want more. At a stretch you can probably get away with 1GB per
TB, but I'm pretty sure that wouldn't be a great experience.

If you have a spare drive to play with, why not? I think Zevo is the
most current ZFS version, though the free version does have some
potentially annoying restrictions (it's probably fine for new users
and playing around).

--
arno s hautala /-| ***@alum.wpi.edu

pgp b2c9d448

Continue reading on narkive:
Loading...