(Failing to) Copy a Time Machine Backup to a Network Drive with asr

The Apple support page for copying a Time Machine backup disk doesn't cover the scenario when your new backup target disk is on the network. If you try to do it by hand using cp, rsync, ditto or other, you will likely fail with inscrutable errors.

Using asr may work, but failed for me after 1 ½ days, 500GB, possibly because I had some kind of network disconnected. To rely on a network being reliable for 3 days is to ignore the 8 fallacies of distributed computing, but if your TM backup is small enough this could work.

  1. Use Disk Utility -> File -> New Image -> Blank Image … to create a new sparsebundle disk image on your network drive. The arrowed options must be set correctly (well, you don't have to use sparse bundle but it is allegedly designed specifically for efficient use across a network):

2.

2. Mount the new disk image by double-clicking it, and also attach your existing Time Machine backup drive. Then, use  -> About This Mac -> System Report… -> Hardware/Storage and look in the column BSD Name to find the device names on which your Source and Target volumes are mounted:

3. Turn off Time Machine backup. Usually by unticking “Back Up Automatically” in the Time Machine preferences, if there is no On/Off switch.

4. Then, use asr on the command line to copy the device that hosted the volume to the device hosting the new volume. Use caffeinate at the same time to stop the computer sleeping instead of copying. In my case that was:

sudo caffeinate asr restore --source /dev/disk3 --target /dev/disk4s2 --erase --puppetstrings --allowfragmentedcatalog

I got this output, and after a few seconds had to type y to confirm:

XSTA    start   512 client
XSTA setup
Validating target…done
XSTA metadata
Validating source…done
Erase contents of /dev/disk4s2 (/Volumes/LaCie2019)? [ny]:

The --puppetstrings option means what most of us might call --progress although the output is quite limited.

Expect a speed of about 4 days per terabyte. I don't know why. Watching the Network tab in Activity Monitor I can see that data is rarely going faster than 5MB/s. Even writing to a spinning disk across a 20 year old 100Mbps network should go faster than that. I tried adding --buffers 10 --buffersize 100MB, but that still only got me to about 3 days per terabyte.

Anyway …

For me it failed. Sorry I lost the error message. So I went to to Finder drag-n-drop. The first time this failed after a day; the second it succeeded after 3 days. 🤷‍♂️

Use NSSM to install SyncThing as a Windows service

SyncThing does what OneDrive & Google Drive can do but under your control, across your machines, with more options, and without having to touch a 3rd party data snooping provider and without having to pay 3rd party Terabyte rates. I use it on my home network both to synchronise configuration across multiple machines and as an at-home backup solution. It's fast, simple, well-maintained and it works.

NSSM is “the Non-Sucking Service Manager” which has a simple GUI to set up commandline programs like SyncThing as a Windows Service.

Install SyncThing

To use SyncThing as a Service, avoid the GUI options such as SyncTrayzor and go for the GitHub download. Choose a directory to install to, such as your Program Files directory.

SyncTrayzor is great for your working machine, where you only need SyncThing to run when you are logged in. For a server which is hosting backups and redundant copies of your files, you want a Windows service running whenever the machine is up.

Install NSSM

NSSM also has no installer as of early 2020. Download & extract to a Program Files directory.

I then added New-Alias nssm "C:\Program Files\nssm-2\win64\nssm.exe" to my PowerShell profile

Launch NSSM

nssm without parameters will show you the commands you can use. The simplest is to use install & edit to get the GUI:
To show service installation GUI: nssm install [<servicename>]
To show service editing GUI: nssm edit <servicename>

So use:

nssm install SyncThing

And then fill in the boxes by finding the path where you installed SyncThing. I only edited the first three tabs: Application, Details, and Log On. The rest can stay as default.

What about the Parameters? See the SyncThing Docs. This is mine:

-no-console -no-browser -no-restart -gui-address=localhost:8384

-no-console -no-browser are because services run headless.
-no-restart because the Windows Service infrastructure has options for handling restarts.
-gui-address=localhost:8384 to make the gui console only available on localhost, not across the network. You may not want this.

You can now use nssm to start/stop/monitor services, not just the ones you have installed with it.

nssm start SyncThing
nssm status SyncThing

Or, you can use the standard Windows Services gui.

Where is the config?

Nssm just edits the Windows service config, which is visible in the Local Services app, which you can launch from Task Manager -> Services

SyncThing keeps config in the place noted in SyncThing Docs unless you add e.g. -home=D:\MyPath to the startup parameters

Where is the SyncThing Gui?

If you followed my example and used -gui-address=localhost:8384 then open that address in your browser and read all about at https://docs.syncthing.net/intro/gui.html

More Options?

See https://docs.syncthing.net/users/syncthing.html

Yes but I want to manage it across my home network?

  1. Change the startup options to use -gui-address=0.0.0.0:8384.
  2. Add the full path to SyncThing.exe as a firewall exception in your Windows firewall.
  3. Restart the service

This will make the browser interface accessible across the network. Then:

  1. Open the the GUI at localhost:8384.
  2. Open the Settings (under the Actions menu, top right).
  3. Open the GUI panel.
    1. Choose HTTPS
    2. Add a username and password. NB I think these are both case sensitive.

Use a Windows NTFS, or Linux, or other–formatted disk for Apple Time Machine Backups

It is well known that you have to use an Apple formatted disk (HFS+) for Time Machine Backups and you can't use an NTFS formatted disk, or any of the preferred *nix disk formats with Time Machine.

Except you can.

Gotchas And Caveats

It's not always a first choice option:

  • If your Windows drive is connected over USB2, it will be much slower than a Mac disk connected over Thunderbolt or FireWire. My first backup of 120GB took about 10 hours, aka all night. But that was still better than no backup whilst waiting for a new drive to arrive in the post.
  • The backup volume will not auto-mount when you plug the drive in. You must manually double-click the <YourMachineName>.sparsebundle in the Finder each time you plug the drive in. After that, Time Machine backups will run as normal.

Instructions

  • Plug in your foreign-formatted disk and get a read/write driver for it
  • Save the the script from tmMakeImage script in, for instance, your Downloads directory.
  • Open a terminal window.
  • If you saved to Downloads, then something like this should make the script executable and show you command line usage:
    cd ~/Downloads
    chmod a+x tmMakeImage
    mv tmMakeImage /usr/local/bin
    tmMakeImage
  • So if your external drive has a Volume called WinDrive, and your Mac drive is about 250GB then this command should initialise it for Time Machine backups:
    tmMakeImage 500GB /Volumes/WinDrive GO
  • Optionally, follow up with an immediate tmutil startbackup

Opening the Time Machine preferences should now show that you have "Time Machine Backups" selected as your backup drive. Don't forget to double-click the sparsebundle each time you attach the drive, to mount your Backup Volume and allow Time Machine to do its stuff.

Background

Using Sparsebundles to create HFS+ formatted drive on a 'foreign' disk format

Apple created, it seems, sparse images and sparse bundles to solve the problem of saving backups on a network drive. Time Machine uses and recognises them. It will even auto-mount the sparsebundle disk image when you re-attach the drive in order to start running a backup.

Local drives and network drives

Most instructions on the web focus on network drives. Good instructions without any command-line stuff is here: http://www.hciguy.com/2010/06/16/using-time-machine-to-backup-your-mac-to-an-ntfs-drive-over-the-network-running-win-7/ For network drives, the sparsebundle name includes your Mac's MAC address which it doesn't need to for a locally-connected drive.

Things Under the Hood of Time Machine Disks

  • You can't create a sparsebundle directly on a network or 'foreign' drive so the typical thing is to create it on your machine first, and then copy it to its final destination.
  • Sparsebundles don't use up empty space. An sparsebundle declared as 500GB but still empty will only take up a few megabytes of real disk space.
  • A sparsebundle used by Time Machine has a com.apple.TimeMachine.MachineID.plist file in the package, which contains the UUID of the physical machine it belongs to. This stops you accidentally using backups on the 'wrong' machine. (Time Machine does let you browse and use 'wrong' backups though).
  • A Time Machine drive must have "Ignore ownership permissions" Off, whereas by default the Mac mounts external drives with "Ignore ownership permissions" set to On. This setting is not stored on the drive itself; it's stored on your machine and can be specified each time the drive is mounted (man hdiutil) or specified permanently (man vsdbutil)
  • You can set the Time Machine destination to a sparsebundle on an attached drive by first mounting it then doing tmutil setdestination /Volumes/Volumenameonceitsbeenmounted. You mount the image either by double-clicking it in Finder, or with hdiutil attach /path/to/image.
  • Learn More:
    man tmutil
    man hdiutil
    man vsdbutil
    
  • You can still use your backup disks for other files alongside your Time Machine backups. Just don't touch the Backups.backupdb folder! The Finder will protect it to some extent; it doesn't let you modify things in there, but it will let you delete and add things.
  • The Script

    Most of the things in the tmMakeImage script can be found on the net going back to 2007, but you'll still find recent answers on the internet saying it can't be done.