Time Machine Backup to a Windows Share — Do it More Reliably

You can find plenty of instructions on the interwebs for setting up time machine to a network share, even a Windows share.

What you can't easily find is, how to do it reliably.

I have some recommendations.

1. Use cron scripts to just-in-time mount and dismount the time machine share

This is my first and biggest point. Dismounting the backup drive after each backup removes most of the reliability problem of network backup. Before doing this, I rarely got through 3 months without some kind of “the network share got dismounted uncleanly and now it won't mount until I run Disk First Aid on it”.

2. When it comes to data security, if you don't have 3 copies you aren't being serious.

This is a lesson you can take from cloud computing. Both Microsoft and Amazon clouds treat 'at least 3 copies' as the basic level for storing data. That means you want at least 2 independent backup systems for anything on your own machine.

If you combine this thought with the standard “don't put all your eggs in one physical location” motto of backup, you realise that you need a cloud or offsite backup as well as your time machine backup. The simplest free solution for your third copy, if 5GB is enough, is to use iDrive or OneDrive.

3. Buy a copy of Alsoft Disk Warrior

This is optional, and certainly less important than the first two points but, running Disk First Aid or fsck doesn't always work. Sad but true. I typically got a “fsck can't repair it properly” incident about once a year. I had a growing stack of hard disks with a year's worth of backup each, all only mountable readonly.
DiskWarrior has so far been reliable in restoring broken volumes back to fully working state. NB as of 2021 DiskWarrior can't yet repair APFS volumes so stay with HFS+ volumes for your time machines.

Help with cron scripts and multiple backups

Cron Scripts

Here are my script and cron table for mounting a TM drive from the network, requesting a backup, and dismounting the TM drive. It uses wakeonlan to wake the server from sleep, and ping to confirm it's up before trying to mount. It uses osascript to mount the volume because that deals with saving the network password in your keychain.

#! /usr/bin/env sh
#
smbServer=NameOfYourServer.local
smbServerMacAddress='00:00:00:00:00:00' # fill in server mac address to be able to wake it from sleep
smbVolumeUrl="smb://$(whoami)@$smbServer/D"
smbMaybeMountedAtPath1="/Volumes/$smbServer/Backups"
smbMaybeMountedAtPath2="/Volumes/D/Backups"
smbMaybeMountedAtMore(){
    dirname $(find /Volumes -iname $tmDiskImageName -maxdepth 3 2>/dev/null | head -n 1) 2>/dev/null
}
tmDiskImageName='TM3.21.sparsebundle'
tmVolumeMountedAtPath='/Volumes/TM3.21'

# wakeonlan
# I got mine from https://github.com/jpoliv/wakeonlan/blob/master/wakeonlan
# otherwise try https://ddg.gg/bash%20script%20wakeonlan
#

#cron jobs get a very truncated path and can't find ping, diskutil, hdiutil, tmutil ...
PATH="$PATH:/sbin:/usr/sbin:/usr/bin:/usr/local/bin"

echo '#-------------------------------------------------------'
date

echo "$0 [email protected]"
if [[ -z "$1" && -x $(which tmux) ]]
then
    echo "tmuxing ..."
    tmux new-session $0 tmuxed
    exit
elif [[ "$1" == *unmount* ]] ; then
    tmutil status
    if [[ ! -d "$tmVolumeMountedAtPath" ]] ; then 
        echo "$tmVolumeMountedAtPath is already unmounted"
    elif [[ $(tmutil status | grep 'Running = 0') ]] ; then 
        echo "unmounting ..."
        /usr/sbin/diskutil unmount $tmVolumeMountedAtPath
    else
        echo "not unmounting $tmVolumeMountedAtPath because tm status says still running."
    fi
    exit
fi

#-------------------------------------------------------
if [[ -x $(which wakeonlan) ]] ; then   wakeonlan $smbServerMacAddress ; fi
for tried in {1..50} ; do ping -c 1 -t 5 $smbServer 2>&1 && break ; done
if (( $tried == 50 )) ; then
    echo "failed to ping $smbServer. Exiting."
    exit
fi
sleep 10
if (( $tried > 2 )) ; then 
    echo "waiting in case it was a cold-ish start..."
    sleep 20
fi
echo 'mounting ...'
osascript -e 'mount volume "'$smbVolumeUrl'"'
echo 'attaching ...'
hdiutil attach $smbMaybeMountedAtPath1/$tmDiskImageName \
    || hdiutil attach $smbMaybeMountedAtPath2/$tmDiskImageName \
    || hdiutil attach $(smbMaybeMountedAtMore)/$tmDiskImageName \
    || echo \
       "Didn't find $tmDiskImageName at $smbMaybeMountedAtPath1 or $smbMaybeMountedAtPath2 or $(smbMaybeMountedAtMore)"
[ $? -eq 0 ] && echo "attached OK"
echo 'requesting backup.'
  tmutil startbackup --auto
  echo 'Done.'

And the crontab lines:

5  10-23/4 * * * /Users/chris/Applications/tmbackupnow.sh >> /Users/chris/Applications/Logs/tmbackupnow.log 2>&1
20,35 10-23/4 * * * /Users/chris/Applications/tmbackupnow.sh -unmount >> /Users/chris/Applications/Logs/tmbackupnow.log 2>&1

This schedule does 3 or 4 backups per working day on top of the local snapshots that time machine does anyway. Possibly this is overkill if you are squeezed for disk space. It tries to mount the TM machine and kick off a backup at 5 minutes past 10am,2pm,6pm,10pm and then tries to dismount the backup disk 15 minutes later and again another 15 minutes later. Adjust the timing to the size & speed of your backup.

At least 3 copies

My third copy, on top of time machine, is syncthing “continuous file synchronization”, which is great. It's like being able to set up a load of open source CloudDrives but using your local network too.

My fourth copy is either github or bitbucket for code; and iDrive or OneDrive for documents and graphics.

My fifth copy will be scripted backups to Azure storage, which seems like the cheapest way to do cloud backups. Meanwhile I'm paying Apple or Microsoft each month for big enough cloud storage.

Thoughts on Workplace Risk Assessments in 2021 – Proportionate vs Exponential Risk and Anecdotal vs Statistical Assessment

Proportionate vs Exponential Risk Assessment

  • Most human scale risk is proportionate. When 2 or 3 careless drivers go on the road, there is a risk of them killing 2 or 3 people.
  • Some risks, such as forest fires, are exponential, not proportionate. If 2 or 3 people drop a cigarette in a dry forest, the risk is not of 2 or 3 small fires endangering 2 or 3 people, the risk is that just one of those fires goes beyond control and endangers thousands of people.
  • Infectious disease is not a proportionate “1 careless person might endanger 1 person” type risk. It is a forest fire, exponential type risk in which one careless person can endanger thousands of people.

On Workplace Risk Assessment

  • Workplace risk assessment is typically done on an anecdotal, amateur, basis with no grasp of statistics. This is mostly ok because the risks are rarely fatal or lifelong; and workplace risks are typically proportionate “2 or 3 mistakes might hurt 2 or 3 people” type risks, not exponential “1 mistake might hurt thousands of people” type risks.
  • Covid is a forest fire, exponential type risk. A risk assessment for Covid that treats it as a proportionate risk is wrong and endangers thousands of unrelated people, not just the people in the workplace doing the risk assessment.
  • An anecdotal approach to risk assessment for Covid is negligent. If you have a risk assessment that relies on 'other people are doing it and the consequences haven't been terrible' then you have a risk assessment that is adequate for low, proportionate risk.
  • If you are tempted by 'but hundreds of people are doing it and the consequences haven't been terrible' then you are gambling, in the same way that someone who drops a cigarette in a dry forest in gambling.

Statistical Risk Assessment

  • A serious approach—if it is really necessary to do a workplace risk assessment for covid— would do the risk matrix using national empirical data and statistical computations for the “how likely” scale; and using “it could spark another outbreak, with widespread death” for the “how bad” scale; and updating weekly.
  • Because the “how bad” scale of risk goes up to “it could spark another outbreak and endanger thousands” you would need a calculation to show that the “how probable” scale is small enough to negate that.

An Simple Alternative Approach

  • An alternative approach to covid risk might be to say that when herd immunity is reached then we have done everything we can reasonably do and all that remains is to get back to life.
  • That seems to me the simple approach. Since the UK in mid-2021 is only a few months away from achieving that, I have no appetite to be impatient now and attempt a risk assessment that is likely to be wrong.

~ expert or well-informed comment more than welcome ~

The Smallest Agile Methodology That Could Possibly Work

People

You need 3 hats. That might mean 3 people, or 3 groups of people, or 1 or 2 people wearing more than 1 hat:

  • 1 person who knows what they want to have.
  • 1 person who knows about how to provide that.
  • 1 person with enough money to pay for it to be provided.

People Checklist

1) Do you have someone to wear each of the 3 hats?

A: I don't have someone who knows what they want.

  • Consider a research project instead, in which you try to work out what you want.
  • Otherwise Stop Now.

A: I don't have someone who knows how to provide what I want

  • Find help. Otherwise Stop Now.

A: I don't know whether we have enough money.

  • Do the next 2 steps, Requirements and Priorities, and then do the Budget step. If you cannot afford to do those steps, then you do not have enough money and you should Stop Now.

Requirements

  • The person who know what they want must express what they want in enough detail that the person who is going to provide it can understand what they are supposed to be providing.

This sounds very simple. It is not. Many many conflicts in personal life, disastrous work projects, and famous cockups in politics follow from the fact that helping person A to express what they want well enough so that person B can deliver it for them is a skill that takes time to learn.

Solution: hire a Business Analyst to help person A to express what they want and to express it as requirements that persons A and B can both agree they understand and are happy with

Requirements Checklist

1) Does anyone involved have previous experience of how easy it is is to trip up on getting someone to express what they want correctly and in enough detail for someone to deliver it?

A: No

  • Find someone. Pay for their help.
  • Or, budget for doing everything 2 to 3 times in order to get it right. (Yes, 2 to 3 times is about right for your first ever project. It is not a worst-case.)

Priorities

  • The person who knows what they want must be able to express what their priorities are amongst the things they want. Either based on what they want fastest, or on what they value most.

This sounds very simple. It is. The only two things that can possibly go wrong are:

  • person A changes their mind.
  • you have not broken the work into small enough chunks. A small enough chunk is a chunk that the person doing the work is confident they can can complete in under a week or so.

Priorities Checklist

1) Can the person who knows what they want prioritise what they want and break it down into small enough chunks so that the person who knows how to deliver can confidently say, of at least the top 2 items, that they can do them in less than a week or so each?

A: No

  • Find someone to help with doing this.
  • Alternatively, budget for 2 to 3 times the cost of the first item for working out as you go along how to do it. If you have already budgeted 2 to 3 times cost because of no experience with requirements, that gets you to 4 to 9 times the cost for the first item. Yes, really.

Delivery

  • The person who knows how to deliver (build, buy, borrow, invent, bodge, whatever) what the person who knows what they want wants, does so.
  • [If there is more work involved than they can do in a week or so, they first chunk the work, in strict priority order, into batches of, say, a week's work.]
  • After they have completed as little of the first piece of work that is still showable to the person who knows what they want, they show it. The person who knows what they want confirm or corrects the understanding of what is wanted. The person who is delivering corrects course.
  • After they have completed the whole of the fist piece of work they show that.
  • [Or, in the larger chunked case, after the first batch they show the person who knows what they want, what they can have so far.]
  • The person who knows what they want then says, “yes I'll take that” or “that looks great, keep going” or “please change … <insert detail here>.”
  • If it is not finished, keep going. If it takes a long time, work to a regular rhythm.

Budget

  • After you have requirements and priorities, you may wish to plan a budget. A budget is often counted in money or in person-days of work.

You may think that offering a price for a job is what any experienced tradesman should be able to do. You may, or may not, be right. There are 3 main risks:

  • Nobody involved in the project has enough experience to estimate a fair budget, but no-one wants to admit to not having enough experience.
  • The people estimating the budget are pressured into underestimating the costs
  • The job, for reasons not yet foreeen, will incur unexpected additional costs.

Budget Checklist

1) Make sure you have reviewed each of the 2 to 3 times multipliers for inexperience in Requirements and Prioritisation.

2) Do you have someone on the project with enough experience to estimate a budget?

A: Yes

  • Is that person the person doing the work?
    A: Yes: Then accept their min/max estimates for costs and their choice of a prudent budget
    A: No: Then have them review their min/max estimates for costs with the person doing the work, let the person doing the work provide detail, and let them come to a consensus.

A: No

  • Find someone to help.
  • If you are able and willing to risk the cost of 1 or 2 weeks work, consider simply starting the work and reviewing progress after 1 or 2 weeks. Then, try again to estimate a budget.

Review

  • Whatever you do, review how you are working after 1 to 2 weeks. Ask yourselves questions to help understand your satisfaction and dissatisfaction, for instance:
    • What is going well?
    • What am I happy with or not happy with?
    • What can we do better?
    • What would I like to change?
      Then, make a plan to do at least one of the things you decided would make things better.

Note you are not asking, “is the thing we are delivering good?” You are asking “is the way we are working good?” The two things are closely linked of course, both in the positives and the negatives.

Review Checklist

1) Did you, in fact, do a review after 1 to 2 weeks?
2) Did you, in fact, choose at least one thing to improve?
3) If you are now 4 weeks into a project have you, in fact, improved the things you intended to improve.

Risk

  • A risk is something that might either stop your project or else make you wish you had not started it. Make a list of them.
  • All of the checklists above represent potential risks. If you did not have a whole-hearted pass for each item of each checklist, list that as a risk.
  • Each time you plan a piece of work, consider the associated risks and how you can make the risk smaller. If you don't really know how to, find help.
  • For risks that you cannot make any smaller, are you able to accept the price of that risk happening? Are you clear on who has what share of the price?

Risk Checklist

1) Did you make a risk list?

A: No

  • Stop Now.

2) Do you have someone on the project with prior experience in managing risk?

A: No

  • That is a risk. Get help.

3) Do the things you haved planned to minimise risk really minimise risk or are they just things you can write down even though they might have no effect in minimising the risk?

Considerations

  • It is a fact of craft work that the right answer is often, “Get someone with the experience to do it”
  • Minimalism is of little use to a beginner. Rather, they first need a step-by-step, then later they can understand how to try minimalism.
    This is why complaints that “<insert agile methodology name here> is terrible” are largely futile. All well-known agile methods gives a simple enough way to start off and then require you to review and change how you work.

You’re Only As Good As Your Last Backup

… is a necessary rule of thumb for computer-based knowledge & design workers. But add the lesson of cloud computing:

Backups: If you don't have 3 copies, you aren't serious.

The standard redundancy for cheap cloud storage options is 3 copies. Anything less is reduced redundancy, sold at discount. You should have at least 2 backups, for instance both a home backup disk and a cloud drive or repo.

A big win, when you plan for multiple copies, is that you no longer need any of them to be highly reliable. What matters more is, how fast can you make another copy if one copy goes down?