In Defense of Over-Engineering

  • Pass it on: Make it easier to hand off to someone else. This particularly applies to administrative scripts. Comment the code. Add help/usage. Make it easy for someone new to use without them having to decipher some arcane idiomatic syntax that they may not be familiar with. “What flavor of regex does this script use anyway?”
  • Make it observable: If you can’t monitor it, you can’t own it and it will bite you in the ass. That backup script may have worked great when you first deployed it, but if you can’t consistently monitor execution one day you may find that it’s pining for the fjords and in fact hasn’t been successful in weeks. Invariably you discover this only when you really need that backup. (I know, I know. You must do restore tests on a regular basis. This is only an example.)
  • Fire and forget: When my solution is deployed, I don’t want to look at it again. Once the solution is fully tested and stable, I don’t want to have to mess with it again until we change something. For infrastructure, that might be changing the footprint by adding a new application cluster. For scripts it might be software upgrades forcing changes to the scripts. Whatever it is, I want to go months without looking at it again.

Pass It On

# Let's save some money!
# Shut down the system when the backup is complete.
# Let sleep scheduler start me back up to backup again tomorrow.
shutdown -h now

Make It Observable

echo "$(date +"$date_format") INFO Starting backup script"

Fire and Forget

# Halt on errors, even in functions
set -eE
# Things to do in the event of an error
fail() {
for file in $(ls $backup_dir/inconsistencies*); do
s3_move $file "inconsistency_reports"
done
start_neo4j
echo "$(date +"$date_format") ERROR Backup script failed!"
}
# If something errors, run fail before exiting
trap "fail" ERR
  • Wait for Neo4j to sync up and become available
  • Stop Neo4j in preparation for an offline backup
  • Run a consistency check
  • Run a backup
  • Move backup to AWS S3 for safekeeping

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store