Rcat (Multipart backup utility for Rclone)

Post Reply
User avatar
sorcerykid
Member
Posts: 1841
Joined: Fri Aug 26, 2016 15:36
GitHub: sorcerykid
In-game: Nemo
Location: Illinois, USA

Rcat (Multipart backup utility for Rclone)

by sorcerykid » Post

Rcat v1.0

The rcat utility generates a multi-volume tar archive and uploads each volume to cloud storage via rclone. The entire process is automated, and therefore is perfectly suitable for a cronjob or as part of any scheduled backup routine.

Since each volume is only retained for the duration of the file transfer, it greatly streamlines the process of backing up and restoring large amounts of data (like an entire filesystem) when free hard drive space would otherwise be insufficient.

While there are a couple third-party tools with functionality similar to rcat, they are coded in Python and are much more sophisticated by comparison:
I prefer the simplest possible implementation when it comes to backing up important data, particularly since customization might be necessary. Hence, I decided a shell script in combination with the multi-volume feature of tar was best suited for this purpose.

The command syntax for a backup operation is as follows:
  • ~/rcat/backup.csh CHUNK_SIZE LOCAL_PATH CLOUD_PATH/CHUNK_NAME.tar
The required script parameters are
  • CHUNK_SIZE - the maximum file size for each volume of the tar archive, specified as a whole number in kilobytes, or optionally in megabytes or gigabytes by suffixing "M" or "G" respectively; sizes are base-2, thus 1M = 1024 kilobytes and 1G = 1024 megabytes
  • LOCAL_PATH - the directory or file to backup; absolute paths will be stripped of the leading slash as is the default behavior of tar
  • CLOUD_PATH - the location where rclone will store the tar archive, typically to the cloud
  • CHUNK_NAME - the base filename for each volume of the tar archive, appended with the ".tar" extension; the resulting files will be named as "CHUNK_NAME.partX.tar", where "X" is the respective volume number.
Note: Be sure to edit the CACHE_PATH in "backup.csh" to point to a temporary directory with sufficient disk space for each volume (including adequate permissions). Once the backup operation completes, the temporary files will be removed automatically.

Here are some example command invocations for backing up a directory and a file:
  • ~/rcat/backup.csh 1G /tmp/worlds/jt2 remote:weeklies/tmp_worlds_jt2.tar
    ~/rcat/backup.csh 100M auth.txt remote:dailies/minetest/worlds/jt2/auth.txt.tar
The command syntax for a restore operation is as follows:
  • ~/rcat/restore.csh CLOUD_PATH/CHUNK_NAME.tar
The required script parameters are
  • CLOUD_PATH - the location where rclone will retrieve the tar archive, typically from the cloud
  • CHUNK_NAME - the base filename for each volume of the tar archive, appended with the ".tar" extension. The files must be named as "CHUNK_NAME.partX.tar," where "X" is the respective volume number.
Note: Be sure to edit the CACHE_PATH in "restore.csh" to point to a temporary directory with sufficient disk space for each volume (including adequate permissions). Once the restore operation completes, the temporary files will be removed automatically.

Here are the corresponding command invocations for restoring a directory and a file:
  • ~/rcat/restore.csh remote:weeklies/pub_worlds_jt2.tar
    ~/rcat/restore.csh remote:dailies/minetest/worlds/jt2/auth.txt.tar
Keep in mind that directories and files will always be restored relative to the current working directory as illustrated below:
  • Backup: /tmp/worlds/jt2 -> Restore: ./tmp/worlds/jt2
  • Backup: ./auth.txt -> Restore: ./auth.txt
Note: You can adapt rcat for use with rsync or any other file-transfer utility by simply editing "transfer.csh" accordingly.

Repository:

https://bitbucket.org/sorcerykid/rcat

Download Archive (.zip)
Download Archive (.tar.gz)

Dependencies:

Rclone can be downloaded from https://rclone.org/downloads/

Installation Instructions:
  • Unzip the archive into your home directory.
  • Rename the rcat-master directory to "rcat".
  • Edit the CACHE_PATH setting in "backup.csh" accordingly.
  • Edit the CACHE_PATH setting in "restore.csh" accordingly.
Source Code License:

The MIT License

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests