Use --fastlist to decrease Backblaze costs

I want to have an emergency copy of my data somewhere. I trust my NAS and my skills, but a NAS can get stolen or burned in a fire. Especially for my business I really need to have those backups.

My NAS runs TrueNAS which has Cloud Sync functionality. I use it to run a daily push to Backblaze, so at worst I will lose a day of changes. Then I also run a monthly sync to keep clean up deleted files. I would prefer to sync zfs snapshots, but alas, so far I've only found rsync.net which would double the costs compared to Backblaze.

Rclone

I also have a VM running Proxmox Backup Server that stores its data on an iSCSI drive hosted on TrueNAS. I don't want to push this "drive" to the cloud every day, so I run Rclone inside the VM to sync the changes every day. TrueNAS also uses Rclone to copy files to the cloud and Backblaze supports Rclone out of the box.

Costs

What I didn't know when I started using Backblaze was that they not only charge for storage (per GB) but also per API call. And more importantly that Rclone makes an API call b2_list_file_names for every directory.

This means that in the case of for example my Proxmox Backups, a lot of calls were made, which ended up costing me (almost) more than the storage. I asked Backblaze support, but they didn't really know what I could do. They just confirmed my hunch: if you enable the --fastlist flag (option) in Rclone, the costs are significantly reduced.

To be fair, this can be found in the Backblaze documentation, although it doesn't mention anything about reducing costs and you have to link it yourself to the way API calls are billed.

• Fastlist: "By default, Rclone lists the contents of a Backblaze B2 bucket one directory at a time (one directory per API call). But, with the --fast-list option, it retrieves a listing of all of the files in the bucket, consuming just one API call per 1,000 files. The --fast-list option uses more memory, but it consumes fewer API transactions. Therefore, Backblaze strongly recommends that you use this option with every Rclone command."
• Billing info for listing file names: "Please note: billing is per 1000 files/versions returned. For example, if 10,000 file names are requested and returned, you will be billed for 10 API calls."

Results

As an example compare my bills for July (without --fastlist enabled) and August (with --fastlist enabled). My costs almosts halved.