Backup Files & Databases from Linux Server to Amazon S3 Using S3cmd

Must read:

Ferdian Alfianto

Ferdian Alfianto

Ferdian Alfianto is an Internet enthusiast, Mac Lover; likes using Wordpress, experimenting with Linux (especially Debian and Ubuntu), tinkering with pfSense routers, happy experimenting with LEMP (Linux, Nginx, MariaDB, PHP) and Redis. You can contact me here.

Amazon S3 is a data storage service solution that is inexpensive but has top markotop features! (If Mr. Bondan says, “Mak Nyusss”!) What are the advantages of Amazon S3? Here are some of the features:

  1. High Scalable, regardless of size and whenever you need storage space, you can easily get it. Need only 100 MB? Monggo. Need 10 TerraByte? Monggo mawon ... :) Instantly you can upload your data regardless of size.
  2. Durability & Reliability data, Amazon S3 infrastructure is designed to serve mission-critical data, which must always be available, 100% uptime, and data must not be lost. For this reason, Amazon S3 spreads the files that are stored to several servers in multi datacenters. So that if any data is damaged / lost on one server, Amazon still has a copy on another server.
  3. Security, Amazon S3 supports several mechanisms that provide flexibility to its customers regarding who can access data, as well as how and where to access it. Amazon S3 also supports data transfer via SSL encryption using the HTTPS protocol.
  4. Fast, Amazon has 8 datacenters spread across America, Ireland, Singapore, Tokyo, and Brazil. We can choose whichever datacenter is closest to our server location, so that the process of uploading / downloading data from our server to Amazon servers and vice versa is faster.
  5. Cheap, the rental price for storage space on Amazon S3 is only $0.125 per gigabyte! yes, only around Rp. 1,200 per 1 GB.

With all the features above, of course Amazon S3 is the best solution for backing up our site files and databases. And the tools for doing this backup process are S3cmd.

S3cmd from S3tools project is a tool that uses the command line (command line) to upload, retrieve and manage data on Amazon S3. Process backup do not need to be done manually, but can be done automatically using cron, which we can set to execute the backup process every day, or hourly, or per week, and so on.

Backup Files

We will install S3cmd first. On the Debian distribution, s3cmd can be installed directly from repository, however, the S3cmd developer doesn't recommend it, because the Debian repo version of S3cmd has never been installed update. To install which version up-to-date, we have to grab it from the S3cmd developer repo, here are the steps:

Import first the signing key with the command:

wget -O- -q | sudo apt-key add -

Add the S3cmd developer repo to source.list with:

sudo wget -O / etc / apt / sources.list.d / s3tools.list

Install S3cmd with the command:

sudo apt-get update && sudo apt-get install s3cmd

Once S3cmd is installed, run the S3cmd configuration with the command:

s3cmd --configure

Then you will be asked to enter Access Key and Secret Key from an Amazon Web Service account. You can find both keys on the page Security Credentials.

Then S3cmd will ask "Encryption password". An encryption password is useful for protecting files in the transfer process to Amazon S3 from being read by others. Enter the encryption password you want.

Then you will be asked path to the GPG program, just hit enter.

And then the question arises whether you want to use the HTTPS protocol when doing the transfer process? With the HTTPS protocol, the transfer process will be much safer, but it will also be slower than using HTTP. And if your server connection uses a proxy, then you cannot choose to use the HTTPS protocol.
Type "yesAnd enter if using the HTTPS protocol, just press enter if not using.

Then came the question "HTTP Proxy server name". If your server connection uses a proxy, enter the IP address / hostname of the proxy. If not through a proxy, just hit enter.

After you have answered all the questions, S3cmd will offer an access test using the data you entered earlier. Press enter if you want to run the test, or type "n”And enter if you don't want to run the test.

Last question, "Save settings? [y / N]". Do you want to save the configuration you just entered? Type "y”And enter if you want to save, or just enter if you don't want to save.

After you enter the S3cmd configuration, we will execute the command to transfer files from our server to Amazon S3. There are three ways to use S3cmd to transfer files, namely by command s3cmd put, s3cmd pull and s3cmd sync.

I personally prefer to use commands s3cmd sync because the way it works is similar to rsync, where later only the files that we change / add / edit will be transferred to Amazon S3, while the files that we don't change / add / edit will not be transferred. The term in the computer world is incremental backupThe commands we use are:

s3cmd sync -r --delete-removed / path / to / file / s3: // bucket-name /

option --delete-removed commands s3cmd to delete files on Amazon S3 if they are not on our server. To / path / to / file / please replace it with path the location of the file you want backup. And to bucket-name is the name bucket which you have to create through your Amazon Web Service account.

If the execution goes well, then we will execute the command so that it runs automatically every day backing up files on our server to Amazon S3. Edit the crontab with the command:

sudo crontab -e

Then enter this line:

00 01-06 * * * s3cmd sync -r --delete-removed / path / to / file / s3: // bucket-name /

The command above is to order cron to run backup commands every day between 1:00 and 6:00 am in the morning.

Press Crtl-X and then Y to save the configuration above.

It's finished configuring the backup file from our server to Amazon S3 using S3cmd. Next we will back up the database to Amazon S3.

Database Backup

Create a script with the name in a directory, for example in the /home/ directory. Type this command:

sudo nano /home/

And enter the following script. Change lines 7 to 10 according to your database information. Replace root with database username, password with database password, bucketname with bucket name (you have to create another special bucket to store the database), and filename (it's up to you what filename you want).

#!/bin/sh # Updates etc at: # Under a MIT license # change these variables to what you need MYSQLROOT=root MYSQLPASS=password S3BUCKET =bucketname FILENAME=filename DATABASE='--all-databases' # the following line prefixes the backups with the defined directory. it must be blank or end with a / S3PATH=mysql_backup/ # when running via cron, the PATHs MIGHT be different. If you have a custom/manual MYSQL install, you should set this manually like MYSQLDUMPPATH=/usr/local/mysql/bin/ MYSQLDUMPPATH= #tmp path. TMP_PATH=~/ PERIOD=${1-day} echo "Selected period: $PERIOD." echo "Starting backing up the database to a file..." # dump all databases ${MYSQLDUMPPATH}mysqldump --quick --user=${MYSQLROOT} --password=${MYSQLPASS} ${DATABASE} > ${TMP_PATH }${FILENAME}.sql echo "Done backing up the database to a file." echo "Starting compression..." tar czf ${TMP_PATH}${FILENAME}.tar.gz ${TMP_PATH}${FILENAME}.sql echo "Done compressing the backup file." # we want at least two backups, two months, two weeks, and two days echo "Removing old backup (2 ${PERIOD}s ago)..." s3cmd del --recursive s3://${S3BUCKET}/$ {S3PATH}previous_${PERIOD}/ echo "Old backup removed." echo "Moving the backup from past $PERIOD to another folder..." s3cmd mv --recursive s3://${S3BUCKET}/${S3PATH}${PERIOD}/ s3://${S3BUCKET}/${S3PATH }previous_${PERIOD}/ echo "Past backup moved." # upload all databases echo "Uploading the new backup..." s3cmd put -f ${TMP_PATH}${FILENAME}.tar.gz s3://${S3BUCKET}/${S3PATH}${PERIOD}/ echo " New backup uploaded." echo "Removing the cache files..." # remove databases dump rm ${TMP_PATH}${FILENAME}.sql rm ${TMP_PATH}${FILENAME}.tar.gz echo "Files removed." echo "All done."

Then chmod the script with the command:

chmod 755 /home/

Then we will execute the database backup command so that it runs automatically every day, per week and per month. Type the command crontab -e and enter the following line:

# Backups daily 0 3 2-31 * 1-6 sh /home/ day # backups per week 0 3 2-31 * 0 sh /home/ week # Backups per month 0 3 1 * * sh /home/ month

Latest articles:

MongoDB logo

Easy to Install MongoDB on Ubuntu 20.04

This tutorial explains how to install and configure MongoDB Community Edition on Ubuntu 20.04. MongoDB is a free, open-source document database. Belongs to the so-called database family

Related article:

MongoDB logo

Easy to Install MongoDB on Ubuntu 20.04

This tutorial explains how to install and configure MongoDB Community Edition on Ubuntu 20.04. MongoDB is a free, open-source document database. Belongs to the so-called database family