Backup with rsync and git – keepconf

4 09 2015

For one side, rsync is a widely-used utility to keep copies of a file on two computer systems the same. It is commonly found on Unix-like systems and functions as both a file synchronization and file transfer program.

For the other side, git is a distributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows.

Combining both utilities produce a good backup solution, but it need a bit of programming to adapt the functionalities for the specific job purpose. Keepconf cover this case and wrap rsync and git with an easy to use layer that makes this task simple and fast.

For a quick view of how keepconf works, it doesn’t need much more than this configuration file for backup and track all /etc/ and /home/ dirs:

[hosts]
server1.my.domain
[files]
/etc/*
/home/*

Installation

There are two ways for install it, one is a deb package and the other is a bash script. Both are in the project website https://github.com/rfrail3/keepconf:

+ Deb install, look for the package into:
https://github.com/rfrail3/keepconf/tree/master/deb-package

+ Script install, get it from:
https://raw.githubusercontent.com/rfrail3/keepconf/master/keepconf-install.sh

Once you can execute “keepconf -V” and see the version number, things go well and you can continue.

Now, its a requirement that the backup server can connect to the servers to backup with a passwordless ssh connection, using ssh-keys. Configure it for the user that will be used for the backup, and test it, when “ssh user@host” can connect without any password prompt, you can continue. In the repo there are a detailed explanation of how to do this step:
https://github.com/rfrail3/keepconf/blob/master/keepconf-hands-on.txt

Configure

There are one small configuration file for define keepconf behavior, located at “/etc/keepconf/keepconf.cfg”. Although keepconf needs all the variables defined there, basically you only need to know:

# Directory with the hosts configuration files
d_conf = /etc/keepconf/hosts/
# Directory for store remote files
d_dest = /var/keepconf/hosts/
# Default global user for connect to other hosts
#rsync_user = backup
# Define type of git repository
#repo = False
repo="local"
#repo="ssh://user@host/rute/to/repo"

As you can see, the default user for connect to other hosts is “backup”, change it if you need. At the same, the default repo is local, keepconf will create it, but you can change to a remote existing git repostiory.

Remove the example conf files that are located where d_conf variable define, you can read it, but for this guide, aren’t needed:
# rm -rf /etc/keepconf/hosts/*

Write the configuration file for the server to backup into the d_conf directory. This files have to end in “.cfg”, for this example use the name “my_host.cfg”. Define there the hostname and the directories that you want:

[hosts]
server1.my.domain
[files]
/etc/*
/home/*
/var/www/*

Initialize the local repo, if is a remote repo, keepconf will fetch it:
# keepconf -i
Initialized empty Git repository in /var/keepconf/hosts/.git

Its time to run it, (In this case, with “-s” for reduce output):

# keepconf -s
- Fetching start
- Processing: /etc/keepconf/hosts/my_host.cfg
- Fetching: server1.my.domain
- Finding files in /etc/keepconf/post-get.d/
- Processing: /etc/keepconf/post-get.d/01-remove-binary
- Fetching done
- Committing start
- Committing into local repository
[master (-commit) 1824ecd] Keepconf commit at 09:53:09 - Friday/September/2015 Hosts ok: [1] server1.my.domain Hosts bad: [0]
Committer: backup_user
- Committing done
-
- Hosts rsync'ed: [1] server1.my.domain
- Hosts not rsync'ed: [0]
- Fetched in: 1.32 seconds
- Committed in: 0.55 seconds
- Completed in: 1.87 seconds

The files are yet fetchet where “d_dest” define, in this case “/var/keepconf/hosts/” and tracked with git inside it as can you see:
# cd /var/keepconf/hosts/ && git log
commit 1824ecd5d640bd21c69b634f1cbadcc75a4f17ee
Author: root
Date: Fri Sep 4 09:53:09 2015 +0200
Keepconf commit at 09:53:09 - Friday/September/2015
Hosts ok: [1] server1.my.domain
Hosts bad: [0]

If the installation was made using the bash script, don’t forguet to add a cron entry. If not, there are a file in “/etc/cront.d/keepconf” that automatically execute it each day.

Thats all!

If you need more, take a look at the documents that the project provide, hands-on and the manual:
https://github.com/rfrail3/keepconf/blob/master/keepconf-hands-on.txt
https://github.com/rfrail3/keepconf/blob/master/keepconf-manual.txt

There are some plugings for munin available for graph the output status:

Keepconf time

Keepconf

More Info:
https://en.wikipedia.org/wiki/Git_(software)
https://en.wikipedia.org/wiki/Rsync
https://github.com/rfrail3/keepconf/

Advertisements

Actions

Information

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: