Bacula: The Open Source Commercial Backup Solution

Download: Bacula.pdf

Other Languages: Polish, Spanish

Contents

  1. Install Bacula
  2. Configure Bacula
    1. Network Binding
    2. Director
    3. Storage Daemon
    4. File Deamon
    5. Console
    6. Troubleshooting Connection
      1. ip settings
  3. Manage bacula
    1. Pools, volumes, lables
    2. bconsole
    3. using bacula
      1. show filesets
      2. status dir
      3. status client
      4. status storage
      5. list jobs
      6. list volumes
      7. run
      8. stop/delete/cancell jobs
      9. prune/purge/delete
    4. restore
      1. restore all
      2. restore select
      3. messages
      4. restore file from x date
    5. Backups
      1. Incremental vs Differential
      2. VMware Images
    6. Repair Mysql Tables
    7. Volume Management
      1. Limiting the volume size
  4. Adding Clients
    1. Linux
    2. Windows
  5. Performance
    1. Initial Setup
    2. Increase Windows speed
    3. Hardware RAID5 vs Software RAID5
    4. Disk speed
      1. Disk Usage
      2. Example
  6. Backing up Databases
    1. Mysql
  7. Offsite Storage
    1. Amazon S3
      1. Amazon S3cmd
      2. rsync and Encryption (gpg)
  8. Bacula and encryption
  9. References

Keywords: Bacula, s3cmd, backup, incremental, differential, full, tape, commercial grade, autochanger, debian, ubuntu, disaster recovery plan, gpg, amazon s3, offsite storage

Install Bacula

aptitude install bacula
aptitude install bacula-director
aptitude install bacula-sd
aptitude install bacula-fd

aptitude install bacula-director-mysql
aptitude install mysql-admin

Configure Bacula

The best way to learn is to print below conf files and read them while you reading this manual.

  1. Director bacula-dir.conf controls what will get run, when it will get run and what client will you be backing up from.
  2. Storage Daemon bacula-sd.conf controls which director can talk to it, where and what device will it store the files. Options: HDD, File, DVD, Autochanger, DLT,DDS, DDS4, Onstream, DDS3, Exbyte, NAS

  3. FileDaemon bacula-fd.conf needs to be installed on all clinets that you want to backup. This program controls getting the files from client system and sending it to director. Configuration controls which director can use it.

  4. All bacula configuration files are in /etc/bacula 

/etc/bacula/
|-- bacula-dir.conf
|-- bacula-dir.conf.dist
|-- bacula-fd.conf
|-- bacula-sd.conf
|-- bconsole.conf

Network Binding

 #DirAddress = 127.0.0.1
or
 #SDAddress = 127.0.0.1
or
 #FDAddress = 127.0.0.1

Director

Brief overview:

  1. Basic unit is a Job ( one job, one client, one schedule, one storage, one pool)
  2. Name – Unique name
  3. Type – What to do: backup, Backup, Migrate, Admin, Restore
  4. Level – Backup level type: Full, Differential, Incremental

  5. FileSet – What files to backup. Name of the FileSet{}

  6. Client – Where to get the files (machine name). Name of the Client{}
  7. Storage – Where to put the files (which hardware). Name of the Storage{}: File, DDS-4, 8mmDrive, DVD, etc.
  8. Pool – Which set of Volumes (tapes, disk) to use. Name of the pool{}
  9. Schedule – When to do it. Name of the Schedule{}

You will need to change the following:

  1. Passsword -You can change the password for dir,sd,fd,console. Make sure it matches in all files.

  2. Under the section FileSet { Name = "Full Set"... change the files you want to back up. Example File=/home/myusername/

  3. Under the section Job { Name = "RestoreFiles" .... change the Where to point to where you want to store restor your files. Default Where=/tmp/bacula-restore

  4. Under the section Catalog { Name = MyCatalog dbname = bacula; user = "bacula"; password = "secretpassword".... make sure it has the correct information. If the you don't have database created use the script supplied to install it. mysql -u bacula -p < /usr/share/bacula-director/make_mysql_tables

Now Start bacula, see if it starts, check the error log to make sure you didn't miss anything. 

/etc/init.d/bacula-director start
or
/etc/init.d/bacula-director restart
cat /var/log/bacula/log

If you get an error in the log, fix it in configuration.

Storage Daemon

Make sure the following changes are done:

  1. All your password match to director configuration
  2. If you use networking that Address=127.0.0.1 is commented out.

  3. Under the section Device { Name = FileStorage Media Type = File Archive Device ... change the Archive Device to where you will store your files. If you are like me and have software RAID5 setup on 4 500GB drives mounted to a /home directory totaling 1.3 TB available space, you can set it to Archive Device = /home/bacula/backups

  4. Here is the place to set other devices. Look at the configuration file for examples on how to set: DDS-4, OnStrem, DVD-Writer, Exbyte 8mm,

  5. Make sure the bacula has file permissions to write to the folder. 

mkdir -p /home/bacula/backups
chown -R bacula:bacula /home/bacula

Start the bacula storage daemon 

/etc/init.d/bacula-sd start
or
/etc/init.d/bacula-sd restart
cat /var/logs/bacula/logs

If you get permission, login/password error in the log, fix it in configuration file.

When you are done with the setup and everything wokrs make sure you read Volume Management Section. We need to setup multiple volumes for our backup so that the rotation happens.

File Deamon

Make sure the following changes are done:

  1. If you use networking that Address=127.0.0.1 is commented out.
  2. Make sure passwords match what directors conf file.

  3. If you can not resolv the hostname of the computer make sure you add the appropriate computer names to /etc/hosts file.

Start the bacula file deamon 

/etc/init.d/bacula-fd start
or
/etc/init.d/bacula-fd restart
cat /var/logs/bacula/logs

If you get permission, login/password error in the log, fix it in configuration file.

Console

Here is a list:

  1. bconsole
  2. bacula-console-gnome
  3. bacula-console-wx
  4. bacula-console-qt

The conf files are located in /etc/bacula/ make sure the console has the password as it is defined in director. 

sudo bconsole
Connecting to Director servername:9101
1000 OK: servername-dir Version: 1.38.11 (28 June 2006)
Enter a period to cancel a command.
*exit

Here is a visual representation on how the config files are connected: http://man.chinaunix.net/network/bacula/bacula_manual/Conf-Diagram.png

Troubleshooting Connection

ip settings

telnet server1 9102
Trying 192.168.1.68...
Connected to server1.local.
Escape character is '^]'.
quit

telnet 127.0.0.1 9102   (localhost)
telnet 192.168.1.123 9102  (local ip address)
telnet servername1 9102 (local servername)

#DirAddress = 127.0.0.1
#SDAddress = 127.0.0.1
#FDAddress = 127.0.0.1

ping servername1

vi /etc/hosts

add this line (replace ip address with yours) 

192.168.1.123 servername1
192.168.1.234 servernamefd

Manage bacula

/etc/init.d/bacula-dir start
/etc/init.d/bacula-sd start
/etc/init.d/bacula-fd start

/etc/init.d/bacula-dir restart
/etc/init.d/bacula-sd restart
/etc/init.d/bacula-fd restart

Pools, volumes, lables

bconsole

Start bconsole and type in help: 

bconsole
help

  Command    Description
  =======    ===========
  add        add media to a pool
  autodisplay autodisplay [on|off] -- console messages
  automount  automount [on|off] -- after label
  cancel     cancel [<jobid=nnn> | <job=name>] -- cancel a job
  create     create DB Pool from resource
  delete     delete [pool=<pool-name> | media volume=<volume-name>]
  disable    disable <job=name> -- disable a job
  enable     enable <job=name> -- enable a job
  estimate   performs FileSet estimate, listing gives full listing
  exit       exit = quit
  gui        gui [on|off] -- non-interactive gui mode
  help       print this command
  list       list [pools | jobs | jobtotals | media <pool=pool-name> | files <jobid=nn>]; from catalog
  label      label a tape
  llist      full or long list like list command
  messages   messages
  mount      mount <storage-name>
  prune      prune expired records from catalog
  purge      purge records from catalog
  python     python control commands
  quit       quit
  query      query catalog
  restore    restore files
  relabel    relabel a tape
  release    release <storage-name>
  reload     reload conf file
  run        run <job-name>
  status     status [storage | client]=<name>
  setdebug   sets debug level
  setip      sets new client address -- if authorized
  show       show (resource records) [jobs | pools | ... | all]
  sqlquery   use SQL to query catalog
  time       print current time
  trace      turn on/off trace to file
  unmount    unmount <storage-name>
  umount     umount <storage-name> for old-time Unix guys
  update     update Volume, Pool or slots
  use        use catalog xxx
  var        does variable expansion
  version    print Director version
  wait       wait until no jobs are running [<jobname=name> | <jobid=nnn> | <ujobid=complete_name>]

When at a prompt, entering a period cancels the command.

using bacula

show filesets

show filesets

FileSet: name=Full Set
      O M
      N
      I /home/myusername/
      I /etc/
      N
      E /proc
      E /tmp
      E /.journal
      E /.fsck
      N

status dir

status dir

And you will see: 

Level          Type     Pri  Scheduled          Name               Volume
===================================================================================
Incremental    Backup    10  01-Aug-08 23:05    server1           bacula20080801
Incremental    Backup    10  01-Aug-08 23:05    server2           bacula20080801
Full           Backup    11  01-Aug-08 23:10    BackupCatalog      bacula20080801
====

Running Jobs:
Console connected at 01-Aug-08 12:20
No Jobs running.
====

Terminated Jobs:
 JobId  Level     Files      Bytes     Status   Finished        Name
========================================================================
     1  Full     11,651  5,808,650,275 OK       01-Aug-08 11:01 server1
     3  Full     63,930  2,651,212,530 OK       01-Aug-08 12:14 server2

status client

Lets get a status on the client 

status client

Pick the client and you will see his status, jobs run etc. 

status client
The defined Client resources are:
     1: server1
     2: server2
Select Client (File daemon) resource (1-2): 2
Connecting to Client server2-fd at server2:9102

server2-fd Version: 1.38.11 (28 June 2006)  i486-pc-linux-gnu debian 4.0
Daemon started 01-Aug-08 12:01, 1 Job run since started.

Terminated Jobs:
 JobId  Level     Files         Bytes  Status   Finished        Name
======================================================================
     3  Full     63,930  2,651,212,530 OK       01-Aug-08 12:14 server2
====
Running Jobs:
Director connected at: 01-Aug-08 12:53
No Jobs running.

status storage

Lets find out what is the status on our storage 

status storage

You should see: 

Automatically selected Storage: File
Connecting to Storage daemon File at server1:9103

server1-sd Version: 1.38.11 (28 June 2006) x86_64-pc-linux-gnu debian 4.0
Daemon started 01-Aug-08 09:19, 3 Jobs run since started.

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level   Files          Bytes Status   Finished        Name
======================================================================
     1  Full     11,651  5,810,545,052 OK       01-Aug-08 11:01 server1
     2  Full     63,930  2,660,294,477 OK       01-Aug-08 12:14 server2
====

Device status:
Device "FileStorage" (/home/bacula/backups) is not open or does not exist.
====

In Use Volume status:
====

list jobs

list jobs

      1 | server2      | 2008-08-01 10:57:43 | B    | F     |    11,651 |  5,808,650,275 | T         |
|     2 | server2      | 2008-08-01 11:42:43 | B    | F     |         0 |              0 | R         |
|     3 | server3      | 2008-08-01 12:05:10 | B    | F     |    63,930 |  2,651,212,530 | T         |
|     4 | server2      | 2008-08-01 13:05:39 | B    | I     |        12 |        125,049 | T         |
|     5 | server1      | 2008-08-01 13:53:08 | B    | F     |     5,900 |    127,019,988 | T         |
|     6 | server1      | 2008-08-01 13:53:39 | B    | F     |     5,900 |    127,019,988 | T         |
|     7 | server2      | 2008-08-01 23:05:04 | B    | I     |        17 |        128,228 | T         |
|     8 | server3      | 2008-08-01 23:05:14 | B    | I     |       888 |     49,182,024 | T         |
|     9 | server1      | 2008-08-01 23:05:45 | B    | I     |         2 |         15,433 | T         |

list volumes

list volumes

Pool: Default
+---------+-------------+-----------+----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
| MediaId | VolumeName  | VolStatus | VolBytes       | VolFiles | VolRetention | Recycle | Slot | InChanger | MediaType | LastWritten         |
+---------+-------------+-----------+----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
|       3 | Volumes0001 | Full      | 26,843,504,883 |        6 |    2,678,400 |       1 |    0 |         0 | File      | 2008-12-08 19:49:05 |
|       4 | Volumes0002 | Full      | 26,843,504,948 |        6 |    2,678,400 |       1 |    0 |         0 | File      | 2008-12-09 03:25:59 |
|       5 | Volumes0003 | Append    | 24,754,608,236 |        5 |    2,678,400 |       1 |    0 |         0 | File      | 2008-12-09 14:18:28 |
+---------+-------------+-----------+----------------+----------+--------------+---------+------+-----------+-----------+---------------------+

run

To run a job type in: 

run
Using default Catalog name=MyCatalog DB=bacula
A job name must be specified.
The defined Job resources are:
     1: server1
     2: BackupCatalog
     3: RestoreFiles
Select Job resource (1-3):

If you select 1 you will be asked: 

Run Backup job
JobName:  Client1
FileSet:  Full Set
Level:    Incremental
Client:   rufus-fd
Storage:  File
Pool:     Default
When:     2003-04-28 14:18:57
OK to run? (yes/mod/no):

stop/delete/cancell jobs

If there is need to stop a job from running you can do the following:

  1. log into bconsole

  2. Issue list jobs to see which one is running R.

  3. Issue cancell and tell it which jobid to cancel cancel jobid=59, you can also use jobname cancel job=myjobname

  4. If you want a job to start again you can do: status dir, get a list of currently scheduled jobs, then for each job execute run yes job=$host-backup.

In the last resort where you need to cancel jobs NOW because soemthing is going wrong, run 

/etc/init.d/bacula-dir stop
/etc/init.d/bacula-dir start
or 
/etc/init.d/bacula-dir restart

prune/purge/delete

list volumes

update

Options are: 

purge files jobid=<jobid>|job=<job-name>|client=<client-name>

purge jobs client=<client-name> (of all jobs)

purge volume|volume=<vol-name> (of all jobs) 

purge volume=myoldvolumename

delete volume=myoldvolumename

cd /home/bacula/backup/
rm myoldvolumename

restore

restore all

To restore run 

restore all

You should see: 

First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.

To select the JobIds, you have the following choices:
     1: List last 20 Jobs run
     2: List Jobs where a given File is saved
     3: Enter list of comma separated JobIds to select
     4: Enter SQL list command
     5: Select the most recent backup for a client
     6: Select backup for a client before a specified time
     7: Enter a list of files to restore
     8: Enter a list of files to restore before a specified time
     9: Find the JobIds of the most recent backup for a client
    10: Find the JobIds for a backup for a client before a specified time
    11: Enter a list of directories to restore for found JobIds
    12: Cancel
Select item:  (1-12):

done
Bootstrap records written to /var/lib/bacula/server1-dir.1.restore.bsr

The job will require the following Volumes:

   bacula20080801


63930 files selected to be restored.

Run Restore job
JobName:    RestoreFiles
Bootstrap:  /var/lib/bacula/server1-dir.1.restore.bsr
Where:      /home/bacula/restore
Replace:    always
FileSet:    Full Set
Client:     server1-fd
Storage:    File
When:       2008-08-01 13:13:32
Catalog:    MyCatalog
Priority:   10
OK to run? (yes/mod/no):

restore select

restore select

cd /home/lucas/
mark myimportantfolder
exit

messages

messages
or
autodisplay on

restore file from x date

list jobs

restore select jobid=1242

To view the rest of the console commands see Bacula Console

Backups

Incremental vs Differential

full (complete dumps)
differential (files changed since last full backup)
incremental (changed files since the last backup of any sort) backups

VMware Images

Repair Mysql Tables

mysql -u bacula -p 

use bacula;

Check tables: 

check table BaseFiles;
check table CDImages;
check table Client;
check table Counters;
check table Device;
check table File;
check table FileSet;
check table Filename;
check table Job;
check table JobMedia;
check table Media;
check table MediaType;
check table Path;
check table Pool;
check table Status;
check table Storage;
check table UnsavedFiles;
check table Version;

Repair Tables: 

repair table BaseFiles;
repair table CDImages;
repair table Client;
repair table Counters;
repair table Device;
repair table File;
repair table FileSet;
repair table Filename;
repair table Job;
repair table JobMedia;
repair table Media;
repair table MediaType;
repair table Path;
repair table Pool;
repair table Status;
repair table Storage;
repair table UnsavedFiles;
repair table Version;      

mysqlcheck --repair --all-databases -p
or
mysqlcheck -u bacula -p --auto-repair=1 bacula

Volume Management

Limiting the volume size

Limit your volume size to about 10%-15% of the HD capacity. IE, 1TB drive, volume size 100GB. And then set your max volume on the pool to ((HD space/volume space) - 1) so you don't need to worry about a full HD.

Pool {
  Name = Default
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 31 days         # one year
  Accept Any Volume = yes             # write on any volume in the pool
  Maximum Volumes = 20
  Maximum Volume Bytes = 25G    #25 GB
  Label Format = Volumes                #(should I pick a different name/ format)

Label Format = "${Pool}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}h${Minute:p/2/0/r}m"

  LabelMedia = yes;                   # lets Bacula label unlabeled media

User defined maximum volume capacity 26,843,545,600 exceeded on device "FileStorage" (/home/bacula/backups
End of medium on Volume "Volumes0001" Bytes=26,843,504,883 Blocks=416,105 at 08-Dec-2008 19:49.
Created new Volume "Volumes0002" in catalog.
Labeled new Volume "Volumes0002" on device "FileStorage" (/home/bacula/backups).
Wrote label to prelabeled Volume "Volumes0002" on device "FileStorage" (/home/bacula/backups)
New volume "Volumes0002" mounted on device "FileStorage" (/home/bacula/backups) at 08-Dec-2008 19:49.

Adding Clients

Linux

#---------Clients -------------
#Setup client to backup part1
Client {
  Name = server2-fd
  Address = server2
  FDPort = 9102
  Catalog = MyCatalog
  Password = "mypassword"         # password for FileDaemon 2
  File Retention = 30 days            # 30 days
  Job Retention = 6 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}

#Second Job for client 2 part 2
Job {
  Name = "server2"
  Client = server2-fd
  FileSet = "Full Set server2"
  JobDefs = "DefaultJob"
  Write Bootstrap = "/var/lib/bacula/server2.bsr"
}

#Fileset for clinet2 part 3
# List of files to be backed up
FileSet {
  Name = "Full Set server2"
  Include {
    Options {
      signature = MD5
    }
#
#  Put your list of files here, preceded by 'File =', one per line
#    or include an external list with:
#
#
    #File = /home/jgoerzen/work/bacula-1.38.11/debian/tmp-build-sqlite
    File = /usr/local/pythonenv
    File = /usr/local/turbogears
    File = /usr/local/src/
    File = /var/www/
  }
#
# If you backup the root directory, the following two excluded
#   files can be useful
#
  Exclude {
    File = /proc
    File = /tmp
    File = /.journal
    File = /.fsck
  }
}

Windows

#
# Default  Bacula File Daemon Configuration file
#
#  For Bacula release 1.38.10 (08 June 2006) -- cygwin 1.5.18(0.132/4/2)
#
# There is not much to change here except perhaps the
# File daemon Name to
#

#
# List Directors who are permitted to contact this File daemon
#
Director {
  Name = server1-dir
  Password = "mypassword"
}

#
# "Global" File daemon configuration specifications
#
FileDaemon {                          # this is me
  Name = windowsserver2-fd
  FDport = 9102                  # where we listen for the director
  WorkingDirectory = "c:/bacula/working"
  Pid Directory = "c:/bacula/working"
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = server1-dir = all, !skipped
}

Performance

Initial Setup

Days = ~20
Bacula Volume = 360GB

Biggest Full Backup:
FD Bytes Written: 63,668,167,347 (63.66 GB)
SD Bytes Written: 63,983,912,475 (63.98 GB)
Rate: 771.2 KB/s
Software Compression: 82.3 %

Others have expressed that they use bacula for: "46 clients, 12400 jobs, and 4.7 million files" and more.

Its been suggested that three settings that can have an impact on performance: VSS, compression and Maximum Network Buffer Size.

Increase Windows speed

Option 1

Option 2 

The primary TCP tuning parameters appear in the registry under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
On the Edit menu, point to New, and then click DWORD Value.
        Type GlobalMaxTcpWindowSize in the New Value box, and then press Enter
        Click Modify on the Edit menu.
        Type the desired window size in the Value data box.

Note. The valid range for window size is 0-0x3FFFC000 Hexadecimal. 

System Key: [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters]
Value Name: GlobalMaxTcpWindowSize
Data Type: REG_DWORD (DWORD Value)
Value Data: 0–0x3FFFFFFF

Set the windows size to: 65536

Hardware RAID5 vs Software RAID5

Disk speed

Disk Usage

iostat -d -x 5

Press ctrl + C to stop 

ctrl + c

Pay special attention to await and util. 

Linux 2.6.18-53.1.4.el5 (your.servername.com)   12/17/2009

Linux 2.6.26-2-amd64 (uicesv10)         02/18/2010      _x86_64_

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.02     7.79    0.29    1.90    18.62    83.85    46.91     0.01    4.79   0.80   0.17
sda1              0.00     0.00    0.00    0.00     0.01     0.00    63.53     0.00    2.84   2.34   0.00
sda2              0.02     7.79    0.29    1.90    18.61    83.85    46.91     0.01    4.79   0.80   0.17
dm-0              0.00     0.00    0.08    9.60     1.77    76.79     8.12     0.05    5.03   0.11   0.10
dm-1              0.00     0.00    0.01    0.04     0.11     0.32     8.00     0.00   14.75   0.19   0.00
dm-2              0.00     0.00    0.21    0.05    16.73     6.74    89.38     0.00    4.25   2.81   0.07

Here are the definitions 

    * rrqm/s : The number of read requests merged per second that were queued to the hard disk
    * wrqm/s : The number of write requests merged per second that were queued to the hard disk
    * r/s : The number of read requests per second
    * w/s : The number of write requests per second
    * rsec/s : The number of sectors read from the hard disk per second
    * wsec/s : The number of sectors written to the hard disk per second
    * avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
    * avgqu-sz : The average queue length of the requests that were issued to the device
    * await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
    * svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
    * %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

First you need to note down following values from the iostat output:

   1. The average service time (svctm)
   2. Percentage of CPU time during which I/O requests were issued (%util)
   3. See if a hard disk reports consistently high reads/writes (r/s and w/s)

If any one of these are high, you need to take one of the following action:

    * Get high speed disk and controller for file system (for example move from SATA I to SAS 15k disk)
    * Tune software or application or kernel or file system for better disk utilization
    * Use RAID array to spread the file system

If utilization is over 50%, its time to start looking at how to distribute the work load.

aptitude update
aptitude install sysstat

iostat -m 5

iostat -m 5 /dev/md5

iostat -x 1

sar -P ALL 1 0
vmstat

aptitude install hdparm
#Perorm read timing.This will display read performance.
hdparm -t /dev/md5

Some statistics on block size of the harddrive 

dd if=/dev/zero of=/home/lucas/bigfile bs=64k count=8192
dd if=/dev/zero of=/home/lucas/bigfile bs=128k count=8192
dd if=/dev/zero of=/home/lucas/bigfile bs=256k count=8192
dd if=/dev/zero of=/home/lucas/bigfile bs=512k count=8192
dd if=/dev/zero of=/home/lucas/bigfile bs=1024k count=8192
dd if=/dev/zero of=/home/lucas/bigfile bs=2048k count=8192
dd if=/dev/zero of=/home/lucas/bigfile bs=4096k count=8192

cat /proc/mdstat   
cat /sys/block/md0/md/stripe_cache_size
echo 1024 > /sys/block/md1/md/stripe_cache_size

Example

#cpu information

cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 65
model name      : Dual-Core AMD Opteron(tm) Processor 2214
stepping        : 3
cpu MHz         : 2200.009
cache size      : 1024 KB

#memory size

free

             total       used       free     shared    buffers     cached
Mem:       8184404    4068088    4116316          0      20448    3156488
-/+ buffers/cache:     891152    7293252
Swap:     18321400         84   18321316



#Active strip size

cat /sys/block/md5/md/stripe_cache_active
387

#Strip cache size in memory hold by OS. As you can see it holds 512 as a default.
cat /sys/block/md5/md/stripe_cache_size
512


#Raid setup description. As you see my md5 is my raid5.
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md5 : active raid5 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      1462211904 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md0 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      979840 blocks [4/4] [UUUU]

# Write a bigfile to /home/ folder which is on my raid5 array
dd if=/dev/zero of=/home/lucas/bigfile bs=1M count=8192

8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 265.931 seconds, 32.3 MB/s

#Delete the file.
rm /home/lucas/bigfile

#Change the cache size
echo 1024 > /sys/block/md5/md/stripe_cache_siz 

#Run command:
dd if=/dev/zero of=/home/unique/bigfile bs=1M count=8192

#A little bit more performance then with the original settings.
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 160.36 seconds, 53.6 MB/s

#Delete the file.
rm /home/lucas/bigfile

#Change the cache size
echo 2048 > /sys/block/md5/md/stripe_cache_size 

dd if=/dev/zero of=/home/unique/bigfile bs=1M count=8192

8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 81.7626 seconds, 105 MB/s

#Delete the file.
rm /home/lucas/bigfile

#Change the cache size
echo 4096 > /sys/block/md5/md/stripe_cache_size 

#Test now.
dd if=/dev/zero of=/home/unique/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 82.7625 seconds, 104 MB/s

#Delete the file.
rm /home/lucas/bigfile

#Change the cache size
echo 8192 > /sys/block/md5/md/stripe_cache_size 

#Test now.
dd if=/dev/zero of=/home/unique/bigfile bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 64.9024 seconds, 132 MB/s

Backing up Databases

Mysql

#Author: Lukasz Szybalski
#License: LGPL
#Feedback: szybalski@gmail.com

#This program creates a mysqlbackup using mysqldump. It gzip the file via a pipe. It creats a backup file for each day it runs. There is a delete line that will delete a backup from today - 14 days. This script is intendent to run daily via cron. When done do cron -e and set the cron line to something like this. Run at 2am every day. Change the folder location,etc.:
#0 2 * * * /home/trac/backup/mysql_backup.sh



# modify the following to suit your environment
export DB_BACKUP="/home/trac/backup/mysql_backup"
export DB_USER="root"
export DB_PASSWD="somepassword"

if [ -d $DB_BACKUP ];
then
        echo "Folder exists, proceeding.."
else
        mkdir -p $DB_BACKUP
        echo "Creating Folder '$DB_BACKUP'"
fi


# title and version
echo ""
echo "MySQL_backup on `hostname`"
echo "----------------------"

#Delete the file that is 14 days old.
rm -f $DB_BACKUP/mysql-`hostname`-`date --date='14 days ago' '+%Y%m%d'`.gz

#Create a new backup
echo "* Creating new backup..."
mysqldump --user=$DB_USER --password=$DB_PASSWD --all-databases | gzip > $DB_BACKUP/mysql-`hostname`-`date +%Y%m%d`.gz
#mysqldump --user=$DB_USER --password=$DB_PASSWD --host=$DB_HOST --databases trac | gzip > $DB_BACKUP/mysql-`hostname`-trac-`date +%Y%m%d`.gz
echo "----------------------"
echo "Done"
exit 0

#RESTORE Procedure (You should always write a restore procedure. Test it too)
# mysql -u root -p
#create user 'trac'@'%' IDENTIFIED BY 'somepass';
#create database trac;
#GRANT ALL PRIVILEGES ON trac.* TO 'trac'@'%' WITH GRANT OPTION;
#FLUSH PRIVILEGES;

#Restore on command line
#mysql -u trac -p database_name_to_restore_to <mysqldump_file_to_restore-20100215 
#mysql -u trac -p claimtrac <mysql-trac-20100215

Offsite Storage

Amazon S3

Maximum Block Size = nnn 
or
Maximum Block Size = 5368709120  #5,368,709,120

Amazon S3cmd

aptitude update
aptitude install s3cmd

#Check the website for most current version
wget http://superb-west.dl.sourceforge.net/sourceforge/s3tools/s3cmd-0.9.9.tar.gz
tar -xzvf s3cmd-0.9.9.tar.gz
cd s3cmd-0.9.9
python setup.py install

No module named etree.ElementTree
No module named elementtree.ElementTree
Please install ElementTree module from
http://effbot.org/zone/element-index.htm

aptitude install python-elementtree

python setup.py install

Configure

s3cmd --configure

Access key:
Secret key:
Password for encryption
Answer whether you want to use https
Test if you can connect

Using s3cmd

s3cmd ls

s3cmd mb s3://someuniquename.my-new-bucket-name

s3cmd ls s3://someuniquename.my-new-bucket-name
#Example
s3cmd ls s3://logix.cz-test
Bucket 'logix.cz-test':
.....

s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml
#or
s3cmd put file-* s3://logix.cz-test/

s3cmd put --acl-private --guess-mime-type storage.jpg s3://logix.cz-test/storage.jpg

s3cmd get s3://logix.cz-test/addrbook.xml addressbook-2.xml
#verify md5sum, you could potentially upload a md5sums to amazon and verify them them during restore.
md5sum addressbook.xml addressbook-2.xml
   39bcb6992e461b269b95b3bda303addf  addressbook.xml
   39bcb6992e461b269b95b3bda303addf  addressbook-2.xml

s3cmd del s3://logix.cz-test/addrbook.xml s3://logix.cz-test/storage.jpg
   Object s3://logix.cz-test/addrbook.xml deleted
   Object s3://logix.cz-test/storage.jpg deleted

s3cmd rb s3://logix.cz-test
   Bucket 'logix.cz-test' removed

s3cmd put --acl-private data_archives_20080[1,2,3,4].zip.gpg s3://logix.cz-test/data_archives_2008_part1/
s3cmd put --acl-private data_archives_20080[5,6,7,8].zip.gpg s3://logix.cz-test/data_archives_2008_part2/
s3cmd put --acl-private data_archives_2008[09,10,11,12].zip.gpg s3://logix.cz-test/data_archives_2008_part3/

Split Files into 5GB

split -b 5120m filename newfilename

split -b 5120m S_drive_shared.tar.gz.gpg S_drive_shared.tar.gz.gpg
#will create
S_drive_shared.tar.gz.gpgaa
S_drive_shared.tar.gz.gpgab
S_drive_shared.tar.gz.gpgac

cat S_drive_shared.tar.gz.gpg* >>S_drive_shared.tar.gz.gpg

for i in *;do md5sum $i >>md5sum.log;done

List all files on amazon S3

for i in `s3cmd ls |sed -e 's/^.\{18\}//'`;do s3cmd ls $i;done >>amazon_s3_20090515.txt

rsync and Encryption (gpg)

Here is a quick guide to gnupg 

aptitude update
aptitude install gnupg

gpg --gen-key

gpg --list-keys
gpg --list-secret-keys

gpg -r NAME --output OUTFILE.gpg --encrypt INFILE.zip
or 
gpg -r NAME --output --encrypt INFILE.zip

gpg -r NAME --output OUTFILE --decrypt INFILE.gpg

To export a public key to an ascii text file, run:
gpg -a --export NAME > yourpublickey.gpg

To export a private (or secret) key to an ascii text file, run:
gpg -a --export-secret-keys NAME > yourprivatekey.gpg

gpg --import yoursecretkey.gpg
#Change the trust level of this key to (5) ultimate
gpg --edit-key NAME
trust
#(choose 5)

md5sum myarchivefile.zip
md5sum myarchivefile.zip.gpg

aptitude install pyp
read password
for i in *.gpg;do echo "${password}"| gpg --passphrase-fd 0 -r NAME --output `ls $i|pyp p[:-4]` --decrypt $i;done

#Other python3 pyp examples
for i in data-2010-*.jpg;do mv -v "$i" `echo $i|pyp "p.replace(' ','').replace('-','')"`;done

Bacula and encryption

apt-get build-dep bacula
aptitude install libssl-dev openssl
aptitude source bacula
cd bacula-2....
vim debian/rules
# (change the options to enable ssl tls)
debian/rules binary
#it should now compile and pack all bacula packages
cd ..
dpkg -i bacula*.deb

References

  1. http://www.bacula.org/presentations/Bacula-UKUUG-talk-20Feb08.pdf

  2. http://www.howto.gr/dr/node/2

  3. http://www.linux-mag.com/id/4429

  4. Debian Config Files in /etc/bacula/bacula-*.conf

  5. http://www.bacula.org/en/dev-manual/Brief_Tutorial.html#TutorialChapter

  6. http://www.isds.duke.edu/~brown/bacula-talk/Bacula_ISDS.pdf

  7. http://www.bacula.org/en/dev-manual/Basic_Volume_Management.html

  8. s3cmd

  9. http://www.cyberciti.biz/tips/linux-disk-performance-monitoring-howto.html

MyWiki: Bacula (last edited 2016-10-27 02:13:19 by LukaszSzybalski)