Add chapter on backups

This commit is contained in:
Sven Vermeulen 2010-09-02 14:26:33 +02:00
parent 7f74416b4f
commit 803bda42a1
4 changed files with 520 additions and 1 deletions

View File

@ -1,5 +1,6 @@
** (2010-09-02) Sven Vermeulen <>
- Add chapter on logfile management
- Add chapter on backups
** (2010-08-27) Sven Vermeulen <>
- Add information on kernel module parameters

View File

@ -18,6 +18,7 @@
<!ENTITY graphicenvironment.xml SYSTEM "linux_sea/15-graphicenvironment.xml">
<!ENTITY installgentoo.xml SYSTEM "linux_sea/16-installgentoo.xml">
<!ENTITY logfilemanagement.xml SYSTEM "linux_sea/17-logfilemanagement.xml">
<!ENTITY backup.xml SYSTEM "linux_sea/18-backup.xml">
<!ENTITY tipsandanswers.xml SYSTEM "linux_sea/90-tipsandanswers.xml">
<!ENTITY glossary.xml SYSTEM "linux_sea/91-glossary.xml">
@ -107,6 +108,7 @@
<!-- Part - Additional services on a Linux system -->
<!-- Part - Addenda -->

View File

@ -7,6 +7,303 @@
<para>If the previous chapters are all read through, you'll most likely
have succesfully installed Gentoo Linux. However, managing your Linux
system is only starting. Within this chapter, we go through how the
various Linux services log events on your system and how you can properly
manage these log files.</para>
<title>System Logger</title>
<para>The system logger is a service that allows for various software
tools to log their events through a unified, standardized interface. For
locally running software, these tools can log through the
<filename>/dev/log</filename> socket. Services that run remotely can send
their events through the network. For a general understanding of logging
daemons, having to deal with remotely running software might bring us too
<para>The <filename>/dev/log</filename> socket is created by the system
logger (which we have very briefly discussed previously) and is made
writeable for everyone. That is to be expected, as there are many tools
that will log through this socket. Every tool that wants to log through
the system logger logs a single line at a time inside this socket. The
system logger is listening "on the other side" of the socket and will
process these log events.</para>
<title>Log Event Meta Information</title>
<para>You will most likely never need to write your own tool that logs
through the system logger, yet understanding how a log event is sent is
vital if you want to manage your log files properly. The reason for this
is because you will want to filter based on criticality and origin of
the events, and these are passed on as part of the log event
<para>When a tool wants to log an event, he needs to provide two
additional fields: </para>
<para>the facility where the event is about, and</para>
<para>the importance level of the event</para>
<para>The <emphasis>facility</emphasis><indexterm>
</indexterm> defines the type of the program that is sending out the
event. This allows the system logger to properly filter messages,
possibly sending them into different log files. Example facilities are
authpriv (security/authorization messages), cron (scheduling messages)
and kern (kernel messages). A full list of facilities can be obtained
through the syslog man page</para>
<programlisting>~$ <command>man 3 syslog</command></programlisting>
<para>The <emphasis>importance level</emphasis><indexterm>
<primary>importance level</primary>
</indexterm> defines the criticality of the event. The set of
supported importance levels is:</para>
<title>System logger importance levels</title>
<tgroup cols="2">
<entry>Messages used for debugging purposes</entry>
<entry>Informational messages</entry>
<entry>A normal, yet somewhat more important message than
<entry>This message needs your attention</entry>
<entry>Something wicked has occurred and will need further
<entry>A critical error has occurred, regular operations might
be interrupted or run in a degraded mode</entry>
<entry>Immediate action needs to be taken</entry>
<entry>The system is unusable (no, this has nothing to do with
<command>emerge</command>, Gentoo Portage' installation
<para>Based on these two fields, log messages are then filtered by the
system logger into one or more log files.</para>
<title>System Logger Configuration</title>
<para>How a system logger is configured depends on the system logger you
use. In this book, I'll focus on the <command>syslog-ng</command>
<para>The configuration file for syslog-ng is
</indexterm>. An example configuration is displayed below.</para>
<programlisting>@version: 3.0
options {
source src {
unix-stream("/dev/log" max-connections(256));
destination messages { file("/var/log/messages"); };
destination cron { file("/var/log/cron.log"); };
destination auth { file("/var/log/auth.log"); };
filter f_messages { not facility(cron, auth, authpriv); };
filter f_cron { facility(cron); };
filter f_auth { facility(auth, authpriv); };
filter f_warnplus { level(warn, err, crit, emerg); };
log { source(src); filter(f_cron); filter(f_warnplus); destination(cron); };
log { source(src); filter(f_auth); destination(auth); };
log { source(src); filter(f_messages); destination(messages); };</programlisting>
<para>It might be easy to read the configuration file from the bottom
up. </para>
<para>The log entries define where messages come from (source),
which filters the system logger applies (filter) and where the
resulting messages are stored in (destination)</para>
<para>The filter entries define what the filters actually do. For
instance, the filter f_warnplus only accepts events with an
importance level of warn or higher.</para>
<para>The destination entries define where the events are stored in
(the logfiles)</para>
<para>The source entry defines where the system logger gets its
messages from (which, in this case, is the /dev/log socket, the
kernel message interface kmsg and its own internal logging)</para>
<para>This fairly simple example immediately shows how flexible the logs
can work. There are many more interesting filters you can apply, such as
match() to match regular expressions within the logged event and
program() to match log events of a particular tool.</para>
<title>Non-Syslog Logfiles</title>
<para>Many tools log through the system logger, but it is not a huge
majority. Lots and lots of tools, server software and others have their
own logging system. This makes it a bit more difficult to fully manage the
logfiles properly. However, if you know where the logfiles are, then
that's a start.</para>
<title>Xorg Logging</title>
<para>The Xorg server stores it log file at
<filename>/var/log/Xorg.0.log</filename>. The trailing 0 denotes that
this is of the current/last startup. The logfile of the start before
that is called <filename>Xorg.1.log</filename>, and so on. </para>
<para>Xorg uses the following notations to identify the various
criticality levels:</para>
<programlisting>Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.</programlisting>
<para>The Xorg server will automatically rotate the log files, by
default 3 times, after which it will erase the oldest logfile.</para>
<title>Maintaining Logfiles</title>
<para>Most tools do not offer log rotation or cleanup by default. It is
therefore recommended to implement some sort of log rotation for your
files. An interesting tool to install for this purpose is
</indexterm>. The tool is triggered by your system scheduler (cron) and
is best configured by creating separate configuration files for your log
files. </para>
<title>Installing Logrotate</title>
<para>First, install logrotate:</para>
<programlisting>~# <command>emerge logrotate</command></programlisting>
<para>Next, make sure that it is scheduled to be ran every day. For
instance, you can have a script called logrotate.cron inside
<filename>/etc/cron.daily</filename> with the following content (Gentoo
users will have this as part of the installation):</para>
/usr/sbin/logrotate /etc/logrotate.conf</programlisting>
<title>Configuring Logrotate</title>
<para>To configure logrotate, create configuration files inside
/etc/logrotate.d. For instance, the following /etc/logrotate.d/syslog-ng
helps us rotate the logfiles that we identified as part of the system
logger configuration:</para>
<programlisting>/var/log/messages /var/log/cron.log /var/log/auth.log {
rotate 6
# It is ok if a logfile is missing
# Only execute postrotate after all 3 logfiles are rotated
# instead of after each logfile
# Commands to run after the rotation has occurred
# This will instruct syslog-ng to reload its configuration
# and log files
/etc/init.d/syslog-ng reload &gt; /dev/null 2&gt;&amp;1 || true
<para>The file informs logrotate that the mentioned logfiles are rotated
6 times on a monthly basis (so you keep 7 months of history). The tool
will automatically rename the "old" logfiles by adding a datestamp to
the filename. You can also instruct logrotate to compress the rotated
logfiles or even move them to a different location.</para>

src/linux_sea/18-backup.xml Normal file
View File

@ -0,0 +1,219 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
<title>Taking Backups</title>
<para>To finalize this book, lets focus on backups and restores. After
all, you don't want to put all that effort in your Linux system, only to
see it vanish with the next power surge or
daughter-spilled-milk-over-my-laptop incident.</para>
<title>Poor Mans' Backup</title>
<para>Not taking any backups is living on the edge. Especially in case of
Gentoo Linux, you don't want to lose all the effort you've put in setting
up your system. A basic, poor man's backup, is nothing more but an
off-line copy of your important files. This can be on a USB stick, an SD
card or a CD/DVD. But what to backup?</para>
<para>The following sections describe a possible set of files and
directories to back up, without resorting to lots and lots of storage
<title>System Settings</title>
<para>First of all, take a copy of your <filename>/etc</filename>
folder. The <filename>/etc</filename> directory contains all system
settings, including your file system table (<filename>fstab</filename>),
Portage-specific settings (<filename>/etc/make.conf</filename>,
<filename>/etc/portage</filename>), service configuration files (in
<filename>/etc/conf.d</filename> and <filename>/etc/env.d</filename>),
... Really, this location is not to be forgotten.</para>
<para>Next, there are a few non-/etc files or directories that you want
to copy as well. </para>
<para><filename>/var/lib/portage/world</filename> is your
world-file, which contains a list of all software you've installed
(well, not their dependencies, but that's not needed anyhow). In
case that you ever need to rebuild your system, this file will tell
you what needs to be installed</para>
<para><filename>/usr/local</filename> contains all non-Portage
installed software and settings. Although you might not have
anything inside that, those of you that do will most likely take a
backup of that as well.</para>
<para><filename>/proc/config.gz</filename> (or
<filename>/usr/src/linux/.config</filename>) is your kernel
configuration. You've probably lost a lot of sweat and oxygen while
setting up your kernel configuration, so you definitely don't want
to lose it</para>
<title>User Settings</title>
<para>Next to the system settings, you'll find that your end user(s) on
your system also have gone through lots of trouble to set up their
system as they want it. End users' configuration settings are stored in
the end users' home directories. There is no single directory for all
configuration settings, so finding which ones to back up and which not
might be difficult. However, one way that works for me (and which
usually doesn't include the end users' personal files) is to backup all
<filename>${HOME}/.??*</filename> files and directories. Most
configuration directories (or files) are hidden files (i.e. starting
with a period).</para>
<title>Sample Script</title>
<para>The following command will make an archive of the above mentioned
settings. </para>
<programlisting>~# <command>tar cvzpf /var/tmp/poormanbackup.tar.gz /etc /var/lib/portage/world /usr/local /proc/config.gz /home/*/.??*</command></programlisting>
<para>Again, this is just a poor man's backup, because it is not really
managed, but it is still better than nothing. Make sure you copy the
poormanbackup.tar.gz to a safe(r) place. Note that it will still be
fairly large, because of the end user copy (as it contains lots of
software caches, like your browsers' cache). If you drop the end user
directories (the <filename>/home/*/.??*</filename>) the file is reduced
from many dozens (up to hundreds) of megabytes down to about half a
<para>To restore files, get the file on your system (mount the USB
stick/CD) and extract the file(s) you need. For instance, to extract all
<filename>/etc/portage/*</filename> files:</para>
<programlisting>~# <command>tar xvzpf /media/usb/poormanbackup.tar.gz -C / etc/portage</command></programlisting>
<para>Note the space between <filename>/</filename> and
<title>More Managed Backups</title>
<para>A slightly more complicated, but probably safer method, is to use
tools that provide you with a managed backup solution. Managed means that
you tell them what to backup (although they do propose certain backup
situations) and they take care of the rest, such as repetitive backups, as
well as easy restore activities in case things go wrong.</para>
<para>If you take a look at the packages that Gentoo offers in its
app-backup category, you'll find that there are plenty of them. I can't
tell you which one is better, because that is a personal taste. But I will
provide a few pointers so you can get started.</para>
<title>Backup Ninja</title>
<para>The backupninja application is a tool which is scheduled to run
every night (for instance) and that will read in the configuration
file(s) in /etc/backup.d. Inside this directory, you define what needs
to be backed up, how often, etc. For instance, the same poor man's
backup settings of above would result in a file similar to the
<para> File <filename>/etc/backup.d/20-basic-system.rdiff</filename> to
backup regular files:</para>
<programlisting>when = daily
options = --exclude-special-files
nicelevel = 15
label = myhostname.mydomain
type = local
keep = 60
include = /etc
include = /var/lib/portage/world
include = /usr/local
include = /home/*/.??*
exclude = /home/*/.mozilla/firefox/*.default/Cache
type = local
directory = /var/backups/backupninja</programlisting>
<programlisting>zcat /proc/config.gz &gt; /etc/kernel-config</programlisting>
<para>File <filename>/etc/backup.d/</filename>:</para>
<programlisting>rsync -avug /var/backups/backupninja /media/usb</programlisting>
<para>The result of this is that the kernel configuration file is
generated first (into /etc/kernel-config) after which (incremental)
backups are taken of the presented locations, and stored in
/var/backups/backupninja. Finally, these files are copied (synchronized)
to /media/usb (which might be a USB stick or external disk drive). If
you ever want to store that backup outside of your home (for instance,
at your parents/kids house), detach the USB stick and store it there.
Just make sure another USB stick is attached and mounted for the next
backup cycle.</para>
<title>Bare Metal Recovery</title>
<para>When you want to make sure your system is up and running in no time,
even when the entire system crashed, you need to take bare metal backups
(full system backups) which you can restore quickly. A popular method to
accomplish this is to use imaging software tools.</para>
<para>If you installed Gentoo using the sysresccd CD, then you already
have the tools at your disposal to perform imaging backups. A popular
tool is <emphasis>PartImage</emphasis><indexterm>
</indexterm>, and is available on the sysresccd.</para>
<para>The tool offers a graphical (or curses) based interface, but you
don't need to use that: the tool also supports command-line
<para>An example usage would be the following. Boot from the sysresccd,
and attach USB storage to your system on which you want to back up your
entire system:</para>
<programlisting>~# <command>mount /dev/sdc1 /media/usb</command></programlisting>
<para>Next, take image(s) of your partitions, or take a full disk
backup. Don't worry, PartImage is able to detect when diskspace isn't
used, and will skip those empty blocks. The following takes a full
backup of /dev/sda in blocks of 700Mbyte (in case you ever want to store
the files on CD or DVD):</para>
<programlisting>~# <command>partimage -b -z1 -o V700 save /dev/sda /media/usb/gentoo_full.sda.partimg.gz</command></programlisting>
<para>If you ever need to restore the system, use the following
<programlisting>~# <command>partimage -e restore /dev/sda /media/usb/gentoo_full.sda.partimg.gz.000</command></programlisting>