Add chapter on backups
Sven Vermeulen
parent
7f74416b4f
commit
803bda42a1
|
|
@ -1,5 +1,6 @@
|
|||
** (2010-09-02) Sven Vermeulen <sven.vermeulen@siphos.be>
|
||||
- Add chapter on logfile management
|
||||
- Add chapter on backups
|
||||
|
||||
** (2010-08-27) Sven Vermeulen <sven.vermeulen@siphos.be>
|
||||
- Add information on kernel module parameters
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@
|
|||
<!ENTITY graphicenvironment.xml SYSTEM "linux_sea/15-graphicenvironment.xml">
|
||||
<!ENTITY installgentoo.xml SYSTEM "linux_sea/16-installgentoo.xml">
|
||||
<!ENTITY logfilemanagement.xml SYSTEM "linux_sea/17-logfilemanagement.xml">
|
||||
<!ENTITY backup.xml SYSTEM "linux_sea/18-backup.xml">
|
||||
|
||||
<!ENTITY tipsandanswers.xml SYSTEM "linux_sea/90-tipsandanswers.xml">
|
||||
<!ENTITY glossary.xml SYSTEM "linux_sea/91-glossary.xml">
|
||||
|
|
@ -107,6 +108,7 @@
|
|||
&installgentoo.xml;
|
||||
<!-- Part - Additional services on a Linux system -->
|
||||
&logfilemanagement.xml;
|
||||
&backup.xml;
|
||||
<!-- Part - Addenda -->
|
||||
&tipsandanswers.xml;
|
||||
&glossary.xml;
|
||||
|
|
|
|||
|
|
@ -7,6 +7,303 @@
|
|||
<section>
|
||||
<title>Introduction</title>
|
||||
|
||||
<para></para>
|
||||
<para>If the previous chapters are all read through, you'll most likely
|
||||
have succesfully installed Gentoo Linux. However, managing your Linux
|
||||
system is only starting. Within this chapter, we go through how the
|
||||
various Linux services log events on your system and how you can properly
|
||||
manage these log files.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>System Logger</title>
|
||||
|
||||
<para>The system logger is a service that allows for various software
|
||||
tools to log their events through a unified, standardized interface. For
|
||||
locally running software, these tools can log through the
|
||||
<filename>/dev/log</filename> socket. Services that run remotely can send
|
||||
their events through the network. For a general understanding of logging
|
||||
daemons, having to deal with remotely running software might bring us too
|
||||
far.</para>
|
||||
|
||||
<section>
|
||||
<title>/dev/log</title>
|
||||
|
||||
<para>The <filename>/dev/log</filename> socket is created by the system
|
||||
logger (which we have very briefly discussed previously) and is made
|
||||
writeable for everyone. That is to be expected, as there are many tools
|
||||
that will log through this socket. Every tool that wants to log through
|
||||
the system logger logs a single line at a time inside this socket. The
|
||||
system logger is listening "on the other side" of the socket and will
|
||||
process these log events.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Log Event Meta Information</title>
|
||||
|
||||
<para>You will most likely never need to write your own tool that logs
|
||||
through the system logger, yet understanding how a log event is sent is
|
||||
vital if you want to manage your log files properly. The reason for this
|
||||
is because you will want to filter based on criticality and origin of
|
||||
the events, and these are passed on as part of the log event
|
||||
itself.</para>
|
||||
|
||||
<para>When a tool wants to log an event, he needs to provide two
|
||||
additional fields: </para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>the facility where the event is about, and</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>the importance level of the event</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<para>The <emphasis>facility</emphasis><indexterm>
|
||||
<primary>facility</primary>
|
||||
|
||||
<secondary>syslog</secondary>
|
||||
</indexterm> defines the type of the program that is sending out the
|
||||
event. This allows the system logger to properly filter messages,
|
||||
possibly sending them into different log files. Example facilities are
|
||||
authpriv (security/authorization messages), cron (scheduling messages)
|
||||
and kern (kernel messages). A full list of facilities can be obtained
|
||||
through the syslog man page</para>
|
||||
|
||||
<programlisting>~$ <command>man 3 syslog</command></programlisting>
|
||||
|
||||
<para>The <emphasis>importance level</emphasis><indexterm>
|
||||
<primary>importance level</primary>
|
||||
|
||||
<secondary>syslog</secondary>
|
||||
</indexterm> defines the criticality of the event. The set of
|
||||
supported importance levels is:</para>
|
||||
|
||||
<table>
|
||||
<title>System logger importance levels</title>
|
||||
|
||||
<tgroup cols="2">
|
||||
<tbody>
|
||||
<row>
|
||||
<entry>DEBUG</entry>
|
||||
|
||||
<entry>Messages used for debugging purposes</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>INFO</entry>
|
||||
|
||||
<entry>Informational messages</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>NOTICE</entry>
|
||||
|
||||
<entry>A normal, yet somewhat more important message than
|
||||
informational</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>WARNING</entry>
|
||||
|
||||
<entry>This message needs your attention</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>ERROR</entry>
|
||||
|
||||
<entry>Something wicked has occurred and will need further
|
||||
investigation</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>CRIT</entry>
|
||||
|
||||
<entry>A critical error has occurred, regular operations might
|
||||
be interrupted or run in a degraded mode</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>ALERT</entry>
|
||||
|
||||
<entry>Immediate action needs to be taken</entry>
|
||||
</row>
|
||||
|
||||
<row>
|
||||
<entry>EMERG</entry>
|
||||
|
||||
<entry>The system is unusable (no, this has nothing to do with
|
||||
<command>emerge</command>, Gentoo Portage' installation
|
||||
tool)</entry>
|
||||
</row>
|
||||
</tbody>
|
||||
</tgroup>
|
||||
</table>
|
||||
|
||||
<para>Based on these two fields, log messages are then filtered by the
|
||||
system logger into one or more log files.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>System Logger Configuration</title>
|
||||
|
||||
<para>How a system logger is configured depends on the system logger you
|
||||
use. In this book, I'll focus on the <command>syslog-ng</command>
|
||||
logger.</para>
|
||||
|
||||
<para>The configuration file for syslog-ng is
|
||||
<filename>/etc/syslog-ng/syslog-ng.conf</filename><indexterm>
|
||||
<primary>syslog-ng.conf</primary>
|
||||
</indexterm>. An example configuration is displayed below.</para>
|
||||
|
||||
<programlisting>@version: 3.0
|
||||
options {
|
||||
stats_freq(43200);
|
||||
};
|
||||
|
||||
source src {
|
||||
unix-stream("/dev/log" max-connections(256));
|
||||
internal();
|
||||
file("/proc/kmsg");
|
||||
};
|
||||
|
||||
destination messages { file("/var/log/messages"); };
|
||||
destination cron { file("/var/log/cron.log"); };
|
||||
destination auth { file("/var/log/auth.log"); };
|
||||
|
||||
filter f_messages { not facility(cron, auth, authpriv); };
|
||||
filter f_cron { facility(cron); };
|
||||
filter f_auth { facility(auth, authpriv); };
|
||||
|
||||
filter f_warnplus { level(warn, err, crit, emerg); };
|
||||
|
||||
log { source(src); filter(f_cron); filter(f_warnplus); destination(cron); };
|
||||
log { source(src); filter(f_auth); destination(auth); };
|
||||
log { source(src); filter(f_messages); destination(messages); };</programlisting>
|
||||
|
||||
<para>It might be easy to read the configuration file from the bottom
|
||||
up. </para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para>The log entries define where messages come from (source),
|
||||
which filters the system logger applies (filter) and where the
|
||||
resulting messages are stored in (destination)</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>The filter entries define what the filters actually do. For
|
||||
instance, the filter f_warnplus only accepts events with an
|
||||
importance level of warn or higher.</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>The destination entries define where the events are stored in
|
||||
(the logfiles)</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para>The source entry defines where the system logger gets its
|
||||
messages from (which, in this case, is the /dev/log socket, the
|
||||
kernel message interface kmsg and its own internal logging)</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
|
||||
<para>This fairly simple example immediately shows how flexible the logs
|
||||
can work. There are many more interesting filters you can apply, such as
|
||||
match() to match regular expressions within the logged event and
|
||||
program() to match log events of a particular tool.</para>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Non-Syslog Logfiles</title>
|
||||
|
||||
<para>Many tools log through the system logger, but it is not a huge
|
||||
majority. Lots and lots of tools, server software and others have their
|
||||
own logging system. This makes it a bit more difficult to fully manage the
|
||||
logfiles properly. However, if you know where the logfiles are, then
|
||||
that's a start.</para>
|
||||
|
||||
<section>
|
||||
<title>Xorg Logging</title>
|
||||
|
||||
<para>The Xorg server stores it log file at
|
||||
<filename>/var/log/Xorg.0.log</filename>. The trailing 0 denotes that
|
||||
this is of the current/last startup. The logfile of the start before
|
||||
that is called <filename>Xorg.1.log</filename>, and so on. </para>
|
||||
|
||||
<para>Xorg uses the following notations to identify the various
|
||||
criticality levels:</para>
|
||||
|
||||
<programlisting>Markers: (--) probed, (**) from config file, (==) default setting,
|
||||
(++) from command line, (!!) notice, (II) informational,
|
||||
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.</programlisting>
|
||||
|
||||
<para>The Xorg server will automatically rotate the log files, by
|
||||
default 3 times, after which it will erase the oldest logfile.</para>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Maintaining Logfiles</title>
|
||||
|
||||
<para>Most tools do not offer log rotation or cleanup by default. It is
|
||||
therefore recommended to implement some sort of log rotation for your
|
||||
files. An interesting tool to install for this purpose is
|
||||
<package>app-admin/logrotate</package><indexterm>
|
||||
<primary>logrotate</primary>
|
||||
</indexterm>. The tool is triggered by your system scheduler (cron) and
|
||||
is best configured by creating separate configuration files for your log
|
||||
files. </para>
|
||||
|
||||
<section>
|
||||
<title>Installing Logrotate</title>
|
||||
|
||||
<para>First, install logrotate:</para>
|
||||
|
||||
<programlisting>~# <command>emerge logrotate</command></programlisting>
|
||||
|
||||
<para>Next, make sure that it is scheduled to be ran every day. For
|
||||
instance, you can have a script called logrotate.cron inside
|
||||
<filename>/etc/cron.daily</filename> with the following content (Gentoo
|
||||
users will have this as part of the installation):</para>
|
||||
|
||||
<programlisting>#!/bin/sh
|
||||
|
||||
/usr/sbin/logrotate /etc/logrotate.conf</programlisting>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Configuring Logrotate</title>
|
||||
|
||||
<para>To configure logrotate, create configuration files inside
|
||||
/etc/logrotate.d. For instance, the following /etc/logrotate.d/syslog-ng
|
||||
helps us rotate the logfiles that we identified as part of the system
|
||||
logger configuration:</para>
|
||||
|
||||
<programlisting>/var/log/messages /var/log/cron.log /var/log/auth.log {
|
||||
rotate 6
|
||||
monthly
|
||||
# It is ok if a logfile is missing
|
||||
missingok
|
||||
# Only execute postrotate after all 3 logfiles are rotated
|
||||
# instead of after each logfile
|
||||
sharedscripts
|
||||
# Commands to run after the rotation has occurred
|
||||
# This will instruct syslog-ng to reload its configuration
|
||||
# and log files
|
||||
postrotate
|
||||
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
|
||||
endscript
|
||||
}</programlisting>
|
||||
|
||||
<para>The file informs logrotate that the mentioned logfiles are rotated
|
||||
6 times on a monthly basis (so you keep 7 months of history). The tool
|
||||
will automatically rename the "old" logfiles by adding a datestamp to
|
||||
the filename. You can also instruct logrotate to compress the rotated
|
||||
logfiles or even move them to a different location.</para>
|
||||
</section>
|
||||
</section>
|
||||
</chapter>
|
||||
|
|
|
|||
|
|
@ -0,0 +1,219 @@
|
|||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
|
||||
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
|
||||
<chapter>
|
||||
<title>Taking Backups</title>
|
||||
|
||||
<section>
|
||||
<title>Introduction</title>
|
||||
|
||||
<para>To finalize this book, lets focus on backups and restores. After
|
||||
all, you don't want to put all that effort in your Linux system, only to
|
||||
see it vanish with the next power surge or
|
||||
daughter-spilled-milk-over-my-laptop incident.</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Poor Mans' Backup</title>
|
||||
|
||||
<para>Not taking any backups is living on the edge. Especially in case of
|
||||
Gentoo Linux, you don't want to lose all the effort you've put in setting
|
||||
up your system. A basic, poor man's backup, is nothing more but an
|
||||
off-line copy of your important files. This can be on a USB stick, an SD
|
||||
card or a CD/DVD. But what to backup?</para>
|
||||
|
||||
<para>The following sections describe a possible set of files and
|
||||
directories to back up, without resorting to lots and lots of storage
|
||||
requirements.</para>
|
||||
|
||||
<section>
|
||||
<title>System Settings</title>
|
||||
|
||||
<para>First of all, take a copy of your <filename>/etc</filename>
|
||||
folder. The <filename>/etc</filename> directory contains all system
|
||||
settings, including your file system table (<filename>fstab</filename>),
|
||||
Portage-specific settings (<filename>/etc/make.conf</filename>,
|
||||
<filename>/etc/portage</filename>), service configuration files (in
|
||||
<filename>/etc/conf.d</filename> and <filename>/etc/env.d</filename>),
|
||||
... Really, this location is not to be forgotten.</para>
|
||||
|
||||
<para>Next, there are a few non-/etc files or directories that you want
|
||||
to copy as well. </para>
|
||||
|
||||
<itemizedlist>
|
||||
<listitem>
|
||||
<para><filename>/var/lib/portage/world</filename> is your
|
||||
world-file, which contains a list of all software you've installed
|
||||
(well, not their dependencies, but that's not needed anyhow). In
|
||||
case that you ever need to rebuild your system, this file will tell
|
||||
you what needs to be installed</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para><filename>/usr/local</filename> contains all non-Portage
|
||||
installed software and settings. Although you might not have
|
||||
anything inside that, those of you that do will most likely take a
|
||||
backup of that as well.</para>
|
||||
</listitem>
|
||||
|
||||
<listitem>
|
||||
<para><filename>/proc/config.gz</filename> (or
|
||||
<filename>/usr/src/linux/.config</filename>) is your kernel
|
||||
configuration. You've probably lost a lot of sweat and oxygen while
|
||||
setting up your kernel configuration, so you definitely don't want
|
||||
to lose it</para>
|
||||
</listitem>
|
||||
</itemizedlist>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>User Settings</title>
|
||||
|
||||
<para>Next to the system settings, you'll find that your end user(s) on
|
||||
your system also have gone through lots of trouble to set up their
|
||||
system as they want it. End users' configuration settings are stored in
|
||||
the end users' home directories. There is no single directory for all
|
||||
configuration settings, so finding which ones to back up and which not
|
||||
might be difficult. However, one way that works for me (and which
|
||||
usually doesn't include the end users' personal files) is to backup all
|
||||
<filename>${HOME}/.??*</filename> files and directories. Most
|
||||
configuration directories (or files) are hidden files (i.e. starting
|
||||
with a period).</para>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Sample Script</title>
|
||||
|
||||
<para>The following command will make an archive of the above mentioned
|
||||
settings. </para>
|
||||
|
||||
<programlisting>~# <command>tar cvzpf /var/tmp/poormanbackup.tar.gz /etc /var/lib/portage/world /usr/local /proc/config.gz /home/*/.??*</command></programlisting>
|
||||
|
||||
<para>Again, this is just a poor man's backup, because it is not really
|
||||
managed, but it is still better than nothing. Make sure you copy the
|
||||
poormanbackup.tar.gz to a safe(r) place. Note that it will still be
|
||||
fairly large, because of the end user copy (as it contains lots of
|
||||
software caches, like your browsers' cache). If you drop the end user
|
||||
directories (the <filename>/home/*/.??*</filename>) the file is reduced
|
||||
from many dozens (up to hundreds) of megabytes down to about half a
|
||||
megabyte.</para>
|
||||
|
||||
<para>To restore files, get the file on your system (mount the USB
|
||||
stick/CD) and extract the file(s) you need. For instance, to extract all
|
||||
<filename>/etc/portage/*</filename> files:</para>
|
||||
|
||||
<programlisting>~# <command>tar xvzpf /media/usb/poormanbackup.tar.gz -C / etc/portage</command></programlisting>
|
||||
|
||||
<para>Note the space between <filename>/</filename> and
|
||||
<filename>etc/portage</filename>!</para>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>More Managed Backups</title>
|
||||
|
||||
<para>A slightly more complicated, but probably safer method, is to use
|
||||
tools that provide you with a managed backup solution. Managed means that
|
||||
you tell them what to backup (although they do propose certain backup
|
||||
situations) and they take care of the rest, such as repetitive backups, as
|
||||
well as easy restore activities in case things go wrong.</para>
|
||||
|
||||
<para>If you take a look at the packages that Gentoo offers in its
|
||||
app-backup category, you'll find that there are plenty of them. I can't
|
||||
tell you which one is better, because that is a personal taste. But I will
|
||||
provide a few pointers so you can get started.</para>
|
||||
|
||||
<section>
|
||||
<title>Backup Ninja</title>
|
||||
|
||||
<para>The backupninja application is a tool which is scheduled to run
|
||||
every night (for instance) and that will read in the configuration
|
||||
file(s) in /etc/backup.d. Inside this directory, you define what needs
|
||||
to be backed up, how often, etc. For instance, the same poor man's
|
||||
backup settings of above would result in a file similar to the
|
||||
following:</para>
|
||||
|
||||
<para> File <filename>/etc/backup.d/20-basic-system.rdiff</filename> to
|
||||
backup regular files:</para>
|
||||
|
||||
<programlisting>when = daily
|
||||
options = --exclude-special-files
|
||||
nicelevel = 15
|
||||
|
||||
[source]
|
||||
label = myhostname.mydomain
|
||||
type = local
|
||||
keep = 60
|
||||
include = /etc
|
||||
include = /var/lib/portage/world
|
||||
include = /usr/local
|
||||
include = /home/*/.??*
|
||||
exclude = /home/*/.mozilla/firefox/*.default/Cache
|
||||
|
||||
[dest]
|
||||
type = local
|
||||
directory = /var/backups/backupninja</programlisting>
|
||||
|
||||
<para>File
|
||||
<filename>/etc/backup.d/19-kernel-config.sh</filename>:</para>
|
||||
|
||||
<programlisting>zcat /proc/config.gz > /etc/kernel-config</programlisting>
|
||||
|
||||
<para>File <filename>/etc/backup.d/21-copy-to-usb.sh</filename>:</para>
|
||||
|
||||
<programlisting>rsync -avug /var/backups/backupninja /media/usb</programlisting>
|
||||
|
||||
<para>The result of this is that the kernel configuration file is
|
||||
generated first (into /etc/kernel-config) after which (incremental)
|
||||
backups are taken of the presented locations, and stored in
|
||||
/var/backups/backupninja. Finally, these files are copied (synchronized)
|
||||
to /media/usb (which might be a USB stick or external disk drive). If
|
||||
you ever want to store that backup outside of your home (for instance,
|
||||
at your parents/kids house), detach the USB stick and store it there.
|
||||
Just make sure another USB stick is attached and mounted for the next
|
||||
backup cycle.</para>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<title>Bare Metal Recovery</title>
|
||||
|
||||
<para>When you want to make sure your system is up and running in no time,
|
||||
even when the entire system crashed, you need to take bare metal backups
|
||||
(full system backups) which you can restore quickly. A popular method to
|
||||
accomplish this is to use imaging software tools.</para>
|
||||
|
||||
<section>
|
||||
<title>PartImage</title>
|
||||
|
||||
<para>If you installed Gentoo using the sysresccd CD, then you already
|
||||
have the tools at your disposal to perform imaging backups. A popular
|
||||
tool is <emphasis>PartImage</emphasis><indexterm>
|
||||
<primary>PartImage</primary>
|
||||
</indexterm>, and is available on the sysresccd.</para>
|
||||
|
||||
<para>The tool offers a graphical (or curses) based interface, but you
|
||||
don't need to use that: the tool also supports command-line
|
||||
backups.</para>
|
||||
|
||||
<para>An example usage would be the following. Boot from the sysresccd,
|
||||
and attach USB storage to your system on which you want to back up your
|
||||
entire system:</para>
|
||||
|
||||
<programlisting>~# <command>mount /dev/sdc1 /media/usb</command></programlisting>
|
||||
|
||||
<para>Next, take image(s) of your partitions, or take a full disk
|
||||
backup. Don't worry, PartImage is able to detect when diskspace isn't
|
||||
used, and will skip those empty blocks. The following takes a full
|
||||
backup of /dev/sda in blocks of 700Mbyte (in case you ever want to store
|
||||
the files on CD or DVD):</para>
|
||||
|
||||
<programlisting>~# <command>partimage -b -z1 -o V700 save /dev/sda /media/usb/gentoo_full.sda.partimg.gz</command></programlisting>
|
||||
|
||||
<para>If you ever need to restore the system, use the following
|
||||
command:</para>
|
||||
|
||||
<programlisting>~# <command>partimage -e restore /dev/sda /media/usb/gentoo_full.sda.partimg.gz.000</command></programlisting>
|
||||
</section>
|
||||
</section>
|
||||
</chapter>
|
||||
Reference in New Issue