l i n u x - u s e r s - g r o u p - o f - d a v i s
L U G O D
 
Next Meeting:
December 2: Social gathering
Next Installfest:
TBD
Latest News:
Nov. 18: Club officer elections
Page last updated:
2003 Feb 28 12:55

The following is an archive of a post made to our 'vox-tech mailing list' by one of its subscribers.

Report this post as spam:

(Enter your email address)
[vox-tech] HOWTO: Make Your Our Debian Mirror
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[vox-tech] HOWTO: Make Your Our Debian Mirror



Thu Feb 27 13:25:40 PST 2003 version 0.1

  This is a quick and dirty setup your own Debian Mirror Howto.
I've been asked how to set one up several times but I've finally been
asked by someone persistent enough to keep asking, you can thank them
for this document :).

  All of the files mentioned in this howto should be downloadable from
the following URL:
  http://simons-clan.com/~msimons/debian/mirror/

  If anyone has comments email me with suggestions.

  Familiarity with the Debian Install process is expected...

  This example is for i386 architecture, but should be easy to change to
suit.

  I also expect to have messed up one or two steps, so run with care.

    Good Luck,
      Mike Simons
      msimons@simons-clan.com


1 - Initial setups:
===================

  In this document the mirror is going to be put in '/home/debian'.
mirror is the directory the archive will actually go.  bin is where the 
scripts go.

===
adduser debian
m=/home/debian
mkdir -p $m/bin $m/mirror
cd $m/bin
wget http://simons-clan.com/~msimons/debian/mirror/debmirror
wget http://simons-clan.com/~msimons/debian/mirror/run
chown -R debian:debian $m
===


debmirror
=========
  The core worker script this howto uses is called "debmirror" it a script 
originally written by Joey Hess, which I found on some Debian mailing list 
over two years ago.  I had been operating my own mirror but when Debian 
moved to package pools my older scripts which just did an rsync/ftp mirror 
broke... so after a short search I found this one from Joey and used it.

  Over time I fixed a number of things that were bothering me about the
way it worked... and later I found that it was part of a debian package
"debmirror".  I sync'ed my version up with debian package version and 
tried to get my changes merged into the main archive... but that never 
happened.

  I have not looked at the debian version of the debmirror script in 
ages.  Ways my version was better than the packaged version include:
====
- Does not download a file unless it actually changed
  (based on file size and timestamp check of remote to local)
- Auto-resumes a file transfer if the transfer was aborted part way 
  through.
- Don't change the local archive's Packages files until *AFTER* all
  the files mentioned inside have been downloaded successfully
  (this was you can use the mirror while it is syncing up, or after
  a failed sync attempt).
- Automatically detect if passive mode ftp is needed or not (attempt
  non-passive mode, if that fails try passive, warn user and continue
  using passive for the rest of the fetch attempts).
- Cleaned-up debugging messages.
====

  You can read the debmirror documentation by downloading the script and 
typing:
  perldoc ./debmirror


run
===
  I have a master script call "run" which calls debmirror in ways it
needs to be called to actually do the mirror process.  This is where 
the configuration goes, like which upstream sites you want to use for
mirroring from and which release of Debian you want.

  I should really explain how to use netselect or netselect-apt to 
find the best mirror to use based on your network location.  
That is not happening now.


2 - Run the mirror script:
==========================

  After you get the "run" script to pull from the mirrors you want...
run it:
===
/home/debian/bin/run
===

  It will take a while... just make sure it's working correctly.
If there are problems fix them before going further.

  If you have --debug or --progress enabled you will probably want to 
redirect things to a file so that you can stop to look at what is being 
printed.

To install a cron entry to automatically do this every morning at 2am:
===
cd /home/debian
wget http://simons-clan.com/~msimons/debian/mirror/crontab
crontab -u debian crontab
===

  From then on I recommend having cron do the updates, output from runs 
would be sent to your "debian" user via email which should be easy to read.


3 - Setup a web server to serve your mirror:
============================================

If you don't already have a apache installed...
===
apt-get -y install apache
===

Add the following to /etc/apache/httpd.conf:
===
# Allow access to Debian mirror tree...
Alias /debian/ /home/debian/mirror/
<Location /debian>
  Options Indexes FollowSymLinks MultiViews
</Location>
===

Restart apache after the config change:
===
/etc/init.d/apache restart
===

  At this point you have a mirror that is fully functional... to 
use it put lines like in your /etc/apt/sources.list:
====
deb file:///home/debian/mirror woody main contrib non-free
deb file:///home/debian/mirror/non-US woody/non-US main contrib non-free
deb file:///home/debian/mirror/debian-security woody/updates main contrib non-free
===

  The rest of the instructions tell you what needs to be done to install
off the mirror.


4 - Manually pull down disk images:
===================================

  In order to be able to do a net install of the base system from your
machine you will need some files which are not part of the Packages.gz
file which gets pulled by the debmirror script.  For now I manually
pull the bf2.4 images.  I've put this into a little script form for
this howto (which will also make my job much easier next time the disk
sets are updated).

To pull down the latest disk set from the mirror:
===
m=/home/debian/mirror/dists/woody/main/disks-i386
mkdir -p $m
cd $m
b=3.0.23-2002-05-21
c=3.0.23-2002-07-18
for x in $b/bf2.4/ $b/images-1.44/bf2.4/ base-images-$c/basedebs.tar; do 
  wget --recursive --continue --passive-ftp -nH --cut-dirs 5 \
    ftp://ftp.us.debian.org/debian/dists/woody/main/disks-i386/$x
done
ln -s $b current
ln -s base-images-$c base-images-current
chown -R debian:debian $m
===

  Be warned that is a manual kludge around the fact that the disk sets
are not recorded in a Packages.gz format, these files will not
automatically be downloaded when a mirror is done, and the paths below
are hard coded for the current (as of today) version.

  ./3.0.23-2002-05-21/bf2.4/drivers.tgz can be copied onto the hard
drive of your target machine to allow you to select "from hard disk" or
"from mounted filesystem" as a source for the base kernel and modules
in the install menu.
  Otherwise you will need the driver-*.bin disks on floppy or CD to 
talk to the network to begin a network install.

  If you have ./base-images-3.0.23-2002-07-18/basedebs.tar on the 
target hard drive, you won't even need to talk to your mirror until
the base system boots up and you are in stage two of the debian install 
process.



5 - Setup a dhcp server to allocated IP addresses:
==================================================

  If you don't already have a DHCP server on the network you will be
installing over you should.

  If you already have a DHCP server do not do the following steps.

Install the DHCP server
===
apt-get -y install dhcp
===

Put the following in /etc/dhcpd.conf
===
option domain-name-servers 192.168.123.1;

option subnet-mask 255.255.255.0;
default-lease-time 600;
max-lease-time 7200;

subnet 192.168.123.0 netmask 255.255.255.0 {
  range 192.168.123.16 192.168.123.254;
  option broadcast-address 192.168.123.255;
  option routers 192.168.123.1;
}
===

Restart dhcp after the config change:
===
/etc/init.d/dhcp restart
===

  note if you don't run a DNS server on your machine you will need to
change the domain-name-servers line to be a coma separated list of 
correct servers.

  note if your mirror machine is not also the gateway you will need to
change the 'option routers'.

  The configuration above leaves some room (1-15) for static manually
configured IP addresses.

  I will not cover setting up DNS or setting up your machine to be a 
masquerading firewall/gateway in this howto... but those are what the
demo box does at the installfests.
  I would suggest if you get the DNS server setup you may want to 
fake advertise your mirror as 'http.us.debian.org' so that you
don't need to do the following steps.


6 - Store a local sources.list file:
====================================

  On the mirror I keep a sources.list file prepared so that at install
time when questions about what apt sources you want to use are asked
one is handy.

===
cd /var/www
wget http://simons-clan.com/~msimons/debian/mirror/sources.list
chown debian:debain sources.list
===

  That file references the local mirror...

  When the second stage installation process asks you for what sources 
you want to use to install the system, select 'edit sources.list by hand'.

Switch to window two (alt-F2), login as root, type
===
cd /etc/apt
telnet 192.168.123.1 80 > sources.list
GET /sources.list
vi sources.list
===

  Delete the first three lines of the file (which are the telnet command
printing garbage.  Save the file.  Switch back to window 1 (alt-F1), 
and exit without saving changes... (^X) you are good to go, the 
install system will begin pulling package lists from your mirror.




TODO:
- move my cvs archive of this script somewhere public
- attempt a sync with the debian package
- teach something how to pull disk sets, automatically.
- explain netselect-apt
_______________________________________________
vox-tech mailing list
vox-tech@lists.lugod.org
http://lists.lugod.org/mailman/listinfo/vox-tech



LinkedIn
LUGOD Group on LinkedIn
Sign up for LUGOD event announcements
Your email address:
facebook
LUGOD Group on Facebook
'Like' LUGOD on Facebook:

Hosting provided by:
Sunset Systems
Sunset Systems offers preconfigured Linux systems, remote system administration and custom software development.

LUGOD: Linux Users' Group of Davis
PO Box 2082, Davis, CA 95617
Contact Us

LUGOD is a 501(c)7 non-profit organization
based in Davis, California
and serving the Sacramento area.
"Linux" is a trademark of Linus Torvalds.

Sponsored in part by:
Sunset Systems
Who graciously hosts our website & mailing lists!