better/best way to set up apt mirror? solved

[id10t]

Been using apt-mirror to mirror some Ubuntu and Debian repositories - the official ones, for amd64 and i386

However, recently the Jessie netinstall disc is failing to verify my mirror and refuses to use it - it adds entries to the /etc/apt/sources.list but they are all commented out.

From what my googling has told me, part of the issue is hash sum mismatches on various index files that aren't "there" for Jessie and so apt-mirror creates them.

After deleting a bunch of stuff, and re-running apt-mirror several times, the error persists, or moves from InRelease to Translastions-en to Packages to ...

Anyway, this has me thinking there must be a better more up to date way of mirroring. But what method?

Suggestions on either method, or method and a helpful link to a good howto are appreciated.

[edbarx]

It depends what you mean by mirror? If what you need is to use another mirror for your updates and any packages that you might need to install, you only need to use a text editor like nano to edit /etc/apt/sources.list. To add to the nauseating repetition that exists on these fora: "do not mix repositories from different distributions and from different Debian major versions".

You can edit text files as root with nano. To save, first hit Ctrl + o, then press Enter and then Ctrl + x to exit.

[id10t]

No, I mean actually set up a mirror. I teach a class and we use Debian several times a semester, and we need to do lots of netinstalls. So instead of hammering both our "to the world" connection and to avoid having 25 extra hits on the various public mirrors, I've had a local apt archive setup for years, usign apt-mirror. But as per my original post, there are now issues with some of the index files and hash sum mismatches, which I've narrowed down to not being in Jessie so apt-mirror creates them...

I've decided to work on using ftpsync - hopefully I'll be able to keep the 750gb of archive I already have and not have to re-download it all....

[thatguychuck]

I hit the same problem and was unable to find a solution. That doesn't mean it doesn't exist, just that I was unable to find it.

I switched to the recommended ftpsync scripts and was able to move my existing repository into the new one.

Keep in mind, there isn't a way to exclude releases, only architectures, with ftpsync. So in addition to stable, testing, and unstable, you also get old stable, old old stable, and a bunch of support files.

Edit: Forgot to add the github issues links, others have had the same. https://github.com/apt-mirror/apt-mirror/issues/26,https://github.com/marblenix/fix-apt-mi ... te-zips.sh

[id10t]

Yeah, I dont' have enough space for *all* of debian, even with only x86 adn amd64 arch. So now I'm trying with debmirror but still getting the error, but only for jessie-updates. Now I'm deleting /mirror/debian/dists/jessie-updates and re-synching ... hopefully by starting fresh on the indexes and such it will fix itself...

[cpoakes]

I don't run a mirror, I use a squid proxy server.. With a mirror, you need to download the entire archive and provide syncronization. If you use a proxy, only the actual files you need are saved to disk and are validated everytime there is a request, updating the cache only when changes (or uncached files) are noted. And as a general solution, it would avoid the mirror config problems you've encountered.

In my case, I have an old NAS (no gigabit ethernet) set up as a dedicated proxy, configured in apt.conf (not as a general proxy for Internet traffic). It is not fast, but it is way faster than my internet connection for repeat/additional installs. And no configuration change from squeeze to wheezy to jessie... .

[debianuser3325]

https://www.debian.org/mirror/ftpmirror

I saw an option on x2go's site to mirror their repo and tried for a few seconds with their suggested command, so my recommendation to start with is below:

Code: Select all

rsync -avP ftp.us.debian.org::debian ~/tmp

I've never done mirroring, but can recommend looking at that source and using their suggested way of doing it.

[edbarx]

Did you consult this resource: https://www.debian.org/mirror/ftpmirror

[id10t]

Did you consult this resource: https://www.debian.org/mirror/ftpmirror

Yes, but you can only exclude architectures. I only want to mirror stable and oldstable for i386 and amd64. Don't have the disk space for a ton of other stuff.

I did however get it working with out errors in netinstall. rm -rf /my_mirror/debian/dists/jessie-backports && debmirror-debian.sh (deleted the /debian/dists portion and resynched)

[id10t]

So a summary for those who may follow ...

Debian 8 (Jessie) changed the way the various Release, Index, etc. files under /debian/dists/releasepath (/debian/dists/jessie-updates in my case) but apt-mirror is still looking for the old way. When apt-mirror doesn't find the file, it uses other existing files to create its own version, which causes hash sum mismatch errors. So, to keep a netinstall image happy during the installation process, changing your mirror method to debmirror may be worth while.

Converting from apt-mirror to debmirror is easy. You do not need to re-download your /debian/pool of .deb files

Note that if you are using apt-mirror for multiple distributions (debian, ubuntu, mint) that you will need separate scripts for each one

The various howtos out there cover most of setting up debmirror from scratch but there are a few gotchas, and a few noteworthy things if you are converting from apt-mirror.

• While well covered in the howtos, BE SURE TO UPDATE YOUR APT GPG KEYS!! Nothing else has a whelk's chance in a supernova of working if you don't do this
• In the debmirror script, be sure to use the --slow-cpu option
• You don't need to re-download your /debian/pool full of deb files
• You do need to delete the directories under /debian/dists - these are the actual indexes of available software, versions, etc Even though I sync'd successfully while keeping the /debian/dists contents around, the netinstaller still wouldn't validate and use my mirror

Here is my working mirror-debian.sh script which calls debmirror at the end

Code: Select all

#!/bin/sh

# Architecture (i386, ppc, amd64, etc.)
# if you don't want -src packages that is handled 
# in the actual debmirror command line arguments
# with --no-source
arch=i386,amd64

# Section (main,contrib,non-free)
section=main,contrib,non-free

# Release of the system (wheezy,jessie,sketch, etc)
# use the real release names here, and create the stable,testing, etc. labels as symbolic links.
# this way when a release happens you don't need to re-download a ton of stuff you already have, just change the links
release=jessie,jessie-backports,jessie-proposed-updates,jessie-updates,wheezy,wheezy-backports,wheezy-proposed-updates,wheezy-updates

# Source server name, minus the protocol and the path at the end
# should be one of the HA mirrors
server=ftp.us.debian.org

# Web path for the above mirror source server, top level of /debian where /debian/dists and /debian/pool can be found
inPath=/debian

# Protocol to use for transfer (http, ftp, hftp, rsync)
proto=http

# Directory to store the mirror in
# this should show up as the /debian directory
# when your clients access the mirror 
outPath=/storage/mirrors/mirror/ftp.us.debian.org/debian

# Start script

debmirror -a $arch \
--no-source \
--slow-cpu \
--md5sums \
--progress \
--passive \
--verbose \
-s $section \
-h $server \
-d $release \
-r $inPath \
-e $proto \
$outPath