How to clone debian repository i386/amd64

Share your own howto's etc. Not for support questions!

How to clone debian repository i386/amd64

Postby pedropt » 2016-11-10 00:32

There are some scripts available to clone debian repository , but if you are an network administrator in a company with 20 to 50 local stations running debian , then you should think in creating a dedicated server so the stations can update their OS on local network instead wasting internet bandwidth .
I use wget to get only i386 and amd64 architectures .
To get all the data without downloading unusefull packages from other debian distributions like armf,armel etc ....
apply this wget command :

To clone debian repository on "/debianrepo/" folder :
Code: Select all
#!/bin/bash
wget -P /debianrepo/ -r -np -nH --timestamping  --reject "*sparc.gz","*s390x.gz","*s390.gz","*powepc.gz","*mipsel.gz","*mips.gz","*kfreebsd-i386.gz","*kfreebsd-amd64.gz","*ia64.gz","*ia64.gz","*armhf.gz","*armel.gz","*ppc64el.gz","*arm64.gz","*powerpc.gz","*mips64el.gz","*hurd-i386.gz","*sparc.deb","*s390x.deb","*s390.deb","*powepc.deb","*mipsel.deb","*mips.deb","*kfreebscd-i386.deb","*kfreebsd-amd64.deb","*ia64.deb","*ia64.deb","*armhf.deb","*armel.deb","*ppc64el.deb","*arm64.deb","*powerpc.deb","*mips64el.deb","*hurd-i386.deb","*sparc.udeb","*s390x.udeb","*s390.udeb","*powepc.udeb","*mipsel.udeb","*mips.udeb","*kfreebsd-i386.udeb","*kfreebsd-amd64.udeb","*ia64.udeb","*ia64.udeb","*armhf.udeb","*armel.udeb","*ppc64el.udeb","*arm64.udeb","*powerpc.udeb","*mips64el.udeb","*hurd-i386.udeb","index.html","*.html","*.html*" ftp.debian.org/debian/


To update the repository :
Code: Select all
#!/bin/bash
wget -P /debianrepo/ -r -np -nH --timestamping  -N --reject "*sparc.gz","*s390x.gz","*s390.gz","*powepc.gz","*mipsel.gz","*mips.gz","*kfreebsd-i386.gz","*kfreebsd-amd64.gz","*ia64.gz","*ia64.gz","*armhf.gz","*armel.gz","*ppc64el.gz","*arm64.gz","*powerpc.gz","*mips64el.gz","*hurd-i386.gz","*sparc.deb","*s390x.deb","*s390.deb","*powepc.deb","*mipsel.deb","*mips.deb","*kfreebscd-i386.deb","*kfreebsd-amd64.deb","*ia64.deb","*ia64.deb","*armhf.deb","*armel.deb","*ppc64el.deb","*arm64.deb","*powerpc.deb","*mips64el.deb","*hurd-i386.deb","*sparc.udeb","*s390x.udeb","*s390.udeb","*powepc.udeb","*mipsel.udeb","*mips.udeb","*kfreebsd-i386.udeb","*kfreebsd-amd64.udeb","*ia64.udeb","*ia64.udeb","*armhf.udeb","*armel.udeb","*ppc64el.udeb","*arm64.udeb","*powerpc.udeb","*mips64el.udeb","*hurd-i386.udeb","index.html","*.html","*.html*" ftp.debian.org/debian/


and only new files will be downloaded .

The ftp.debian.org can be replaced to one debian mirror instead the original debian repository to not overload the server .
The name extensions under "" are the ones that will be ignored by wget , so if you want to clone all architectures from debian repo then use this code :

Code: Select all
#!/bin/bash
wget -P /debianrepo/ -r -np -nH --timestamping  --reject "*.html","*.html*" ftp.debian.org/debian/


An advise : Be prepared to have at least 2TB HDD to get all data from all architectures , and be prepared to wait for at least 3 days to download everything 24H/d , debian repository have a bandwidth limit for each connection +- 3M/s .

You can use rsync instead the ftp if you want .

Last notes
After the clone of the repository , an update will be fast , it will take +- 2 hours to be done because the most of the files are already in your HDD .
You have to setup your webserver to that directory on where you downloaded the repository in your HDD and allow file list on server config .

In case you want to clone only 1 specific repository then follow these instructions using debmirror:
http://lgallardo.com/en/2012/12/06/como-crear-un-mirror-de-debian-y-ubuntu-con-debmirror/
pedropt
 
Posts: 15
Joined: 2016-10-12 13:30

Return to Docs, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 3 guests

fashionable