Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

How to clone debian repository i386/amd64

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
pedropt
Posts: 15
Joined: 2016-10-12 13:30

How to clone debian repository i386/amd64

#1 Post by pedropt »

There are some scripts available to clone debian repository , but if you are an network administrator in a company with 20 to 50 local stations running debian , then you should think in creating a dedicated server so the stations can update their OS on local network instead wasting internet bandwidth .
I use wget to get only i386 and amd64 architectures .
To get all the data without downloading unusefull packages from other debian distributions like armf,armel etc ....
apply this wget command :

To clone debian repository on "/debianrepo/" folder :

Code: Select all

#!/bin/bash
wget -P /debianrepo/ -r -np -nH --timestamping  --reject "*sparc.gz","*s390x.gz","*s390.gz","*powepc.gz","*mipsel.gz","*mips.gz","*kfreebsd-i386.gz","*kfreebsd-amd64.gz","*ia64.gz","*ia64.gz","*armhf.gz","*armel.gz","*ppc64el.gz","*arm64.gz","*powerpc.gz","*mips64el.gz","*hurd-i386.gz","*sparc.deb","*s390x.deb","*s390.deb","*powepc.deb","*mipsel.deb","*mips.deb","*kfreebscd-i386.deb","*kfreebsd-amd64.deb","*ia64.deb","*ia64.deb","*armhf.deb","*armel.deb","*ppc64el.deb","*arm64.deb","*powerpc.deb","*mips64el.deb","*hurd-i386.deb","*sparc.udeb","*s390x.udeb","*s390.udeb","*powepc.udeb","*mipsel.udeb","*mips.udeb","*kfreebsd-i386.udeb","*kfreebsd-amd64.udeb","*ia64.udeb","*ia64.udeb","*armhf.udeb","*armel.udeb","*ppc64el.udeb","*arm64.udeb","*powerpc.udeb","*mips64el.udeb","*hurd-i386.udeb","index.html","*.html","*.html*" ftp.debian.org/debian/
To update the repository :

Code: Select all

#!/bin/bash
wget -P /debianrepo/ -r -np -nH --timestamping  -N --reject "*sparc.gz","*s390x.gz","*s390.gz","*powepc.gz","*mipsel.gz","*mips.gz","*kfreebsd-i386.gz","*kfreebsd-amd64.gz","*ia64.gz","*ia64.gz","*armhf.gz","*armel.gz","*ppc64el.gz","*arm64.gz","*powerpc.gz","*mips64el.gz","*hurd-i386.gz","*sparc.deb","*s390x.deb","*s390.deb","*powepc.deb","*mipsel.deb","*mips.deb","*kfreebscd-i386.deb","*kfreebsd-amd64.deb","*ia64.deb","*ia64.deb","*armhf.deb","*armel.deb","*ppc64el.deb","*arm64.deb","*powerpc.deb","*mips64el.deb","*hurd-i386.deb","*sparc.udeb","*s390x.udeb","*s390.udeb","*powepc.udeb","*mipsel.udeb","*mips.udeb","*kfreebsd-i386.udeb","*kfreebsd-amd64.udeb","*ia64.udeb","*ia64.udeb","*armhf.udeb","*armel.udeb","*ppc64el.udeb","*arm64.udeb","*powerpc.udeb","*mips64el.udeb","*hurd-i386.udeb","index.html","*.html","*.html*" ftp.debian.org/debian/
and only new files will be downloaded .

The ftp.debian.org can be replaced to one debian mirror instead the original debian repository to not overload the server .
The name extensions under "" are the ones that will be ignored by wget , so if you want to clone all architectures from debian repo then use this code :

Code: Select all

#!/bin/bash
wget -P /debianrepo/ -r -np -nH --timestamping  --reject "*.html","*.html*" ftp.debian.org/debian/
An advise : Be prepared to have at least 2TB HDD to get all data from all architectures , and be prepared to wait for at least 3 days to download everything 24H/d , debian repository have a bandwidth limit for each connection +- 3M/s .

You can use rsync instead the ftp if you want .

Last notes
After the clone of the repository , an update will be fast , it will take +- 2 hours to be done because the most of the files are already in your HDD .
You have to setup your webserver to that directory on where you downloaded the repository in your HDD and allow file list on server config .

In case you want to clone only 1 specific repository then follow these instructions using debmirror:
http://lgallardo.com/en/2012/12/06/como ... debmirror/

Post Reply