using local mirrors without any internet access

[splinter]

Hi,

I've a (simplified) setup of following core components:

INTERNET------SECURITY_APPLIANCE------SERVER------INTERNAL_NETWORK---[many many local desktops, servers, virtual machines, vnets, ...]

The SERVER is providing a complete local mirror of any Debian repository required by any of the systems in INTERNAL_NETWORK, including the update and security repository.
The mirroring of the repositories works flawless, usage of this mirror works flawless for installing and updating any software.

Currently there's still one big problem left:
Whenever apt-get update is executed, systems in INTERNAL_NETWORK try to access security-cdn.debian.org which is not configured by any means.
The sources.list contains only entries to targeting the local SERVER, including Debian main, updates and security, so no external repository gets mentioned in sources.list, nor any config files in sources.list.d.
When completely disabling any internet access, so no access from INTERNAL_NETWORK to any external source besides SERVER is possible, it's no longer possible to run apt-get update without error messages, as apt-get can not access the nowhere configured repositories of security-cdn.debian.org due to denied network routing.

So I'm currently searching for another solution than simply creating a local DNS aliasing infrastructure to force solution of security-cdn.debian.org to SERVER, as this would have negative effects on configuration transparency in the network as a whole.

The reason it's important to be able to run apt-get update successfully without any internet access available is simply due to requirements regarding reproducibility and availability.
Explanation for reproducibility: Simply restore a mirror snapshot from backup and produce exact same results.
Explanation for availability: ISP may have downtimes or routing issues, but this shall not impact any working.

It would be nice if anyone could help me finding a solution to this problem, as I'm totally out of ideas where to search for any solutions.
I've searched man pages, the debian.org website, google search results, but I've not been able to find any solution other than creating intransparent DNS aliases for the internal network, which will most likely fire back one day when trying to investigate other issues which might arise.

Kind regards

[p.H]

http://security.debian.org/* redirects to http://security-cdn.debian.org/* so there may be an issue with your mirroring.

[splinter]

The official mirrors get fully mirrored to the SERVER, which is the only system which shall have access to the internet and be reachable by the INTERNAL_NETWORK.

The problem is limited to the systems in the INTERNAL_NETWORK which have their /etc/apt/sources.list configured to only use SERVER as their only Debian repository server.

So they should not at all even try to resolve DNS for security.debian.org nor security-cdn.debian.org , as those are not configured in any /etc/apt/sources.list nor /etc/apt/sources.list.d/* files.

So as I understand, the apt-get update command seems to have some internally hard coded mechanisms which try to access an official internet/cloud(?) located debian security repository, which is not configured in any sources.list files I could find.

Usually I would say such a behaviour is a good default for most users and use cases, but should be somehow possible to be deactivated.
Sadly, I did not yet find any option to disable this behaviour by now, and my colleagues are no help for this specific kind of problems, either.

Best regards

[p.H]

So they should not at all even try to resolve DNS for security.debian.org nor security-cdn.debian.org , as those are not configured in any /etc/apt/sources.list nor /etc/apt/sources.list.d/* files.

Unless the configured "mirror" redirects to security-cdn.debian.org, as security.debian.org does.
How does your mirroring process handle the fact that security.debian.org is not a real mirror but just a redirector to security-cdn.debian.org ?

So as I understand, the apt-get update command seems to have some internally hard coded mechanisms which try to access an official internet/cloud(?) located debian security repository, which is not configured in any sources.list files I could find.

Not AFAIK.

[splinter]

The mirroring process uses the ftpmirror scripts package, running on SERVER, targeting the TLD related official Debian repository (ftp.TLD.debian.org), running the mirror task according to specified rules once a day only per segment (debian, debian-updates and debian-security) as this is sufficient for systems not facing any internet access.
(SERVER itself updates itself from official online repositories, as do other systems which are in DMZ which is not included in the simplified environment I've shown.)

On SERVER the downloaded Debian repository gets published to INTERNAL_NETWORK using apache httpd, having many vhosts.

The "clients" (any Debian system residing within INTERNAL_NETWORK, not having any access to the internet by default) use the following /etc/apt/sources.list:

[code]
# deb http://debian.intern.DOMAIN.TLD/debian/ stretch main

deb http://debian.intern.DOMAIN.TLD/debian/ stretch main contrib non-free
deb-src http://debian.intern.DOMAIN.TLD/debian/ stretch main contrib non-free

deb http://security.intern.DOMAIN.TLD/debian-security stretch/updates main contrib non-free
deb-src http://security.intern.DOMAIN.TLD/debian-security stretch/updates main contrib non-free

# stretch-updates, previously known as 'volatile'
deb http://debian.intern.DOMAIN.TLD/debian/ stretch-updates main contrib non-free
deb-src http://debian.intern.DOMAIN.TLD/debian/ stretch-updates main contrib non-free
[/code]

EDIT: debian.intern.DOMAIN.TLD as well as security.intern.DOMAIN.TLD are aliases to SERVER /EDIT

This way SERVER is the only system having access to the internet and having configured any entry containing something like *.debian.org, therefore none of the other systems should ever try to resolve something like security.debian.org nor security-cdn.debian.org, at least that's my expectation.

As said, getting everything from SERVER works flawless, only apt-get update requires internet access nowhere specified security-cdn.debian.org, which is quite frustrating by now.

Kind regards

[Dai_trying]

To troubleshoot this you could use inxi -r to get a list of repositories along with the file that contains the information, this might show a file in /etc/apt/sources.list.d/ that is directing to the real Debian server instead of your INTERNAL_NETWORK. You can also get a list from apt policy but it doesn't give the exact location of the file you will need to edit.
If you have no repositories outside of you INTERNAL_NETWORK then apt should not go looking for them...

[splinter]

Thanks for the hint to inxi, that's new to me.
I've installed and executed inxi -r, unfortunately it lists only the desired entries, and gives no hint to why apt-get update tries to access the official repositories.

[splinter]

Problem solved.

The access to official repositories was caused by a misconfigured vhost on SERVER, most likely a dumb copy paste error from an older dmz migration.

Background:
The accessed vhost had rewrite rules to forward 404 towards official repositories, which returned 302 instead of 404. This forwarding has been removed, now all works as expected. This shows how strict firewall rules aid in discovering human errors.

Thanks for your replies!!!

[Dai_trying]

Great news, glad you got it sorted.