Odd Apache error - like pollution between two virtual hosts

Kernels & Hardware, configuring network, installing services

Odd Apache error - like pollution between two virtual hosts

Postby drkirkby » 2018-07-02 21:53

I have about 6 websites hosted on a virtual private server (VPS) running Debian 9. The problem is Google seems to be crawling site A for things that never have been on site A, but are in site B.

One website
https://www.g8wrb.org/

has a lot of valve data sheets on it. For example.

https://www.g8wrb.org/data/Eimac/4CX10000D.pdf

They have never been on this website, which is my company website.

https://www.kirkbymicrowave.co.uk/

yet when I look in the server logs for https://www.kirkbymicrowave.co.uk/, I see Google has been crawling over it, looking for files that should never be there, then returning loads of 404 errors as it cant find the files!

Taking one line from the access log of the second site

access-kirkbymicrowave.co.uk-port-443.log.4:66.249.64.150 - - [28/Jun/2018:22:45:00 +0000] "GET /data/Eimac/4CX10000D.pdf HTTP/1.1" 302 6161 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

So why has Google gone looking for /data/Eimac/4CX10000D.pdf on the https://www.kirkbymicrowave.co.uk/ site, when it has never been there? If instead it looked on https://www.g8wrb.org/, it would indeed find the file exists.

It seems thereś some sort or "pollution" for want of a better expression between the two virtual hosts. Some of the re-redirects you see below are that each site now has SSL certifates, and standard HTTP requests get sent to the SSL version of the site.

Also odd, is this attempt to access a file, this time by Bing.

access-kirkbymicrowave.co.uk.log.5:157.55.39.111 - - [16/Jun/2018:14:15:25 +0000] "GET /data/svetlana/pdf/4cx1600b.pdf HTTP/1.1" 404 520 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Note that the file it is looking for /data/svetlana/pdf/4cx1600b.pdf is all in lower case. Not only is that looking on the wrong site, but the wrong case too.

Here's a bit more of the server logs, if aiids anyone. I can post the virtual host configuration files if need be.


access-kirkbymicrowave.co.uk.log.5:207.46.13.185 - - [16/Jun/2018:08:36:55 +0000] "GET /data/svetlana/pdf/4cx5000a_8170.pdf HTTP/1.1" 404 525 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
access-kirkbymicrowave.co.uk.log.5:207.46.13.185 - - [16/Jun/2018:08:37:50 +0000] "GET /data/svetlana/pdf/4cx1500a.pdf HTTP/1.1" 404 520 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
access-kirkbymicrowave.co.uk.log.5:207.46.13.34 - - [16/Jun/2018:12:17:16 +0000] "GET /data/svetlana/pdf/4cx7500a.pdf HTTP/1.1" 404 520 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
access-kirkbymicrowave.co.uk.log.5:157.55.39.70 - - [16/Jun/2018:13:38:36 +0000] "GET /data/svetlana/pdf/4cx800a.pdf HTTP/1.1" 404 519 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
access-kirkbymicrowave.co.uk.log.5:157.55.39.111 - - [16/Jun/2018:14:15:25 +0000] "GET /data/svetlana/pdf/4cx1600b.pdf HTTP/1.1" 404 520 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
access-kirkbymicrowave.co.uk.log.5:157.55.39.111 - - [16/Jun/2018:16:09:21 +0000] "GET /data/svetlana/pdf/4cx400a.pdf HTTP/1.1" 404 519 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
access-kirkbymicrowave.co.uk.log.6:157.55.39.226 - - [16/Jun/2018:05:27:39 +0000] "GET /data/eimac/4cx15000a.pdf HTTP/1.1" 404 514 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
root@localhost:/var/log/apache2# grep 4CX *kir*
access-kirkbymicrowave.co.uk.log.5:66.249.66.70 - - [16/Jun/2018:06:50:59 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX3000A.pdf HTTP/1.1" 404 573 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:08:05:51 +0000] "GET /complete-list.php/yagi/data/Penta/data/RCA/data/EEV/4CX5000A.pdf HTTP/1.1" 404 554 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:08:31:32 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX1000A.pdf HTTP/1.1" 404 573 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:09:46:58 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Penta/4CX7500A.pdf HTTP/1.1" 404 573 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:10:01:28 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/EEV/4CX5000A.pdf HTTP/1.1" 404 571 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:11:09:29 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX250_K_and_M.pdf HTTP/1.1" 404 579 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:12:21:20 +0000] "GET /complete-list.php/pentodes/data/RCA/data/Telefunken/data/Raytheon/4CX300A.pdf HTTP/1.1" 404 567 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:12:54:28 +0000] "GET /complete-list.php/pentodes/data/HP/data/Penta/data/Eimac/4CX35000C.pdf HTTP/1.1" 404 560 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.66 - - [16/Jun/2018:12:58:09 +0000] "GET /complete-list.php/pentodes/data/RCA/data/National_Electronics/data/Penta/4CX7500A.pdf HTTP/1.1" 404 575 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.66 - - [16/Jun/2018:13:09:11 +0000] "GET /complete-list.php/pentodes/data/HP/data/Russian/data/Eimac/4CX25000C.pdf HTTP/1.1" 404 562 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.66 - - [16/Jun/2018:13:22:49 +0000] "GET /complete-list.php/pentodes/data/HP/data/GEC/data/Tesla/4CX250M.pdf HTTP/1.1" 404 556 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:14:49:26 +0000] "GET /complete-list.php/help/data/STC/data/Amperex/data/Penta/4CX1500A.pdf HTTP/1.1" 404 558 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:15:43:49 +0000] "GET /complete-list.php/help/data/Econco/data/Siemens/data/Penta/4CX1500A.pdf HTTP/1.1" 404 561 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.66 - - [16/Jun/2018:15:57:50 +0000] "GET /complete-list.php/pentodes/data/HP/data/Machlett_Laboratories/data/Eimac/4CX300Y.pdf HTTP/1.1" 404 574 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.5:66.249.66.68 - - [16/Jun/2018:16:33:36 +0000] "GET /complete-list.php/help/data/Eimac/data/Machlett_Laboratories/data/Penta/4CX1500A.pdf HTTP/1.1" 404 574 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.66 - - [15/Jun/2018:20:24:00 +0000] "GET /complete-list.php/yagi/data/Burle/data/THOMSON-CSF/data/Raytheon/4CX300A.pdf HTTP/1.1" 404 566 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [15/Jun/2018:22:06:50 +0000] "GET /complete-list.php/help/data/Thales/data/THOMSON-CSF/data/Penta/data/Burle/data/RCA/data/STC/styles/screen.css HTTP/1.1" 404 599 "http://g8wrb.org/complete-list.php/help/data/Thales/data/THOMSON-CSF/data/Penta/data/Burle/data/RCA/data/STC/4CX250B.pdf" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36"
access-kirkbymicrowave.co.uk.log.6:66.249.66.68 - - [15/Jun/2018:23:22:58 +0000] "GET /complete-list.php/yc156/data/STC/data/Telefunken/data/Eimac/4CX5000A.pdf HTTP/1.1" 404 562 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.66 - - [15/Jun/2018:23:38:28 +0000] "GET /complete-list.php/yagi/data/Amperex/data/Tesla/data/Eimac/4CX25000A.pdf HTTP/1.1" 404 561 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [16/Jun/2018:00:18:03 +0000] "GET /complete-list.php/help/data/Russian/data/HP/data/Tesla/4CX250M.pdf HTTP/1.1" 404 556 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [16/Jun/2018:01:01:03 +0000] "GET /complete-list.php/yagi/data/Amperex/data/Burle/data/Eimac/4CX1000K.pdf HTTP/1.1" 404 560 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [16/Jun/2018:01:12:18 +0000] "GET /complete-list.php/yagi/data/Amperex/data/Siemens/data/Eimac/4CX350FJ.pdf HTTP/1.1" 404 562 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.68 - - [16/Jun/2018:01:48:17 +0000] "GET /complete-list.php/pentodes/data/RCA/data/STC/data/Raytheon/4CX300A.pdf HTTP/1.1" 404 560 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.68 - - [16/Jun/2018:02:13:49 +0000] "GET /complete-list.php/yagi/data/National_Electronics/data/unknown/data/Eimac/4CX15000R.pdf HTTP/1.1" 404 576 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.68 - - [16/Jun/2018:04:19:29 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX250R.pdf HTTP/1.1" 404 572 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.66 - - [16/Jun/2018:04:37:23 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX10000D.pdf HTTP/1.1" 404 574 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.68 - - [16/Jun/2018:04:44:17 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX30000G.pdf HTTP/1.1" 404 574 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [16/Jun/2018:05:13:30 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Penta/4CX1500A.pdf HTTP/1.1" 404 573 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.66 - - [16/Jun/2018:05:16:57 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX1500A.pdf HTTP/1.1" 404 573 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [16/Jun/2018:05:42:25 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Tesla/4CX250M.pdf HTTP/1.1" 404 572 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.66 - - [16/Jun/2018:06:00:27 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX300A.pdf HTTP/1.1" 404 572 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk.log.6:66.249.66.70 - - [16/Jun/2018:06:22:27 +0000] "GET /complete-list.php/thanks/data/HP/data/Machlett_Laboratories/data/Eimac/4CX5000R.pdf HTTP/1.1" 404 573 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk-port-443.log.4:66.249.64.154 - - [28/Jun/2018:22:11:07 +0000] "GET /data/Svetlana/pdf/4CX800A.pdf HTTP/1.1" 302 6171 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk-port-443.log.4:66.249.64.152 - - [28/Jun/2018:22:30:40 +0000] "GET /data///Svetlana/ceramic/4CX250BC_pg4.gif HTTP/1.1" 302 6193 "-" "Googlebot-Image/1.0"
access-kirkbymicrowave.co.uk-port-443.log.4:66.249.64.150 - - [28/Jun/2018:22:45:00 +0000] "GET /data/Eimac/4CX10000D.pdf HTTP/1.1" 302 6161 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk-SSL.log.4:66.249.64.94 - - [28/Jun/2018:22:11:08 +0000] "GET /data/Svetlana/pdf/4CX800A.pdf HTTP/1.1" 404 6325 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access-kirkbymicrowave.co.uk-SSL.log.4:66.249.64.67 - - [28/Jun/2018:22:30:40 +0000] "GET /data///Svetlana/ceramic/4CX250BC_pg4.gif HTTP/1.1" 404 6325 "-" "Googlebot-Image/1.0"
access-kirkbymicrowave.co.uk-SSL.log.4:66.249.64.67 - - [28/Jun/2018:22:45:01 +0000] "GET /data/Eimac/4CX10000D.pdf HTTP/1.1" 404 6325 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
error-kirkbymicrowave.co.uk.log.6:[Fri Jun 15 22:06:50.150272 2018] [:error] [pid 12259] [client 66.249.66.70:54112] script '/var/www/html/kirbymicrowave.co.uk/complete-list.php' not found or unable to stat, referer: http://g8wrb.org/complete-list.php/help ... CX250B.pdf
drkirkby
 
Posts: 6
Joined: 2018-06-17 23:24

Re: Odd Apache error - like pollution between two virtual ho

Postby drkirkby » 2018-07-02 22:20

To add something else to this.

If one to incorrectly type the Kikkby Microwave website, using two w's, rather than the usual 3, so

http://ww.kirkbymicrowave.co.uk/

it gets directed to

https://www.dhars.org.uk/

which is my amateur radio club, hosted on the same server. But I don't understand why that redirection is taking place. There's nothing configured on the server to redirect this.

Is there some sort of catch-all, such that if no virtual hosts match, the webserver servers files from some random virtual host? Or maybe its not random, but chosen in some way I don't understand.

Also, odd, is this one, Again, a mis-typed domain

https://ww.steam-boilers.co.uk/

with two w's, rather than the usual 3. That gives a security error

ww.steam-boilers.co.uk uses an invalid security certificate.
The certificate is only valid for dhars.org.uk
Error code: SSL_ERROR_BAD_CERT_DOMAIN

So why does a mis-typed domain, try to use another virtual host?

Dave
drkirkby
 
Posts: 6
Joined: 2018-06-17 23:24


Return to System configuration

Who is online

Users browsing this forum: No registered users and 10 guests

fashionable