Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Adblocking with /etc/hosts

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Adblocking with /etc/hosts

#1 Post by Head_on_a_Stick »

/etc/hosts can be used to block unwanted sites system-wide, see https://en.wikipedia.org/wiki/Hosts_(file)

Configuration of this file can be tedious and time-consuming:
http://www.makeuseof.com/tag/modify-man ... ile-linux/

Fortunately, there is a more efficient method available.
:)

Code: Select all

git clone http://git.r-36.net/hosts-gen && cd hosts-gen
sudo make install
sudo cp examples/gethostszero /bin
sudo chmod 775 /bin/gethostszero
sudo /bin/gethostszero
sudo hosts-gen
This will populate /etc/hosts with a list of the most egregious web abusing sites and will block them automatically.

To update the list, use:

Code: Select all

sudo gethostszero
sudo hosts-gen
With thanks to Christopher Lohmann for this solution.

To update the file automatically, a systemd timer could be used.
https://wiki.archlinux.org/index.php/Systemd/Timers

For this, create these two files:

/etc/systemd/system/hostszero.timer

Code: Select all

[Unit]
Description=Update /etc/hosts daily.

[Timer]
OnCalendar=daily
AccuracySec=1h
Persistent=true

[Install]
WantedBy=timers.target
/etc/systemd/system/hostszero.service

Code: Select all

[Unit]
Description=Update /etc/hosts

[Service]
Type=oneshot
ExecStart=/bin/gethostszero
ExecStart=/bin/hosts-gen
Then enable this with:

Code: Select all

sudo systemctl enable hostszero.timer
Disable it with:

Code: Select all

sudo systemctl disable hostszero.timer
Check the status with:

Code: Select all

systemctl status hostszero.timer
Which should look something like this:

Code: Select all

empty@TheLab:~$ systemctl status hostszero.timer
● hostszero.timer - Update /etc/hosts daily.
   Loaded: loaded (/etc/systemd/system/hostszero.timer; enabled)
   Active: active (waiting) since Sun 2016-07-24 18:00:27 BST; 2min 42s ago
empty@TheLab:~$
The original /etc/hosts file is available here:
http://someonewhocares.org/hosts/zero/hosts
deadbang

User avatar
dasein
Posts: 7680
Joined: 2011-03-04 01:06
Location: Terra Incantationum

Re: Adblocking with /etc/hosts

#2 Post by dasein »

Integrating host-file-based blocking like this with a local DNS server like DNSMasq is a killer combination.

Not only does the combination make it easy to protect an arbitrarily large network from a single hosts file, it allows globally "suspect" domains (e.g., doubleclick) to be blocked with a single line.

Thanks for a fine HOWTO, HoaS.

tomazzi
Posts: 730
Joined: 2013-08-02 21:33

Re: Adblocking with /etc/hosts

#3 Post by tomazzi »

...and how this is different from using privoxy?
...or just a noscript?
Odi profanum vulgus

rbern
Posts: 8
Joined: 2011-07-16 11:18

Re: Adblocking with /etc/hosts

#4 Post by rbern »

tomazzi wrote:...and how this is different from using privoxy?
...or just a noscript?
Hi tomazzi,

I once had an idea for this, using another similar app called adsuck (https://github.com/conformal/adsuck), but it was never packaged for debian or ubuntu.

I think privoxy (still) does not support https, but apparently it will https://sourceforge.net/p/ijbswa/support-requests/1667/.

IMO, having noscript in every computer on a network uses more resources (RAM/CPU/bandwidth/browsing speed) than a single router computer filtering all those from the same computers in the network, depending on the size of the network.

Even in a single computer, this could save the RAM that is used by noscript and save all CPU cycles that noscript takes to filter all IP and addresses. More browsing speed with less CPU usage.

This seems interesting to me.

User avatar
edbarx
Posts: 5401
Joined: 2007-07-18 06:19
Location: 35° 50 N, 14 º 35 E
Been thanked: 2 times

Re: Adblocking with /etc/hosts

#5 Post by edbarx »

I created a .deb package for this. The procedure is simple.

The Method (howto):
  1. Download the sources

    Code: Select all

    git clone http://git.r-36.net/hosts-gen && cd hosts-gen
  2. Rename the directory to hosts-gen-0.1.0. This is required by dh_make.
  3. To Debianize the sources issue the next command while in hosts-gen-0.1.0

    Code: Select all

    dh_make --createorig
  4. cd to the debian subdirectory. Create a new file named hosts-gen.install using touch.

    Code: Select all

    cd debian
    touch hosts-gen.install
  5. Open hosts-gen.install in nano or another text editor. Add this line:

    Code: Select all

    examples/gethostszero /bin
  6. cd back to the parent directory hosts-gen-0.1.0
  7. Use dpkg-buildpackage to build a Debian package.
  8. Use dpkg -i package-name to install the package.
EDIT:
To check periodically for updates as instructed earlier in this thread, use the debian/hosts-gen.postinst file to add the required script.
Debian == { > 30, 000 packages }; Debian != systemd
The worst infection of all, is a false sense of security!
It is hard to get away from CLI tools.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: Adblocking with /etc/hosts

#6 Post by Head_on_a_Stick »

@tomazzi: I had never heard of privoxy :oops:

Thanks for posting, that looks very nice -- if I can throw together something cogent, I will post a HowTo for that as well :)

@edbarx: Brilliant, thanks!

I will see if I can package it for my OBS repository with the systemd unit files...
deadbang

User avatar
edbarx
Posts: 5401
Joined: 2007-07-18 06:19
Location: 35° 50 N, 14 º 35 E
Been thanked: 2 times

Re: Adblocking with /etc/hosts

#7 Post by edbarx »

Post package installation script:
The debian/hosts-gen.postinst file can be used to populate the /etc/hosts file with the required entries. Just add the necessary commands.

Post package removal script:
The debian/hosts-gen.postrm file should be used to remove any remaining files generated by the package. It should also contain script to restore modified system files (/etc/hosts) to their original condition.
Debian == { > 30, 000 packages }; Debian != systemd
The worst infection of all, is a false sense of security!
It is hard to get away from CLI tools.

kevinthefixer
Posts: 190
Joined: 2018-05-05 22:30

Re: Adblocking with /etc/hosts

#8 Post by kevinthefixer »

This is very similar to the adblockers used in Android, which I've had very good luck with. So I thought I'd give it a try. However, I didn't get very far as the git repository seems to have vanished. Anyone have an updated URL?


User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: Adblocking with /etc/hosts

#10 Post by Head_on_a_Stick »

deadbang

User avatar
Bloom
df -h | grep > 90TiB
df -h | grep > 90TiB
Posts: 503
Joined: 2017-11-11 12:23
Been thanked: 26 times

Re: Adblocking with /etc/hosts

#11 Post by Bloom »

My solution: first, make a copy of your hosts file.

Code: Select all

sudo cp /etc/hosts /etc/hosts.0
Then, run this script:

Code: Select all

sudo ./newhosts
And the contents of this newhosts script:

Code: Select all

#!/bin/bash
# get MVPS hosts file to block ad sites and merge it with our hosts file
wget -O - http://winhelp2002.mvps.org/hosts.txt | sed 's/[[:space:]]*#.*$//g;' |
	grep -v localhost | tr ' ' '\t' | tr -s '\t' | tr -d '\015' | sort -u >/tmp/hosts0
echo 0.0.0.0 safebrowsing.google.com >>/tmp/hosts0
echo 0.0.0.0 survival24x7.com >>/tmp/hosts0
echo 0.0.0.0 www.survival24x7.com >>/tmp/hosts0
echo 0.0.0.0 hanstrkr.com >>/tmp/hosts0
echo 0.0.0.0 www.hanstrkr.com >>/tmp/hosts0
echo Attention! This should be run as ROOT!
cat /etc/hosts.0 /tmp/hosts0|sudo dd of=/etc/hosts
rm -f /tmp/hosts0

kevinthefixer
Posts: 190
Joined: 2018-05-05 22:30

Re: Adblocking with /etc/hosts

#12 Post by kevinthefixer »

Ran into dependency problems with this one (running Stretch). @Bloom's "newhosts" file worked a trick though. This should work on any system using /etc/hosts, I would think. Outlook mail loads in half the time, no noticeable difference in Gmail (no noticeable ads in Gmail either).

bedtime
Posts: 146
Joined: 2012-12-16 19:34
Has thanked: 1 time
Been thanked: 6 times

Re: Adblocking with /etc/hosts

#13 Post by bedtime »

debiman wrote:i've been using https://github.com/gaenserich/hostsblock for years.
I installed this program, and it renamed my home directory to 'hostsblock'. Anyhow, long story short, I ended up losing a bunch of files whilst uninstalling and getting things back to normal—config files and scripts that took hours and hours to make. :(

Please be careful if you chose to install this app.

milomak
Posts: 2158
Joined: 2009-06-09 22:20
Been thanked: 1 time

Re: Adblocking with /etc/hosts

#14 Post by milomak »

how is this better than say using ublock origin?
Desktop: A320M-A PRO MAX, AMD Ryzen 5 3600, GALAX GeForce RTX™ 2060 Super EX (1-Click OC) - Sid, Win10, Arch Linux, Gentoo, Solus
Laptop: hp 250 G8 i3 11th Gen - Sid
Kodi: AMD Athlon 5150 APU w/Radeon HD 8400 - Sid

User avatar
dilberts_left_nut
Administrator
Administrator
Posts: 5343
Joined: 2009-10-05 07:54
Location: enzed
Has thanked: 12 times
Been thanked: 66 times

Re: Adblocking with /etc/hosts

#15 Post by dilberts_left_nut »

milomak wrote:how is this better than say using ublock origin?
Because you deploy it on your dns relay and it covers every device on your network.
AdrianTM wrote:There's no hacker in my grandma...

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: Adblocking with /etc/hosts

#16 Post by Head_on_a_Stick »

kevinthefixer wrote:
Ran into dependency problems with this one (running Stretch)
You can use the supplied hosts file directly instead.
milomak wrote:how is this better than say using ublock origin?
It blocks the bad addresses for all software and it operates with zero overhead, unlike a plugin.
deadbang

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: Adblocking with /etc/hosts

#17 Post by debiman »

bedtime wrote:
debiman wrote:i've been using https://github.com/gaenserich/hostsblock for years.
I installed this program, and it renamed my home directory to 'hostsblock'. Anyhow, long story short, I ended up losing a bunch of files whilst uninstalling and getting things back to normal—config files and scripts that took hours and hours to make. :(

Please be careful if you chose to install this app.
you are right, and i am very, very sorry.
i guess you were using the install.sh script? I had problems with it, too, but the developer said they fixed it.
may i ask which version / how you installed? maybe i should re-open that issue:
https://github.com/gaenserich/hostsblock/issues/87

fwiw, the AUR installation method has always been reliable, and so is the script once you have it installed properly.

edit:
seems i cannot re-open the issue, i created a new one:
https://github.com/gaenserich/hostsblock/issues/88

bedtime
Posts: 146
Joined: 2012-12-16 19:34
Has thanked: 1 time
Been thanked: 6 times

Re: Adblocking with /etc/hosts

#18 Post by bedtime »

debiman wrote:
bedtime wrote:
debiman wrote:i've been using https://github.com/gaenserich/hostsblock for years.
I installed this program, and it renamed my home directory to 'hostsblock'. Anyhow, long story short, I ended up losing a bunch of files whilst uninstalling and getting things back to normal—config files and scripts that took hours and hours to make. :(

Please be careful if you chose to install this app.
you are right, and i am very, very sorry.
It's not at all your fault; you cannot be responsible for a program that somebody else maintains. We all know that there is a small risk when playing around with Github stuff. I took the chance, and I take responsibility.
i guess you were using the install.sh script? I had problems with it, too, but the developer said they fixed it.
may i ask which version / how you installed? maybe i should re-open that issue:
https://github.com/gaenserich/hostsblock/issues/87

fwiw, the AUR installation method has always been reliable, and so is the script once you have it installed properly.

edit:
seems i cannot re-open the issue, i created a new one:
https://github.com/gaenserich/hostsblock/issues/88
Not sure of the version, but I know that I used the generic OS .zip file and installed it using the install.sh: https://github.com/gaenserich/hostsbloc ... master.zip

Thank you so much for re-opening the bug. That was awesome of you! *gives a star* :D

I wonder why they opted to have this program make its own user? Security? Seems unnecessary to me. :?


Here is my—very crude—hosts setup that I use for now if anybody is interested:

Code: Select all

#!/bin/sh

if ! [ $(id -u) = 0 ]; then
   echo "This script must be run as root. Exiting."
   exit 1
fi

hostfile="/etc/hosts"
cd /home/user/scripts
echo "Updating hosts..."

echo "127.0.0.1       localhost\n127.0.1.1       debian.mydomain debian\n" > $hostfile

# Pick one filter below: Unified hosts = (adware + malware), Unified hosts + fakenews + gambling + social, Unified hosts + fakenews + gambling + porn + social
hostblockurl="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"
#hostblockurl="https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-social/hosts"
#hostblockurl="https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn-social/hosts"

# Append to hosts
wget -O - $hostblockurl >> $hostfile

file="hosts.txt"
# Check if primary (longterm) hosts file exists.
if [ -f "$file" ]; then
        echo "$file found. Skipping download."
else
        echo "$file not found. Retrieving file..."
        wget https://hosts-file.net/download/hosts.zip; unzip hosts.zip
fi

# Append to hosts
cat hosts.txt >> $hostfile

# Fetch and append secondary updated hosts (goes along with the above primary host file)
hostblockurl="https://hosts-file.net/hphosts-partial.txt"
wget -O - $hostblockurl >> $hostfile

blocked=$(wc -l /etc/hosts | awk '{print $1;}')
echo "Hosts blocked: "$blocked
As can be seen, it combines the Steve's Github hosts with the huge hosts-file.net hosts. Yes, it's redundant and inefficient (750,000+ hosts blocked, but who knows if half of them even still exist or how many overlap). It will suffice until I have the time to make it more proper.

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: Adblocking with /etc/hosts

#19 Post by debiman »

just to clarify:
gaenserich's hostsblock itself works well; it's the install.sh script that doesn't.

but i agree that it's slightly overkill for the situation; otoh it's good to have multiple sources combined.
bedtime wrote:I wonder why they opted to have this program make its own user? Security? Seems unnecessary to me. :?
it didn't at first, then at some point they changed it.
makes sense to me; it's never good to download stuff from the internet as root (btw, apt is doing the same nowadays).
if you want to know exactly why i guess you'd need to go trawling the archlinux forums thread.

i'm keeping the new issue alive.
i know it might take a while til the developer answers, but they will evtl.

bedtime, which debian version?
anything particular we need to know about?

Post Reply