Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[Tips] View configuration files without lines commented with a # hash.

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
kedaha
Posts: 3521
Joined: 2008-05-24 12:26
Has thanked: 33 times
Been thanked: 77 times

[Tips] View configuration files without lines commented with a # hash.

#1 Post by kedaha »

[Edited based on replies below]
Remember the old saying that sometimes you can't see the wood for the trees? With what magic wand can one wave the trees away and make them invisible?
With grep
I think the use of this command could be useful—but not always—in forum posts to obviate referencing lengthy files crammed with irrelevant commented lines either here or on pastebin sites.
Let's start with a configuration file that everyone is familiar with:
Of course, cat will show the everything:

Code: Select all

cat /etc/apt/sources.list
# deb cdrom:[Debian GNU/Linux 11.2.0 _Bullseye_ - Official amd64 NETINST 20211218-11:12]/ bullseye main

#deb cdrom:[Debian GNU/Linux 11.2.0 _Bullseye_ - Official amd64 NETINST 20211218-11:12]/ bullseye main

deb http://deb.debian.org/debian/ bullseye main
# deb-src http://deb.debian.org/debian/ bullseye main

deb http://security.debian.org/debian-security bullseye-security main
# deb-src http://security.debian.org/debian-security bullseye-security main

# bullseye-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://deb.debian.org/debian/ bullseye-updates main
# deb-src http://deb.debian.org/debian/ bullseye-updates main

# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.
Now, if you only want to see the uncommented lines, you can use this command, which suffices for this file.
Note the full stop after | grep . in the above command, which is to remove empty lines. [Edited in view of p.H.'s question]

Code: Select all

$ grep -v ^\# /etc/apt/sources.list | grep  .
However, a much better grep command, for reasons given below, is:

Code: Select all

$ grep -Ev '^\s*$|^\s*\#' /etc/apt/sources.list
And you'll only see the active repos:

Code: Select all

deb http://deb.debian.org/debian/ bullseye main
deb http://security.debian.org/debian-security bullseye-security main
deb http://deb.debian.org/debian/ bullseye-updates main
The sources.list file is not so long:

Code: Select all

$ wc -l </etc/apt/sources.list 
20
Alternatively one can use the command inxi -r or inxi --repos in full but other files can have a lot of commented lines.
Another file which everyone should be familiar with is grub.
The next command shows that it has 32 lines:

Code: Select all

$ wc -l </etc/default/grub 
32
[Edit Thanks to p.H. & Bloom's replies]: cat is not needed:

And while:

Code: Select all

grep -v ^\# /etc/default/grub | grep  .
Is OK for many common default files, as p.H points out, it does nothing to either
remove lines which contain spaces/tabs before #
or
It does not remove lines which contain only spaces/tabs.
So, after a bit of searching I came across this code thanks to gbrener at stackoverflow, which does precisely that:

Code: Select all

$ grep -Ev '^\s*$|^\s*\#' filename
For example:

Code: Select all

$ grep -Ev '^\s*$|^\s*\#' /etc/default/grub
Which results in:

Code: Select all

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
And the command has whittled the number of lines down from 32 to only 5, which improves readability.
Now here's a real humdinger of a file:

Code: Select all

# Example config file /etc/vsftpd.conf
#
# The default compiled in settings are fairly paranoid. This sample file
# loosens things up a bit, to make the ftp daemon more usable.
# Please see vsftpd.conf.5 for all compiled in defaults.
#
# READ THIS: This example file is NOT an exhaustive list of vsftpd options.
# Please read the vsftpd.conf.5 manual page to get a full idea of vsftpd's
# capabilities.
#
#
# Run standalone?  vsftpd can run either from an inetd or as a standalone
# daemon started from an initscript.
listen=NO
#
# This directive enables listening on IPv6 sockets. By default, listening
# on the IPv6 "any" address (::) will accept connections from both IPv6
# and IPv4 clients. It is not necessary to listen on *both* IPv4 and IPv6
# sockets. If you want that (perhaps because you want to listen on specific
# addresses) then you must run two copies of vsftpd with two configuration
# files.
listen_ipv6=YES
#
# Allow anonymous FTP? (Disabled by default).
anonymous_enable=NO
#
# Uncomment this to allow local users to log in.
local_enable=YES
#
# Uncomment this to enable any form of FTP write command.
write_enable=YES
#
# Default umask for local users is 077. You may wish to change this to 022,
# if your users expect that (022 is used by most other ftpd's)
local_umask=022
#
# Uncomment this to allow the anonymous FTP user to upload files. This only
# has an effect if the above global write enable is activated. Also, you will
# obviously need to create a directory writable by the FTP user.
#anon_upload_enable=YES
#
# Uncomment this if you want the anonymous FTP user to be able to create
# new directories.
#anon_mkdir_write_enable=YES
#
# Activate directory messages - messages given to remote users when they
# go into a certain directory.
dirmessage_enable=YES
#
# If enabled, vsftpd will display directory listings with the time
# in  your  local  time  zone.  The default is to display GMT. The
# times returned by the MDTM FTP command are also affected by this
# option.
use_localtime=YES
#
# Activate logging of uploads/downloads.
xferlog_enable=YES
#
# Make sure PORT transfer connections originate from port 20 (ftp-data).
connect_from_port_20=YES
#
# If you want, you can arrange for uploaded anonymous files to be owned by
# a different user. Note! Using "root" for uploaded files is not
# recommended!
#chown_uploads=YES
#chown_username=whoever
#
# You may override where the log file goes if you like. The default is shown
# below.
#xferlog_file=/var/log/vsftpd.log
#
# If you want, you can have your log file in standard ftpd xferlog format.
# Note that the default log file location is /var/log/xferlog in this case.
#xferlog_std_format=YES
#
# You may change the default value for timing out an idle session.
#idle_session_timeout=600
#
# You may change the default value for timing out a data connection.
#data_connection_timeout=120
#
# It is recommended that you define on your system a unique user which the
# ftp server can use as a totally isolated and unprivileged user.
#nopriv_user=ftpsecure
#
# Enable this and the server will recognise asynchronous ABOR requests. Not
# recommended for security (the code is non-trivial). Not enabling it,
# however, may confuse older FTP clients.
#async_abor_enable=YES
#
# By default the server will pretend to allow ASCII mode but in fact ignore
# the request. Turn on the below options to have the server actually do ASCII
# mangling on files when in ASCII mode.
# Beware that on some FTP servers, ASCII support allows a denial of service
# attack (DoS) via the command "SIZE /big/file" in ASCII mode. vsftpd
# predicted this attack and has always been safe, reporting the size of the
# raw file.
# ASCII mangling is a horrible feature of the protocol.
#ascii_upload_enable=YES
#ascii_download_enable=YES
#
# You may fully customise the login banner string:
#ftpd_banner=Welcome to blah FTP service.
#
# You may specify a file of disallowed anonymous e-mail addresses. Apparently
# useful for combatting certain DoS attacks.
#deny_email_enable=YES
# (default follows)
#banned_email_file=/etc/vsftpd.banned_emails
#
# You may restrict local users to their home directories.  See the FAQ for
# the possible risks in this before using chroot_local_user or
# chroot_list_enable below.
#user_sub_token=$USER
#local_root=/home/$USER/ftp
#
# You may specify an explicit list of local users to chroot() to their home
# directory. If chroot_local_user is YES, then this list becomes a list of
# users to NOT chroot().
# (Warning! chroot'ing can be very dangerous. If using chroot, make sure that
# the user does not have write access to the top level directory within the
# chroot)
#chroot_local_user=YES
#chroot_list_enable=YES
# (default follows)
#chroot_list_file=/etc/vsftpd.chroot_list
#
# You may activate the "-R" option to the builtin ls. This is disabled by
# default to avoid remote users being able to cause excessive I/O on large
# sites. However, some broken FTP clients such as "ncftp" and "mirror" assume
# the presence of the "-R" option, so there is a strong case for enabling it.
#ls_recurse_enable=YES
#
# Customization
#
# Some of vsftpd's settings don't fit the filesystem layout by
# default.
#
# This option should be the name of a directory which is empty.  Also, the
# directory should not be writable by the ftp user. This directory is used
# as a secure chroot() jail at times vsftpd does not require filesystem
# access.
#@secure_chroot_dir=/var/run/vsftpd/empty
#
# This string is the name of the PAM service vsftpd will use.
pam_service_name=vsftpd
#
# This option specifies the location of the RSA certificate to use for SSL
# encrypted connections.
#rsa_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
#rsa_private_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
# This option specifies the location of the RSA certificate to use for SSL
# encrypted connections.
rsa_cert_file=/etc/letsencrypt/live/example.com/fullchain.pem
rsa_private_key_file=/etc/letsencrypt/live/example.com/privkey.pem
ssl_enable=YES
allow_anon_ssl=NO
# Options to force all communications over SSL - why would you want to
# allow clear these days? Comment them out if you don't want to force
# SSL though
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO

require_ssl_reuse=NO
ssl_ciphers=HIGH
pasv_min_port=40000

pasv_max_port=41000
userlist_enable=YES
userlist_file=/etc/vsftpd.userlist
userlist_deny=NO
#
# Uncomment this to indicate that vsftpd use a utf8 filesystem.
#utf8_filesystem=YES
How many lines?

Code: Select all

$ wc -l </etc/vsftpd.conf 
177
Now viewing the uncommented lines:

Code: Select all

$ grep -Ev '^\s*$|^\s*\#' /etc/vsftpd.conf 
listen=NO
listen_ipv6=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES
pam_service_name=vsftpd
rsa_cert_file=/etc/letsencrypt/live/example.com/fullchain.pem
rsa_private_key_file=/etc/letsencrypt/live/example.com/privkey.pem
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
pasv_min_port=40000
pasv_max_port=41000
userlist_enable=YES
userlist_file=/etc/vsftpd.userlist
userlist_deny=NO
Brings it down from 177 to only 27 lines.
Thank you for reading.
Last edited by kedaha on 2022-02-16 12:40, edited 1 time in total.
DebianStable

Code: Select all

$ vrms

No non-free or contrib packages installed on debian!  rms would be proud.

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: TIP: View configuration files without lines commented with a # hash.

#2 Post by p.H »

kedaha wrote: 2022-02-15 11:35grep -v ^\#
It does not remove lines which contain spaces/tabs before #.
kedaha wrote: 2022-02-15 11:35| grep .
I guess this it to remove empty lines ? It does not remove lines which contain only spaces/tabs.
kedaha wrote: 2022-02-15 11:35 cat /etc/apt/sources.list | wc -l
Useless Use of Cat.

Code: Select all

wc -l pathname

kedaha
Posts: 3521
Joined: 2008-05-24 12:26
Has thanked: 33 times
Been thanked: 77 times

Re: TIP: View configuration files without lines commented with a # hash.

#3 Post by kedaha »

Many thanks p.H. for your comments.
I'd like to improve the tip so any comments and suggestions are welcome.

Code: Select all

grep -v ^\#
p.H wrote:It does not remove lines which contain spaces/tabs before #.

Yes, good point but I still haven't looked at how they may be omitted.
p.H wrote:

Code: Select all

 grep .
I guess this it to remove empty lines?
Yes, because if omitted then empty lines don't get removed.
p.H wrote:It does not remove lines which contain only spaces/tabs.
I don't know how that might be done but it would make the output cleaner if such empty lines were removed too.
kedaha wrote: 2022-02-15 11:35 cat /etc/apt/sources.list | wc -l
Useless Use of Cat.

Code: Select all

wc -l pathname
But using cat, only the number is returned

Code: Select all

$ cat /etc/apt/sources.list | wc -l 
20
whereas

Code: Select all

$ wc -l /etc/apt/sources.list
parrots the pathname as well as the line count:

Code: Select all

20 /etc/apt/sources.list
DebianStable

Code: Select all

$ vrms

No non-free or contrib packages installed on debian!  rms would be proud.

User avatar
Bloom
df -h | grep > 90TiB
df -h | grep > 90TiB
Posts: 504
Joined: 2017-11-11 12:23
Been thanked: 26 times

Re: TIP: View configuration files without lines commented with a # hash.

#4 Post by Bloom »

Code: Select all

wc -l </etc/apt/sourcces.list
if you don't want the filename and just the number of lines.

kedaha
Posts: 3521
Joined: 2008-05-24 12:26
Has thanked: 33 times
Been thanked: 77 times

Re: TIP: View configuration files without lines commented with a # hash.

#5 Post by kedaha »

I did a bit of searching and came across this little piece of code at how-to-print-a-file-excluding-comments-and-blank-lines-using-grep-sed, where there's an interesting discussion about the subject. I've tried it and it does the trick:

Code: Select all

grep -Ev '^\s*$|^\s*\#' filename
.
To try it, I messed up a copy of my sources.list with hashes, and blank spaces before and after # and tabs and a few silly comments:

Code: Select all

#deb cdrom:[Debian GNU/Linux 11.2.0 _Bullseye_ - Official amd64 NETINST 20211218-11:12]/ bullseye main

deb http://deb.debian.org/debian/ bullseye main
# deb-src http://deb.debian.org/debian/ bullseye main

                                                                                                                                                                                             >
                #Hello Debian!
        #that's all folks!
    #Where's my beer?

        #       #  #            #

deb http://security.debian.org/debian-security bullseye-security main
# deb-src http://security.debian.org/debian-security bullseye-security main

                #no more blobs! 



# bullseye-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://deb.debian.org/debian/ bullseye-updates main
# deb-src http://deb.debian.org/debian/ bullseye-updates main

# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.

Now the code removes all the text comments & spaces/tabs:

Code: Select all

$ grep -Ev '^\s*$|^\s*\#' sources.list.txt
It only displays:

Code: Select all

deb http://deb.debian.org/debian/ bullseye main
deb http://security.debian.org/debian-security bullseye-security main
deb http://deb.debian.org/debian/ bullseye-updates main
So I think the first post can now be edited to include the improved commands.
DebianStable

Code: Select all

$ vrms

No non-free or contrib packages installed on debian!  rms would be proud.

bullseye
Posts: 2
Joined: 2022-12-04 23:46

Re: TIP: View configuration files without lines commented with a # hash.

#6 Post by bullseye »

I prefer:

Code: Select all

grep '^[^#]' /etc/apt/sources.list{,.d/*}
which only gives me the information I want - without blank lines, or lines that are commented out. i.e.

Code: Select all

:~$ grep '^[^#]' /etc/apt/sources.list{,.d/*}
/etc/apt/sources.list:deb http://deb.debian.org/debian/ bullseye main non-free contrib
/etc/apt/sources.list:deb https://deb.debian.org/debian-security bullseye-security main contrib non-free
/etc/apt/sources.list:deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
/etc/apt/sources.list:deb http://deb.debian.org/debian/ bullseye-backports main contrib non-free
/etc/apt/sources.list:deb http://security.debian.org/ bullseye-security contrib main non-free
/etc/apt/sources.list.d/balena-etcher.list:deb https://dl.cloudsmith.io/public/balena/etcher/deb/debian bullseye main
/etc/apt/sources.list.d/balena-etcher.list:deb-src https://dl.cloudsmith.io/public/balena/etcher/deb/debian bullseye main
/etc/apt/sources.list.d/brave-browser-release-.list:deb [arch=amd64] https://brave-browser-apt-release.s3.brave.com/ bullseye main
/etc/apt/sources.list.d/google-chrome.list:deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main
/etc/apt/sources.list.d/home:stevenpusser.list:deb http://download.opensuse.org/repositories/home:/stevenpusser/Debian_11/ /
/etc/apt/sources.list.d/ivideon.list:deb [signed-by=/usr/share/keyrings/ivideon-archive-keyring.gpg] http://packages.ivideon.com/ubuntu stable non-free
/etc/apt/sources.list.d/signal-xenial.list:deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main
/etc/apt/sources.list.d/virtualbox.list:deb [arch=amd64] http://download.virtualbox.org/virtualbox/debian bullseye contrib
/etc/apt/sources.list.d/vivaldi.list:deb [arch=amd64] https://repo.vivaldi.com/stable/deb/ stable main
/etc/apt/sources.list.d/zoneminder.list:deb https://zmrepo.zoneminder.com/debian/release-1.36 buster/

itmicp
Posts: 38
Joined: 2013-05-11 04:45
Location: France
Has thanked: 6 times
Been thanked: 6 times

Re: [Tips] View configuration files without lines commented with a # hash.

#7 Post by itmicp »

Hi

EDIT: I change use by used

With bash and for .list files, I used this one:

Code: Select all

grep -Env "^#|^$" /etc/apt/sources.list{,.d/*.list}
which gives the same return as yours,
and …
- with the line number in the file,
- whitout blank lines,
- whithout lines from files whose extension name is not .list

=======
and I used the same grep regex for the other configuration files:

Code: Select all

grep -Env "^#|^$" /etc/ssh/ssh{,d}_config
=====
But after reading your posts, I will use the regex you suggest:

Code: Select all

grep -Env '^\s*$|^\s*\#' /etc/apt/sources.list{,.d/*.list}

Code: Select all

grep -Env '^\s*$|^\s*\#' /etc/ssh/ssh{,d}_config

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: [Tips] View configuration files without lines commented with a # hash.

#8 Post by p.H »

bullseye wrote: 2022-12-04 23:55 /etc/apt/sources.list{,.d/*}
Warning: this is a "bashism", not part of the POSIX shell specification. It won't work in a script using the default /bin/sh interpreter (dash).

itmicp
Posts: 38
Joined: 2013-05-11 04:45
Location: France
Has thanked: 6 times
Been thanked: 6 times

Re: [Tips] View configuration files without lines commented with a # hash.

#9 Post by itmicp »

The three below will work with the sh shell (<=> /usr/bin/dash)

Code: Select all

grep '^[^#]' /etc/apt/sources.list /etc/apt/sources.list.d/*

Code: Select all

grep -Env '^\s*$|^\s*\#' /etc/apt/sources.list /etc/apt/sources.list.d/*.list

Code: Select all

grep -Env '^\s*$|^\s*\#' /etc/ssh/ssh_config /etc/ssh/sshd_config

Post Reply