Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[Solved] is this /etc/libvirt/hooks/qemu redirect okay?

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

[Solved] is this /etc/libvirt/hooks/qemu redirect okay?

#1 Post by Chiefahol2 »

hello again,

I'm trying to set up iptables instead of ufw on Debian 10, i basically need to forward 5 external ports to a kvm/qemu instance (22, 80, 443, 8448, 3478), host occupies 192.168.1.170 on the local network, guest connects via virbr0 (virtio) adapter, it's IP is 192.168.122.182 (with gateway 192.168.122.1).

Here is the default firewall setup for the host:

Code: Select all

*filter

# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allows all outbound traffic
# You could modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

# Allows Federation port for Synapse.
-A INPUT -p tcp --dport 8448 -j ACCEPT

# Allows coturn service.
-A INPUT -p tcp --dport 3478 -j ACCEPT

# Allows SSH connections 
# The --dport number is the same as in /etc/ssh/sshd_config
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

# Now you should read up on iptables rules and consider whether ssh access 
# for everyone is really desired. Most likely you will only allow access from certain IPs.

# Allow ping
#  note that blocking other types of icmp packets is considered a bad idea by some
#  remove -m icmp --icmp-type 8 from this line to allow all kinds of icmp:
#  https://security.stackexchange.com/questions/22711
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls (access via 'dmesg' command)
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
# enable logging generally
#-A INPUT -j LOG

# Reject all other inbound - default deny unless explicitly allowed policy:
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT
Does this look okay? I essentially want to allow outgoing and deny incoming packets, except for the 5 ports i wish to forward (22, 80, 443, 8448, 3478).

Here is the guide for just forwarding port 22 to a kvm instance just using 'hooks' on the hostOS: https://wiki.libvirt.org/page/Networkin ... onnections

I've modified it like so, i'm also curious if this looks alright:

Code: Select all

#!/bin/bash

# IMPORTANT: Change the "VM NAME" string to match your actual VM Name.
# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "kvm1-website.org" ]; then

   # Update the following variables to fit your setup
   GUEST_IP=192.168.122.182

   GUEST_PORT1=22
   HOST_PORT1=22

   GUEST_PORT2=80
   HOST_PORT2=80

   GUEST_PORT3=443
   HOST_PORT3=443

   GUEST_PORT4=8448
   HOST_PORT4=8448

   GUEST_PORT5=3478
   HOST_PORT5=3478

   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
	/sbin/iptables -D FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
	/sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT1 -j DNAT --to $GUEST_IP:$GUEST_PORT1
	/sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT2 -j DNAT --to $GUEST_IP:$GUEST_PORT2
	/sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT3 -j DNAT --to $GUEST_IP:$GUEST_PORT3
	/sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT4 -j DNAT --to $GUEST_IP:$GUEST_PORT4
	/sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT5 -j DNAT --to $GUEST_IP:$GUEST_PORT5
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
	/sbin/iptables -I FORWARD -o virbr0 -d  $GUEST_IP -j ACCEPT
	/sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT1 -j DNAT --to $GUEST_IP:$GUEST_PORT1
	/sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT2 -j DNAT --to $GUEST_IP:$GUEST_PORT2
	/sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT3 -j DNAT --to $GUEST_IP:$GUEST_PORT3
	/sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT4 -j DNAT --to $GUEST_IP:$GUEST_PORT4
	/sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT5 -j DNAT --to $GUEST_IP:$GUEST_PORT5
   fi

fi
Any inputs appreciated, thank you.

Edit: macvtap with passthrough > virtio and a VLAN, at least for this humble application (home server)
Last edited by Chiefahol2 on 2019-09-13 08:29, edited 2 times in total.

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#2 Post by Chiefahol2 »

i first removed ufw and reset the iptables rules like so:

Code: Select all

# sudo iptables -P INPUT ACCEPT
# sudo iptables -P FORWARD ACCEPT
# sudo iptables -P OUTPUT ACCEPT
# sudo iptables -t nat -F
# sudo iptables -t mangle -F
# sudo iptables -F
# sudo iptables -X

# sudo ip6tables -P INPUT ACCEPT
# sudo ip6tables -P FORWARD ACCEPT
# sudo ip6tables -P OUTPUT ACCEPT
# sudo ip6tables -t nat -F
# sudo ip6tables -t mangle -F
# sudo iptables -F
# sudo iptables -X

# sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Code: Select all

# sudo iptables-restore < /home/pcadmin/PortForward/iptables.test.rules

# sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             127.0.0.0/8          reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8448
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:3478
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere  
After applying the hook script,
closing the kvm guest with virt-manager,
and restarting the libvirtd service,

Code: Select all

# nano /etc/libvirt/hooks/qemu
# chmod -x /etc/libvirt/hooks/qemu

# sudo service libvirtd restart
i then fired up the guest vm and tried to ssh into it:

Code: Select all

$ ssh -vvv pcadmin@192.168.1.170
OpenSSH_7.9p1 Debian-10, OpenSSL 1.1.1c  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.170 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.170 [192.168.1.170] port 22.
debug1: connect to address 192.168.1.170 port 22: Connection refused
ssh: connect to host 192.168.1.170 port 22: Connection refused
This guest VM is able to access the web, but the port forwarding does not seem to be working properly. :?

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#3 Post by Chiefahol2 »

the pre-routing rules in the hook script are not being applied it seems:

Code: Select all

# sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1886  126K ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    7   548 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    7   548 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68
*strokes mustashe*

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#4 Post by Chiefahol2 »

after looking at the iptables result i can't see the commands listed in the hook file?

i decided to try entering these manually:

1) reset all the firewall rules in the same way as above

2) applied list of rules:

Code: Select all

# sudo iptables-restore < /home/user/PortForward/iptables.test.rules

# sudo iptables-restore < /home/pcadmin/PortForward/iptables.test.rules2
# sudo iptables -t nat -I PREROUTING -p tcp --dport 22 -j DNAT --to 192.168.122.182:22
# sudo iptables -t nat -I PREROUTING -p tcp --dport 80 -j DNAT --to 192.168.122.182:80
# sudo iptables -t nat -I PREROUTING -p tcp --dport 443 -j DNAT --to 192.168.122.182:443
# sudo iptables -t nat -I PREROUTING -p tcp --dport 8448 -j DNAT --to 192.168.122.182:8448
# sudo iptables -t nat -I PREROUTING -p tcp --dport 3478 -j DNAT --to 192.168.122.182:3478

# sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             127.0.0.0/8          reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8448
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:3478
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.182     
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere
3) Tested ssh access:

Code: Select all

$ ssh -vvv user@192.168.1.170
OpenSSH_7.9p1 Debian-10, OpenSSL 1.1.1c  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.170 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.170 [192.168.1.170] port 22.
debug1: connect to address 192.168.1.170 port 22: Connection timed out
ssh: connect to host 192.168.1.170 port 22: Connection timed out
I can't seem to ping 8.8.8.8 from the guestos, i get 'Destination Port Unreachable', this setup might be blocking the guests web access entirely. :?

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#5 Post by Chiefahol2 »

oow forgot to clear the /etc/libvirt/hooks/qemu file, also resetting libvirt.d service..

hooray the guest has access to the web again,

Code: Select all

$ ssh -vvv user@192.168.1.170
OpenSSH_7.9p1 Debian-10, OpenSSL 1.1.1c  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.170 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.170 [192.168.1.170] port 22.
debug1: connect to address 192.168.1.170 port 22: Connection refused
ssh: connect to host 192.168.1.170 port 22: Connection refused
still unable to connect though, darn

CwF
Global Moderator
Global Moderator
Posts: 2672
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 196 times

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#6 Post by CwF »

I'm blurry on your purpose.

Personally, I ditched virbr and use macvtap only for shared physical ports.
By default, all guest see each other, host is invisible. The way it should be.
To involve the host in the network, physically loop an additional host port to an external hub/switch.
To give a single VM access to/from the host, vfio pass an additional port to the external hub/switch.

Such hardware solution will stay working, software workarounds will continually need fixed.
Further, it would be best to have separate routes for internet and intranet.

You also could look into using the spice channel instead of the eth interface.

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#7 Post by Chiefahol2 »

CwF wrote:I'm blurry on your purpose.

Personally, I ditched virbr and use macvtap only for shared physical ports.
By default, all guest see each other, host is invisible. The way it should be.
To involve the host in the network, physically loop an additional host port to an external hub/switch.
To give a single VM access to/from the host, vfio pass an additional port to the external hub/switch.

Such hardware solution will stay working, software workarounds will continually need fixed.
Further, it would be best to have separate routes for internet and intranet.

You also could look into using the spice channel instead of the eth interface.
Thank you for the thoughtful response, i wanted to have several kvm instances running eventually and have them all use 1 physical interface. I wanted to install a networking card with 4 extra ports but i ended up needing the pci lane for a gcard. :?

I'll have a look into macvtap, are you saying that the host won't be able to access the local network if a guest is using macvtap? That could be a problem. I don't need the guest to be able to access the host OS,i just need them both to be accessible from the local network.

CwF
Global Moderator
Global Moderator
Posts: 2672
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 196 times

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#8 Post by CwF »

Right, every vm can tap the same physical port. All those vm's will see each other by default, the host is not looped in, but does use the interface.

So for example if the first port is internet of some sort, all vm's tap that, and everything has internet, including the host. The host is invisible to the vm's, the vm's see each other, a dhcp router will give individual ip's to all. You can, but should not use this for vm to vm traffic.

Then the second port is the intranet and the same is true, all tapped vm to vm traffic, no host.

A third port is the used as I mentioned for intranet. If no host traffic is needed, no third port is needed.

If vm's are all internal and no need for external traffic, that second port doesn't even need a cable and the vm intranet traffic is entirely internal. If there is a cable and switch on that second port, anything hooked into the switch will see all, here's where that 3 port comes in if needed, a host line or an assisted vm.

This solution needs none of your trickery above, just default behavior. You would not be able to ssh between host/vm in either direction without a third port, and there should be no need to. Each vm has the spice channel for that if you need it, that's how the viewer connects.

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#9 Post by Chiefahol2 »

I'm exploring the macvtap options in virt-manager, there's 4:

Bridge
VEPA
Private
Passthrough

Which type should i be trying? Sorry for the noob question, i assume 'passthrough' mode, which I'll try test now.

Edit: passthrough mode works great, thank you! :D
Last edited by Chiefahol2 on 2019-09-13 04:09, edited 1 time in total.

CwF
Global Moderator
Global Moderator
Posts: 2672
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 196 times

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#10 Post by CwF »

Bridge

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#11 Post by Chiefahol2 »

Oww i selected passthrough and it seems to be working, the router now see's 2 MAC's and assigns each a local IP address. Is there anything wrong with using this?

CwF
Global Moderator
Global Moderator
Posts: 2672
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 196 times

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#12 Post by CwF »

if it works, it works!

User avatar
Chiefahol2
Posts: 110
Joined: 2016-08-06 22:49

Re: is this /etc/libvirt/hooks/qemu redirect okay?

#13 Post by Chiefahol2 »

You have been very helpful, thank you for your time.

Post Reply