The Invention of the PBJ

Updated: Mar 3, 2021


2020 has been one to remember for sure. Of all the terrible things that happened there is one that flew under the radar. Mostly because I didn't write this blog until 2021, but lets ignore that.


A burning question had been asked. "Can you run a Fortinet Firewall inside a Palo Alto Firewall inside a Checkpoint Firewall on a Ubuntu GNS3 box?". "Thats crazy!" you say. Why would you even want to do that?


Do you know what they said to David Chubinashvili when he applied Peanut Butter to Jelly and modified his beard to accommodate said sandwich? I'll tell you.


They said "That isn't your name and you didn't invent the PBJ".



Let that sink in for moment.


Ubuntu Donor Box - Create the chroot container for later user.

# To start things off we need to create a Linux chroot. This will be more or less a

# container to run everything in. Think of it as an old school Docker container.

# We've made a few of these but here is our latest version. These commands

# all need to be run on a donor Linux system. In this case I used Ubuntu 20.04

# (codenamed Focal) even though I use arch, btw.

root@Focal:~# debootstrap focal focal
root@Focal:~# cp /etc/apt/sources.list focal/etc/apt/
root@Focal:~# chroot focal bash -l 
root@Focal:~# apt update
root@Focal:~# apt install qemu-system tcpdump
root@Focal:~# apt upgrade 
root@Focal:~# dpkg-reconfigure locales
 - select "en_US.UTF-8 UTF-8" ; assuming English
root@Focal:~# dpkg-reconfigure tzdata
root@Focal:~# echo PA-Chroot > /etc/debian_chroot
root@Focal:~# exit
root@Focal:~# tar -zxvf focal-chroot.tgz focal

Checkpoint Firewall - Level 0 - Priming the Checkpoint for running the Palo Alto Chroot.

# Upload said focal-chroot.tgz to a Checkpoint firewall. I've put mine in /home/admin/

# Next load up the tun driver. This is a special network interface that

# will link our QEMU VM's virtual nic to a bridge interface in Checkpoint shortly.


[Expert@CHKP8040:0]# modprobe tun

# Next we need to load the KVM kernel module.

# In addition we need to make sure running nested VM support is enabled.

# This allows terrible things like running a VM inside of a VM in a speedier way.

# The KVM module gives us access to virtualization extensions

# (Intel VT or AMD-V). I'll be using kvm_intel module because this is running

# on a Intel box (Dell R720).


[Expert@CHKP8040:0]# modprobe kvm_intel nested=1

# Ok now we're going to get a little crazy. We need to load a driver (nbd) that

# will allow us to mount a qcow2 virtual hard drive image. This way we can

# add files and do terrible things to qcow2 images. More on this later.

# Note: Could also use guestmount to modify qcow2, but I opted for ndb.

[Expert@CHKP8040:0]# modprobe nbd max_part=8

# Now we're going to extract our Ubuntu chroot container on our Checkpoint box.


[Expert@CHKP8040:0]# cd /home/admin
[Expert@CHKP8040:0]# tar -zxvf focal-chroot.tgz

# Create tap interfaces in checkpoint using the tap driver we loaded earlier

# we're only going to use tap0 for this write up.

# 1,2 are for future use that didn't make it into this write up.


[Expert@CHKP8040:0]# ip tuntap add mode tap tap0
[Expert@CHKP8040:0]# ip tuntap add mode tap tap1
[Expert@CHKP8040:0]# ip tuntap add mode tap tap2
[Expert@CHKP8040:0]# ip link set tap0 up
[Expert@CHKP8040:0]# ip link set tap1 up
[Expert@CHKP8040:0]# ip link set tap2 up

# We'll be adding eth2 and tap0 to a bridge interface called br0

# Eth2 is connected to a production network and doesn't have an

# IP attached to it but is state up.


[Expert@CHKP8040:0]# clish
CHKP8040> add bridging group 0 interface eth2
CHKP8040> save config

# add tap0 to br0

# I don't do this via clish as there isn't a good way to create

# tap0 at boot before the bridge gets configured at boot time.


[Expert@CHKP8040:0]# brctl addif br0 tap0

# mount dev, proc and sys into the chroot container.

# Linux needs these!


[Expert@CHKP8040:0]# mount --rbind /dev /home/admin/focal/dev
[Expert@CHKP8040:0]# mount --bind /sys /home/admin/focal/sys
[Expert@CHKP8040:0]# mount --bind /proc /home/admin/focal/proc

# Now we need to take the focal-chroot, Palo Alto VM and Fortinet VM

# and copy it inside the chroot image so we can

# .. uh... make friends with them.


[Expert@CHKP8040:0]# cp PA-VM-KVM-9.1.2.qcow2 focal/root/
[Expert@CHKP8040:0]# cp focal-chroot.tgz focal/root
[Expert@CHKP8040:0]# cp FGT_VM64_KVM-v6-build1142-FORTINET.qcow2 focal/root

Palo Alto Ubuntu Chroot - Level 1 - Getting ready to run our Palo VM for the first time.


# log into the chroot


[Expert@CHKP8040:0]# chroot focal bash -l
(PA-Chroot)root@CHKP8040:/#

Palo Alto VM - Level 2 - Start PA VM and commit initial config.


# Now we start the PA VM up!

# Note we're just going to fire it up so it provisions.

# We'll then shut it down to add a new user once complete.

# It takes about 10 mins to finish this initial boot.

# default login is admin / admin

# Note: I'm hard coding the MACs of the nics so the VM doesn't get upset

# should they change on the next boot up.

    (PA-Chroot)root@CHKP8040:/# /usr/bin/qemu-system-x86_64 \
    -name PA-VM -m 8192M -smp cpus=8 \
    -cpu host -enable-kvm -machine q35 \
    -boot order=c \
    -drive file=/root/PA-VM-KVM-9.1.2.qcow2,if=virtio,index=0,media=disk,id=drive0 \
    -uuid 4ebdb5f2-9ff4-4f43-96fc-02711a4b2327 \
    -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no -device e1000,netdev=mynet0,mac=52:55:00:d1:54:01 \
    -netdev tap,id=mynet1,ifname=tap1,script=no,downscript=no -device e1000,netdev=mynet1,mac=52:55:00:d1:54:02 \
    -netdev tap,id=mynet2,ifname=tap2,script=no,downscript=no -device e1000,netdev=mynet2,mac=52:55:00:d1:54:03 -nographic

# PA-VM Login prompt!

# Login (admin / admin) and you'll be prompted to change the password,

# commit the new password and shutdown the system from the PA shell.


        
        admin@PA-VM> configure
        Entering configuration mode
        [edit]
        admin@PA-VM> commit force
        .......55%70%98%..........100%
        Configuration committed successfully
        [edit]
        admin@PA-VM> request shutdown system
        y
        (PA-Chroot)root@CHKP8040:/#
        

Palo Alto Ubuntu Chroot - Level 1 - Continue configuration and rooting() the PA VM.


# Now we'll modify the image once shut down to

# add a 2nd admin user with uid 0 (root privs).

# Connect the qcow2 of the PA image to /dev/nbd0.

# This will allow us to mount to root filesystem

# which is partition 2.


    (PA-Chroot)root@CHKP8040:/# qemu-nbd --connect /dev/nbd0 /root/PA-VM-KVM-9.1.2.qcow2

# Mount the root filesystem of the PA VM and copy focal chroot tgz file into the

# PA's root fileststem.


    (PA-Chroot)root@CHKP8040:/# mount /dev/nbd0p2  /mnt

# Copy focal chroot to the PA's root filesystem. This will become the place where we

# run the Fortinet VM.


    (PA-Chroot)root@CHKP8040:/# cp /root/focal-chroot.tgz /mnt/root/

# Copy Fortinet qcow to PA's root filesystem also.


    (PA-Chroot)root@CHKP8040:/# cp /root/FGT_VM64_KVM-v6-build1142-FORTINET.qcow2 /mnt/root

# Enter the Palo Alto VM's filesystem


    (PA-Chroot)root@CHKP8040:/# chroot /mnt bash -l

# Add admin2 user. Default shell will be bash.

# This will allow us to login with a root shell on the

# Palo Alto firewall.


        bash-4.2# adduser admin2
        bash-4.2# usermod -u 0 -o admin2
        bash-4.2# passwd admin2
        bash-4.2# exit

# We are now back in the chroot of focal

# umount PA qcow2 root filesystem.


    (PA-Chroot)root@CHKP8040:/# umount /mnt

# disconnect qcow2 from nbd driver.


    (PA-Chroot)root@CHKP8040:/#  qemu-nbd -d /dev/nbd0
    /dev/nbd0 disconnected

# Now fire up the PA firewall again. We should be able to login as admin2 now

# if you ever want to get back to the PA shell issue this command.

# also network access for mgmt interface should be working.

su - admin


Palo Alto VM - Level 2 - Configure the Palo Alto VM for running the Fortinet VM.


    (PA-Chroot)root@CHKP8040:/# /usr/bin/qemu-system-x86_64 \
    -name PA-VM -m 8192M -smp cpus=8 -cpu host -enable-kvm \
    -machine q35 \
    -boot order=c \
    -drive file=/root/PA-VM-KVM-9.1.2.qcow2,if=virtio,index=0,media=disk,id=drive0 \ 
    -uuid 4ebdb5f2-9ff4-4f43-96fc-02711a4b2327 \
    -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no -device e1000,netdev=mynet0,mac=52:55:00:d1:54:01 \
    -netdev tap,id=mynet1,ifname=tap1,script=no,downscript=no -device e1000,netdev=mynet1,mac=52:55:00:d1:54:02 \
    -netdev tap,id=mynet2,ifname=tap2,script=no,downscript=no -device e1000,netdev=mynet2,mac=52:55:00:d1:54:03 -nographic


# Login as admin2

# /opt/planlogs has the most free diskspace so we'll put our focal chroot

# that will store the Fortinet VM in there. BTW updating the contents

# of /etc/debian_chroot is a quick way to keep the prompts clear.


        [root@PA-VM ~]# cd /opt/panlogs
        [root@PA-VM ~]# tar -zxvf /root/focal-chroot.tgz
        [root@PA-VM ~]# echo Fortinet-Chroot > /opt/panlogs/focal/etc/debian_chroot

# Now we need to mount up linux dirs


        [root@PA-VM ~]# mount --rbind /dev /opt/panlogs/focal/dev/
        [root@PA-VM ~]# mount --bind /sys /opt/panlogs/focal/sys
        [root@PA-VM ~]# mount --bind /proc /opt/panlogs/focal/proc


# copy the Fortinet qcow2 to the chroot.


        [root@PA-VM ~]# cp FGT_VM64_KVM-v6-build1142-FORTINET.qcow2 /opt/panlogs/focal/root

# make sure nested kvm is enabled on hyper visor


        [root@PA-VM ~]# modprobe kvm_intel nested=1

# bring up a br0 interface


        [root@PA-VM ~]#  ip link add name br0 type bridge
        [root@PA-VM ~]#  ip link set dev br0 up

# 1-2 are for future use.


        [root@PA-VM ~]# ip tuntap add mode tap tap4
        [root@PA-VM ~]# ip tuntap add mode tap tap5
        [root@PA-VM ~]# ip tuntap add mode tap tap6
        [root@PA-VM ~]# ip link set tap4 up
        [root@