I have pfSense currently running on a Proxmox machine with the following settings:

eth0 -> vmbr0 (pfWAN) eth1 -> vmbr1 (pfLAN)

eth0 is currently connected to my ISP modem, with Proxmox having a static IP of 192.168.2.20. I'm looking to ditch my ISP modem (I'm having issues with port forwarding, verified not an issue with pfSense settings) and have pfSense directly connected to the Internet. Will I be able to access Proxmox via my public IP? If so, how/will I be able to lock down Proxmox access to local traffic only?

As a sidenote, my ISP transmits their network traffic via VLAN35 (Bell Internet in Canada if anyone's wondering), do you guys know if 1, having a SFP to Ethernet converter will mess with that, and 2, if the setup is possible with Proxmox? I apologize if this isn't the right subreddit to ask and will take this down if need be.

EDIT: Thank you for all your responses. There seems to be some contention on if I can get rid of my ISP modem. I'm with Bell Fibre Internet in Canada. They supply a modem/router combo but it's really just a router with a SFP port. Bell uses PPPoE (for their Fibre at least) so AFAIK there's nothing special that the modem does other than providing a physical plugin. There are reddit posts dedicated on how to get rid of the box, with a YouTube video here. As for bandwidth throttling, I'm paying for Gigabit Internet and don't have anything capable of handling over that speed anyways, so the hardware throttles itself. My main concern is how to access Proxmox GUI and blocking remote access while having the pfSense VM be the router that directly accesses the Internet.