My setup consists of an x86 “network appliance”. They’re low-cost, low-power, sometime headless units that can run most any open-source software solution. Even with RAM as bad as it is, you can find these for less than $200 from multiple sources if you’re okay with 2.5Gbps. If you did need more than that, you’d have to build something custom likely, unless it’s a small-business class device.
If you’re just getting into the network software side of things, I recommend experimenting off-Internet for a while with a few different operating systems. Find which one has a user-interface you enjoy and can wrap your head around since in a personal homelab, they almost all meet the requirements. My personal choice for software ended up being OPNSense.
I run all of my other services that aren’t network related on separate hardware via a managed switch just to keep things isolated and the rest is up to you.
it’s worth mentioning mine is over 10 years old and was that cheap when I bought it. If you can find a fanless design, these will last ages since there’s a few simple non-moving components that do the same kind of work all day every day and don’t take a ton of abuse. Even after all these years, the CPU on mine sits mostly idle even with heavy traffic. Modern software is just very efficient.
Another reason the router/firewall lasts is because I use a dedicated WiFi antenna. This is another part that gets the same work every day. I’ve upgraded mine twice, but it was almost double the speed each time, and I didn’t have to spend extra money on routing hardware that was still plenty powerful.