There should be no harm in using Next even if you are building something strictly front end. The workflow is practically identical to a create-react-app project if you use the Pages directory (which I recommend over the new App directory for your case). If anything, you'll benefit from server side rendering!

If you are looking to minimize bundle size as much as possible, however, Preact-vite or Preact-astro should be good as mentioned before. It's a 3kb version of React, so there's nothing more to learn to use it :)