Hi,
Are you running a dedicated server? Even if you do, the 32 GB could be virtualization.
In regards to your question, the Prestashop Memory limit refers per/each PHP process.
Having it too high wouldn't make sense. I would say that you should increase to 1024 max 2MB in case your run some heavy Amazon Sync jobs, but if that's not the case than avoid it.
Imagine you add 2Mb,a nd you run 2 PHP processes over 1MB - the 3rd process will kill your server for sure. There is a more complex calculation in regards to the MB /visitor ratio, but having a too high MB can kill your Apache at the end(just ask your server)
In case you face some CPU issue, rather ask your hosting to provide you with a slow_querry_log, teh same applies on an Nginx server you can check each process.
I hope my answer helps you a bit
Regards,
Leo