The general solution to the problem was the discovery of an inherent flaw in the "recently viewed items" module. It causes the module's memory usage to scale with the number of active items in the products table.
At 1 million items, the module needs >400MB of RAM. At 3.4 million, it will need ~1.5GB of RAM as the issue scales linearly. The module is pulling back a list of all of the active product IDs in the database 3 times.
Dropping the "recently viewed items" module as well as the "items in the same category" module which has similar but not as pronounced problems, the memory usage for a product display page dropped from 416MB to 5MB. In addition, the load time for the page decreased from 4sec to 19ms.
While I would certainly expect that the load times and memory usage would increase with the use of these 2 modules, they do not scale properly for large product catalogs.
I will further address this issue in the development forums.