Doug - I think KB 20344 may be in play -
https://goo.gl/GeikaH - I was the original author of this article way back when but it's more shorter and easier to read now :) Is App Server running as a service? If so, adjust the 3rd value for the services heap size. Bump it up from 768 to 2304. Reboot of server is required to take effect. But if App Server is running as an app (desktop app) , it is still possible with this many users and if on Server 2008 R2 or less, that you may need to actually adjust 2nd value for the desktop heap size. In the article it says desktop heap size starts at 20480 but for older o/s's, it is 3072 if memory serves. You could bump it to 3584 or 4096 then reboot.