Because this only happens after inactivity I assumed it had something to do with Power Saving so I changed the Energy Saver settings and that fixed it!
Go to System Preferences:
Click Energy Saver:
Set Computer sleep to Never:
Was once an enthusiastic PepperByte employee but is now working elsewhere. His blogs are still valuable to us and we hope to you too.