No, you were right. "Doing shit" is generally doing stuff. It needs a qualifier in order to mean doing nothing.