Overtired means so tired that you cannot think or do things as you usually do, and you become annoyed more easily. Parents say it about small children who have missed sleep and then fall to pieces. Adults use it too: you stayed up too late, and now every small thing feels sharp. It is ordinary spoken English, a bit softer and more everyday than exhausted. Though actually, it is not only very tired. The extra idea is that the tiredness has started to change your mood and your judgment. A child who is overtired may cry and refuse to sleep, which sounds backwards, but that is a pattern people really talk about. Over- here means too much, past the useful point of being tired.