Can You over Exaggerate?

It's correct. It means to exaggerate in a way which is inappropriate(ly excessive) for the circumstances. Exaggeration is a technique used to create interest in a story, but it's over-use (obviously subjective - some would call x over-exaggeration, some wouldn't) is over-exaggeration.

What does it mean to over exaggerate?

transitive + intransitive. : to exaggerate (something) to an excessive degree overexaggerating the threat/danger/risk Its effect/impact/importance was greatly overexaggerated.

Is over exaggerate an oxymoron?

No, "overexaggerate" is not an oxymoron. It means to exaggerate to a greater degree. The word can be found in the dictionary.