Is it bad to overbrush your retainer?

I just got my braces off, and now i have to wear a clear retainer. But im really scared that i might stain it, so i brush it about 4 times a day. Is that bad? Or should i not brush it that much?