Is it bad if I never wash or rinse my tea tumbler?!

I recently purchased a Starbucks tumbler that is made up of 21% post consumer recycled content and is BPA free. I use it everyday for tea. Green tea, camomile tea, and chai tea mostly. I never rinse it or wash it because I love the way the tea that was once in the tumbler makes the new tea taste. Also, I have the tea plain, no sugar, milk, honey, lemon ect. So, it's basically water and leaves in my tumbler.