NetStream.time inaccurate

I am doing a playlist kind of thing and I know how long a stream will be so I checking the stream.time to provide a progress indicator. None of that is an issue, it all works fine, except I noticed the progress indicator jumping and jittering a bit. So I logged the data. Here is an example of what I see for the stream.time on a 200ms interval:
0
0.271
0.533
0.396
0.951
1.238
1.525
1.396

If you will notice, the time is non-linear. The video is smooth and does not jitter. I did notice that When I restart everything it seems to smooth out and stop doing it. But after a few minutes it starts jittering again. Has anyone noticed this? Is this a known issue? Any clue what might be going on here?

Thanks