This is what I often felt when running at school (not versus the track team, just the general workout folk). If I passed runners at home too, I would probably feel it then as well. But as it currently stands, I rarely see another runner at home.
But yeah, I'd be running on the track and if a faster person joins while i'm doing a long run and not going full force, then i'd be all like [on the inside] "yeah, i'm in the midst of a nine mile run dude. You're not as cool as you might think."