We know of 4 so far: 2 had been left on abandoned planets (Aether/Reality and Power), one was left on Earth by the Asgardians (Space/Tesseract), one was held by Thanos (Sceptre/Mind), until he gave it to Loki. That leaves 2, Soul and Time.

At the end of Age of Ultron Thor is aware of all of this, and wants to know what is going on, so he goes to seek out the answer to what is going on. Now I believe that the soul stone is inside of Mjolnir. The hammer has powers unlike those of any other Asgardian weapons that we've seen being used, and apparently has (what I believe to be) some degree of awareness as it can judge those who try to wield it, it can judge their souls to see if they are worthy. Also, Mjolnir wasn't created by the Asgardians, it was created by an ancient race of dwarves...a race long gone, like those who build the temples where the other stoned were found?

We also know that Mjolnir was forged in the heart of a dying star. Now, what do you get when some stars die? A black hole. What is at the centre of a blackhole? A singularity, What are the infinity stones? Fragments of the singularity that the universe formed from...okay that link is a bit tenuous, but oh well.

Also: Ragnorok (title of the next Thor movie) is the Norse apocalypse, the end of time.
Time stone link? Possibly