I believe Bran IS the tree man (henceforth, TM). [Possible spoilers across GoT series]

Overview

There have been many iterations of Bran, hence I will refer to them by number. Bran(1), Bran(2) etc

Originally the storyline ran much as it has now. Bran(1) got injured. travelled to the north with Meera and Hodor. Found the cave. Bran(1) stayed there until the tree grew around him. Essentially he turned into TM.

Meera and Hodor with died OR, i have a feeling, all the weird treewomen are each iterations of Meera.

Bran(1) learned how to travel back in time. He travelled back in time (not location) to when Bran(2) was a boy (so Bran(1) remained in the tree as TM). He tries to reach out to Bran(2) in his dreams, telling him to search out the three-eyed raven - he wanted Bran(2) to come find him. Bran(1)/TM waits, he probably spent his intervening time looking at events in history (searching for something?). Eventually Bran(2) reaches the cave. Bran(1) teaches Bran(2) how to time travel. I believe eventually we will discover that Bran(1) dies, Bran(2) takes his place and the process repeats. We do not know how many times this has happened.

I guess this iterations of Bran will be the last - this iterations of Bran will chance upon a key bit of knowledge - very likely R L=J.

Quotes

[S4E10] TM: "I've been watching you, all of you, all of your lives with a thousand eyes"

When TM says "all of your lives" he is not referring to just the length of their life, but the iterations of them.

[S6E03] TM: "Stay too long where you don't belong and you will never return"

This has a double meaning. One to do with being in the dreams, the second is self-referential. If Bran stays there too long he will become TM and stay there forever.