According to Kevin Feige, large parts of The Incredible Hulk were written in such a way that "people who wanted to think that it was a sequel could think of it as a sequel, and people who wanted to come in off the street and had never seen the first one can enjoy it as well" (source). They eventually changed his origin story to affirm that it is not a sequel. However, if one were to view that as a retcon (after all, the origin story is not emphasized in the second film) would any other plot holes come up?

After seeing Hulk for the first time today, I can't help but feel that it just makes sense to see them as continuous. The first film largely deals with his backstory and ostracization, and the second film explores how he copes with his new abilities. Any other thoughts? I actually enjoyed the first three-quarters of Hulk.

Edit: One more question. Hulk ends with Banner treating patients in South America while in hiding, which is the same thing he is doing at the beginning of The Avengers. What plot holes would arise if they had never made The Incredible Hulk and let the first movie lead straight into The Avengers?

Edit 2: As AemsOne pointed out, the Hulk is actually in India at the beginning of The Avengers.