It was in a flashback, where we learn that Harry had actually met Daphne and her family in the zoo before Hogwarts (from book1) and they recognise who he is, before Dumbledore shows up and obliviates him. Daphne’s father tried to intervene but i can't remember what happened then. Afterwards Harry doesn’t remember Daphne for years in Hogwarts and acts hostile towards her which causes her to be very upset.

Anyway i'm pretty sure Dumbledore was actively obliviating Harry before Hogwarts, showing up at the Dursleys whenever he did some accidental magic, started recovering his memories, or learn about magic (and also suppressing his magical core?). He might have even encouraged, or put the the Dursleys under a spell to abuse him. It was the typical "Ron was never Harry's friend and only wanted his money alongside Hermione" fic.

Please help me find this as i have been looking for it for AGES! It's basically a typical manipulative Dumbledore story which there is so many of, and that's why i have trouble finding it.