I'm at the very end of fan-translating a novel series (僕はやっぱり気づかない), and what might be the dramatic payoff of the entire series is

僕は気づかなかったけど、築かなかったわけじゃない

This is in itself a one-liner, something along the lines of 'I might not notice, but it's not like it didn't build up.' The story is about this guy who can never notice the supernatural phenomenon going on around him, but even if he remains uninvolved to them, he still builds up human relationships over the course of these events.

The story itself points out this is a terrible pun, and the fact it is a pun influences quite a bit of dialogue after it.

I've been thinking for around three days, but I can't think up anyway to convey this in English. I'm at the second to last chapter of the very last volume, last one before the epilogue so giving up is not an option. It can be as bad of a pun as humanly possible, it can be physically painful, but it just needs to make sense. Any suggestions?

Edit: Thank you for the help, everyone. I'm going with Umishi's suggestion of

"Just because I couldn't connect, doesn't mean I didn't connect"

Here is a link to the Chapter in Question. Thanks to your contributions, I managed to finish the volume within the day