I'm doing my second run of the game and just cleared out the top floor of Felogyr's Fireworks, but when I try to loot the bodies or even pick up a single smokepowder bomb, guards come running from off-map to arrest me.

I've run through the fight three times now, the first where I stole after and had to reload a save because the guard initiated combat during my attempt to persuade them to leave. The second I had my group sneaking on the top floor and some random kid who was outside of the building ran in followed by guards. The third I decided to experiment and had Astarion steal something during the fight and sure enough, I see three dots run into the local map from outside the area into the fight and enter the initiative order. This despite him being under Greater Invisibility, and unviewed by anybody, and the invisibility actually dropped because of it.

So is it just me? Is this a bug that I'm not aware of? Or is the building just haunted by a somehow very Lawful spirit that abhors thievery?