Except that NLewis and Pwol, the people who built it, didn't approve of the block removal.
By that logic, if I came across an intricately-built mansion in the middle of nowhere and thought the scene would look better if the house wasn't there, I could destroy the whole thing without penalty. Nice try.