I have not read Roger Scruton's The Aesthetics of Architecture but I assume he more fully develops the point he makes in this interview:
One must think of the people who really use a building, who are those who pass it by. They don't know anything about the structure. But they do jolly well know the impact of that façade. Just think of what's involved in going through a doorway, and the difference between a sheet of glass which you can't actually identify the handle of, and something which arches over you and guides you in.
These elementary experiences are part of the difference between a building which welcomes and a building which creates anxiety. Nobody can deny that modern cities are increasingly places of great anxiety. And if you don't think architecture is one reason for this, it's because you don't have any eyes.
I have never made this connection between anxiety and architecture, but if it is accurate, I think it could be a good criticism of contemporary architecture. The Louvre entrance is arguably the best example of a confusing entrance; the Louvre's multiple glass pyramids that cover an underground entrance are worse than a simple sheet of glass with a hidden handle.
Later in the interview, Scruton says that if it is true that our lives are ones of anxiety, it would be the purpose of architecture to strengthen our lives in the face of anxiety; to simply reinforce this position, Scruton claims, is to show contempt for human beings.
If this is at all interesting, or if you want to hear Scruton briefly talk about liberalism, conservatism, deconstructionism, or pornography read the interview at salon.com
Please note that the interview has two pages to it; the link for the second page is at the bottom of the first.