I had a look round Manchester yesterday. It was your standard small American city. At 108,000 people it's small, and the lack of size probably contributes to the lack of any good museums, but their art gallery isn't too bad. I also have a thing about State Capitols (I'm a politics junkie I'm afraid), so I visited that too.
I visited a few very pretty (almost too pretty) villages and Hildene (Abe Lincoln's son's house), in Vermont today. It's a very pretty state, all hills and trees, but the things that make me suspicious about New Hampshire makes me suspicious about Vermont. It won't to be big enough to have good museums, but if I can find some 'countryside things' to do, I'm sure I'll enjoy it though.
I will give one one word of warning though, a lot of the museums in the more rural areas of Vermont seem to be seasonal (some of the drives and Norman Rockwell's former house for example), so if you go, go between May and October.