If memory serves, small tiny hovels will always have negative attributes. Medium size houses are OK and Manor Houses give a lot of happiness.
However, a big component of all this is where you place the house (whichever one you pick). If you place the house beside an industrial building, that gives MORE negative happiness (who really wants to live next to a granary or smelter in all fairness?). On the flip side, placing houses near buildings like temples and such increase the happiness of the house.
It works the other way also though, so if people are happy you built a lovely temple and then you cram a whole bunch of ugly small houses/hovels around it, then the temple might loose some happiness if I recall (it's been a while). But it all balances out. A quick win might be to place trees and ornaments around the ugly buildings and houses. That should help decrease the negative effects.