Kapa1611 is pretty much right.
It's just when you have an excess of population, i.e. more people than there are jobs to satisfy, you'll have "lazy" villagers.
They're basically just spares. As memory serves, even if you raised the totem in your village centre to max, the worship site seems to...