Preheat your oven to 370F and greasy a loaf pan. Set aside.
Using your mixer, cream the butter and sugar together for about 10 minutes. This is key to a fluffy cake. Take the time to scrape the sides of the bowl as well during this time as well.
Whisk together the milk, sour cream (or yogurt), egg, vanilla, and lemon zest in a separate bowl.
Once butter and sugar are creamed, add this liquid mixture. Incorporate until mixed throughly.
Add the nutmeg, flour, baking powder, and salt, and mix until combined. Do not over mix.
Pour into prepared loaf pan and set aside while making the topping.
To make topping, stir together the chopped pears, oats, sugar, and nutmeg. Sprinkle this mixture over the batter. Bake in the oven about 35 minutes, or until golden brown around the edges.