The slightest bit of noise can disturb your sleep, even if it doesn't wake you.
If you're exposed to any noise, keep your windows shut and put in some insulation or double glazing.
If you live on a noisy road or next to rowdy neighbours, you might want to think about splashing out on soundproofing for your walls (and ceiling and floor if you live in a flat). At the very least, put up a barrier of some sort (bookcases or shelves, for example), to reduce sound levels.