worm in the apple

Noun

 * 1) A serious flaw in something otherwise good; a problem that ruins everything else.