A part of a town or city that is particularly impoverished (and usually dangerous or undesirable as a result). "Tracks" refers to railroad tracks, which are sometimes thought of as demarcating different economic areas of a town. I was always looked down on as a kid because I grew up on the wrong side of the tracks. His mother didn't want him dating anyone from the wrong side of the tracks.