on the wrong side of the tracks
In a part of a town or city that is particularly impoverished (and usually dangerous or undesirable as a result). His mother constantly warned him about spending time on the wrong side of the tracks.
from/on the ˌwrong side of the ˈtracks
(informal) from or living in a poor area or part of town: She married a man from the wrong side of the tracks.