For generations Hispanics were migrating to the states you mentioned. They colonized first. See Pobladores https://en.m.wikipedia.org/wiki/Los_Angeles_Pobladores

Another group of people came along and decided that was enough of that. The land belongs to them now and put up a border after what Ulysses S. Grant called "the most unjust war" ever waged. Please correct me here if I'm wrong.