You would not see matter falling through the horizon, but you would see the horizon grow beyond the point where you last saw the infalling matter. Just look at the diagrams. People have also done numeric simulations of this for realistic spacetimes, I believe you can find several of them on the LIGO websites. There are some mysteries here, but black hole formation and growth is not one of them.