it doesnt even need to be explainable. If a model comes up with an equation that reconciles quantum physics with gravity then we could try to reason how it came to it, but we don't need to. As long as the model described in the equation is understandable by humans it should be enough. However most ML models are black boxes that solve problems without generating models of the world (that we know of)

Physicists and mathematicians are uncomfortable with computer-generated proofs. I think they 'd be less uncomfortable with computer-generated theories because they can understand those