This is something which has always irked me about the "black box" arguments regarding artificial neural nets. Sure, it's hard to interpret exactly "why" an ANN gives the result it gives (other than "it did the maths and that was the result") but people always gloss over the fact that we really don't know much at all about how the human brain generates its results. We just accept individual introspection as ground truth when, as any psychologist will tell you, it's anything but reliable.
>> but people always gloss over the fact that we really don't know much at all about how the human brain generates its results.
If you look carefully, you'll see that interpretability is an issue in domains
with either safety or liability requirements. Sometimes both are required-
like in medicine where a doctor must be able both to make decisions that do
not endanger their patient and justify those decisions if something goes
wrong.
In such domains, all technology tends to have very high reliability stnadards, in the sense that it is made so that it's possible to make predictions about its behaviour under various assumptions.
With neural nets, the black-box nature translates to a lack of
predictability, which is unacceptable.
Btw, in the same kind of domains, human introspection seldom comes into play.
Human actors are normally trained professionals who can explain their actions
with minimal recourse to introspection. For instance, a doctor will tell you
"I administered 50cc of Chloraxine because it has been shown to be beneficial
in cases where the patient suffers from third-degree burns etc etc", not "I
felt ambitious and bold and decided to give the patient some Chloraxine".
For that matter, you can look at a neural net and say "it's because this number times this number plus this number ... > 0". Which is a very low level explanation, but complete and correct. Humans don't provide complete and correct explanations at any conceptual level.
And there are lots of ways to get higher-level explanations from neural nets. For instance, if you retrain without a subset of samples and it gives a different answer, you can say that those excluded samples were relevant.
An explanation needs to have explanatory power, which is a statistical concept. Also specificity. The "sum of numbers" explanation is not specific - it can be used to justify anything. Additionally it has low power as minor changes will result in a different outcome.
A Markov model can give you some of that information, unlike a neural net, but it is still hard to get them to be specific.
An expert system can be specific and use an approach with good statistical power, but it's at a mercy of anyone putting the data in. (unless somehow extended to statistically evaluate the inputs too somehow)
Yeah, but if a driverless car does something stupid, people want an exact diagnosis ("sensor X failed", "the vision algorithm misidentified Ms. Whatsit as a paper bag due to a bug on line 714", etc.) whereas a human driver can say "I thought they were going to brake" and we just accept that that's why the person did it.
The difference is that neural nets are technologicl artifacts that we design and manufacture, unlike our own minds. We generally make sure that when we design and create such things, we know how to control them.
For example, a car breaking system that randomly failed to stop the car under conditions that were impossible to understand would be unacceptable.
When a component of a safety-critical system fails, we want to know why it fails. Not because we're curious, but because we want to avoid failure in the future. If the component is a black box that can't be reasoned about, then we can't know whether it will fail again in the future.
Also: "Ms Whatsit"? In view of the Uber accident, that is extremely insensitive.
> whereas a human driver can say "I thought they were going to brake" and we just accept that that's why the person did it.
We don't accept it. That person is cross-examined, doubted, their statement corroborated with other evidence - were they on their phone? What does the car telemetry say - when did they start braking? How fast were they going? What do witnesses say?
The algorithm itself did not provide this diagnostic data, unlike a human who sometimes can. (e.g. was distracted by a phone; it was too dark; the person entered the street from between the cars etc.)
Of course they can also lie.
Blackbox data is always in need of careful interpretation.
This is clearly the fundamental flaw of any study that attempts to study the mind. You can't publicly observe thoughts. You can convince yourself that the mind functions in a particular way because you may find yourself synced in with a group where it appears your own thoughts are reliable predictors for everything that are not your own thoughts, but this is clearly flawed.
All (and obviously, I don't say this lightly) you have to do is ask yourself the question whether you have the ability to select a different thought. A single different thought to use as a ground. You can ask yourself that question for the rest of your life.
Causality when it comes to mental events, while removing the ability to accept introspection as a valid means of reasoning. This is pure absurdity at it's finest hour.
Yes; however deductive reasoning is stronger evidence if you have it. It seems that’s what the OP might be implying, although I haven’t seen much of this in model research (empirical evidence is much easier to gather).
[0] http://edwardfeser.blogspot.com/2011/01/against-neurobabble....
[1] http://edwardfeser.blogspot.com/2012/03/reading-rosenberg-pa...
[2] http://edwardfeser.blogspot.com/2017/01/revisiting-ross-on-i...