If the glass is half full, that means it’s also half empty.
After finishing my doctorate I did a postdoc in an AI lab. These were the early, heady days of expert systems, a technology predicated on making explicit the tacit knowledge of human experts, converting the heuristics of human decision-making into conceptual objects and rules for manipulating them that could be run on computers. Our core group consisted of cognitive psychologists and computer scientists, and in building systems we would collaborate with “domain experts” in medicine, business, law, engineering, and other practical disciplines. A standard division of labor was established: the domain experts provided the expertise; the psychologists did the “knowledge engineering,” which consisted of making explicit what the experts knew and how they used that knowledge; the computer scientists designed and built the computer systems encoding the engineered expert knowledge.
Early on I came to a sobering realization: human experts aren’t nearly as good as computers at using knowledge. Humans have limited processing capacity, and so they can’t remember very many things at once, can’t pay attention to very many features of the task in front of them, can’t deal with very many variables at the same time. To compensate for their limitations, humans take various short-cuts and work-arounds in solving complex problems. Computers have limitations too, especially in their ability to acquire new knowledge, but in their ability to process lots of information they vastly outperform humans. Equipped with knowledge already learned by human experts, computers can manipulate this knowledge more efficiently, and more accurately, than can the human experts.
I remember giving a talk in DC to a gathering of all the AI postdocs funded under the same national grant program, working in labs at MIT, Harvard, Stanford, U. of Minnesota, UC San Diego, maybe others (my memory has degraded since then). Most of the talks were about AI work in progress. I talked about the differences between human and computer decision-making. Instead of fancy slides I drew overheads by hand with a black marker. I drew out a simple binary decision tree that went maybe 7 layers deep, pointing out ways in which knowledge and logic interact in actual decision-making tasks, describing how computers are not vulnerable to the same sorts of biases as humans in working through even a fairly simple decision. I remember one of the colleagues at my university telling me afterward that he thought my talk sucked. But I also remember discussing the implications of my presentation with the overall head of the grant program nationwide and one of the pioneering figures in expert systems. It turned out that his group was moving away from having computers imitate human heuristic knowledge toward more reliance on what computers are best at: manipulating numerical information via quantitative algorithms.
While I did some work on a pediatric cardiology expert system, I spent most of my time as a postdoc doing knowledge engineering on two other projects. One was a system for designing so-called fractional factorial experiments, where the domain expert was a statistics professor in the business school. The other was a system for making credit decisions, the domain expert being a professional credit analyst in the insurance industry. In both cases, through conversation and observation, I was gradually able to identify the information the experts looked for in the “task domain” and the ways in which they used this information to render decisions. As had been the case in other domains, these experts used short-cuts and rules of thumb to compensate for human processing limitations. I put together alternative “inference engines” for both of these task domains, with decision-making processes predicated on the heavy number-crunching capacity of computers. I also went ahead and did the programming on both of these systems.
The results should have been predictable. Both the experimental design system and the credit rating system were excellent at performing their respective tasks. Where it was possible to evaluate their decisions in comparison with the “right” answers, the computer systems outperformed the human experts. The human experts acknowledged their machinic doubles’ excellence, even at times conceding their superiority. But they didn’t trust these hybrid expert systems, using their own human knowledge but processing it algorithmically rather than heuristically. They couldn’t understand how these systems thought, how they arrived at their decisions. The systems’ reasoning procedures, more efficient, more consistent, and arguably more accurate than their own, were too opaque, too alien for the human experts to grasp. I concluded that the only way systems like the ones I built would ever be used in real-world decision-making would be if the human experts weren’t sitting around looking over the expert systems’ shoulders second-guessing their decisions. You would need lower-level human technicians to feed the computer systems with data, to read the output, and to enact the systems’ decisions without constantly grousing about robots ruling the world and all the rest of the tedious all-too-human resentment my systems seemed to provoke.