Abstract: How do we design AI to augment collective innovation? The wisdom of crowds hinges on the independence and diversity of their members’ information and approach.

Here I explore how the wisdom of scientific and technological crowds for sustained discovery and invention operate through a process of collective abduction — the collision of deduction and induction — wherein unexpected findings stimulate innovators to forge new insights to make the surprising unsurprising. Drawing on tens of millions of research papers and patents across the life sciences, physical sciences and patented inventions, here I show that surprising designs and discoveries are the best predictor of outsized success and that surprising advances systematically emerge across, rather than within researchers or teams; most commonly when innovators from one field surprisingly publish problem-solving results to an audience in a distant and diverse other. This relates to other research I describe that shows how across innovators, teams and fields, connection and conformity is associated with both reduced replication and impeded innovation.

Using these principles, I simulate processes of scientific and technological search to demonstrate the relationship between crowded fields and constrained collective inferences, and I demonstrate how inverting the traditional artificial intelligence approach to avoid rather than mimic human search enables the design of diversity that systematically violates established field boundaries and is associated with marked success of innovation predictions. I conclude with a discussion of prospects and challenges in a connected age for sustainable innovation through the design and preservation of difference.

Bio: James Evans is Max Palevsky Professor of Sociology, Director of Knowledge Lab and the Computational Social Science program at the University of Chicago and the Santa Fe Institute.