Combining Deep Learning and Case-based Reasoning for Robust, Accurate, Explainable Classification

Artificial Intelligence has made dramatic progress over just the last few years due primarily to deep machine learning using artificial neural networks. While traditional AI approaches require programmers to design hand-crafted, problem-specific rules for extracting features, deep learning uses models and algorithms that directly learn from low-level data, delivering unprecedented accuracy on a wide variety of problems. Despite these successes, however, state-of-the-art deep learning fails for important classes of problems. For example, deep learning-based image recognition struggles to detect objects that are small, that require fine-grained reasoning about appearance, that lack distinctive visual features, or for which few training examples exist to train a network’s vast number of parameters. Even with large training datasets, deep networks can be confused by minor noise caused randomly or by an adversary, and cannot explain their decisions to help debug these failures. These shortcomings will prevent deep learning from being safely applied in many domains that could most benefit from AI, including military, intelligence, security, medical, and legal applications where decisions must be justified, data is scarce, and the consequences of failure are great. 

This project is investigating a path towards reliable, explainable, accurate machine learning by uniting deep learning with case-based reasoning (CBR), which are models that explicitly encode domain knowledge including common sense and human expertise. CBR makes decisions by consulting key training examples seen in the past through domain-specific similarity rules, adapting them to the target problem through adaptation rules, testing this solution, and using feedback to decide to add the example to the knowledge base. This approach has many advantages over deep learning, because it: (1) allows human expertise to be directly encoded by asking them to write cases and adaptation/similarity rules, (2) can reason from limited training data given appropriate adaptation and similarity rules, (3) improves over time in an inertia-free way without requiring retraining, (4) can explain its answers by citing the cases and adaptation rules that it used, and (5) mimics the prototype theory that humans are thought to use. Despite these advantages and numerous practical successes of CBR, deep learning has become much more popular because it delivers better accuracy on many practical problems when large datasets are available, can better discover statistical patterns that may not be apparent to a human, and high-quality, easy-to-use software frameworks are available.

We are investigating tight couplings of deep learning with case-based reasoning in order to directly inject domain knowledge into classification algorithms, taking advantage of the complementary strengths of these two approaches, in order to produce highly accurate classifiers that incorporate human expertise, are more robust to noisy exemplars, better tolerate small and biased training sets, and explain their decisions. The challenge is how to combine two techniques that have largely been studied and developed separately, with minimal communication or collaboration between the two communities, in part because they have largely focused on different application domain. Our team is ideal to address this gap, consisting of two established PIs with complementary expertise who nonetheless have been close colleagues for nearly a decade. The team will also include postdocs and Ph.D. students with developing expertise in deep learning and case-based reasoning.

This project is funded by the Office of Naval Research, “Combining Deep Learning and Case-based Reasoning for Robust, Accurate, Explainable Classification,” Oct 2019 – Sept 2022.


Principal Investigators:

Current students:

  • Yue Chen, Ph.D. student in Computational Linguistics
  • Larry Gates, Ph.D. student in Computer Science
  • Chang Liu, M.S. student in Computer Science
  • Vibhas Vats, M.S. student in Computer Science
  • Xizi Wang, Ph.D. student in Computer Science
  • Kaithleen Wilkerson, Ph.D. student in Computer Science
  • Zachary Wilkerson, Ph.D. student in Computer Science
  • Xiaomeng Ye, Ph.D. student in Computer Science
  • Ziwei Zhao, Ph.D. student in Computer Science


The following publications are directly related to this project:

Full lists of publications by PI Crandall and PI Leake are also available.

Courses and educational materials

The PIs teach several courses that are related to the project. We make lecture notes and other materials available here with the hope that they may be useful to others.

  • Probabilistic Approaches to Artificial Intelligence.
    Uncertainty is a fact of everyday life, caused in part by incomplete and noisy observations, imperfect models, and (apparent) nondeterminism of the social and physical world. Much (and in some cases most) recent work across a range of computing disciplines (including artificial intelligence, robotics, computer vision, natural language processing, data mining, information retrieval, bioinformatics, etc.) has used probabilistic frameworks to explicitly address this uncertainty. This course will introduce the statistical, mathematical, and computational foundations of these frameworks, with a particular focus on a popular and very general framework called Probabilistic Graphical Models. We will also cover related topics in optimization and probability theory. We will study applications of these techniques across a range of AI disciplines, with perhaps a bias towards computer vision, and students will be encouraged to choose a final project that aligns with their own research interests.
  • Computer Vision. This is an introductory-level graduate course that broadly introduces the field of computer vision. The course covers vision broadly, from fundamental signal processing through the latest approaches to key problems like recognition, stereo, and tracking. The course has also evolved to focus more on deep learning and machine learning in general, since these are the core interests of most students and the community at large. The course centers around 4 challenging, open-ended programming projects, and a final project in which students complete a research project of their choice and presente it at a public poster session. A gallery of the projects from this class is available.
  • Elements of Artificial Intelligence. This is a graduate-level course that covers both AI fundamentals and cutting-edge techniques in machine learning, probabilistic inference, deep learning, and applications including computer vision and natural language processing. In addition to the residential section, the PI developed a new version of B551 for online data science students, including redesigned lectures, projects, and activities for the online format. The new version features video lectures organized in 5-10 minute segments, separated by short activities, in an attempt to replicate an in-person active learning classroom environment.

Broader Impacts

Examples of outreach and dissemination activities organized or conducted by the PIs include:

  • The AI Talk Series, a new weekly seminar for topics related to computer vision, artificial intelligence, machine learning, data mining, and other broadly-related areas. Videos of talks are on made publicly available on the web to enhance the reach of the seminar.
  • Talks and workshops at IU Mini University, a summer returning education program for alumni and community members.

Point of contact

For more information, please contact PI David Crandall or PI David Leake


This material is based upon work supported by the Office of Naval Research grant N00014-19-1-2655. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. Government.

Last updated: August 14, 2020 at 15:44 pm
The IU Computer Vision Lab's projects and activities have been funded, in part, by grants and contracts from the Air Force Office of Scientific Research (AFOSR), the Defense Threat Reduction Agency (DTRA), Dzyne Technologies, EgoVid, Inc., ETRI, Facebook, Google, Grant Thornton LLP, IARPA, the Indiana Innovation Institute (IN3), the IU Data to Insight Center, the IU Office of the Vice Provost for Research through an Emerging Areas of Research grant, the IU Social Sciences Research Commons, the Lilly Endowment, NASA, National Science Foundation (IIS-1253549, CNS-1834899, CNS-1408730, BCS-1842817, CNS-1744748, IIS-1257141, IIS-1852294), NVidia, ObjectVideo, Office of Naval Research (ONR), Pixm, Inc., and the U.S. Navy. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.