Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks

Authors: Satoshi Tsutsui and David Crandall.
Diagram showing a Compound figure being passed into a CNN, to create a version of the figure with bounding boxes around each component of the figure.

Abstract

A key problem in automatic analysis and understanding of scientific papers is to extract semantic information from non-textual paper components like figures, diagrams, tables, etc. This research always requires a very first preprocessing step: decomposing compound multi-part figures into individual subfigures. Previous work in compound figure separation has been based on manually designed features and separation rules, which often fail for less common figure types and layouts. Moreover, no implementation for compound figure decomposition is publicly available.

This paper proposes a data driven approach to separate compound figures using modern deep Convolutional Neural Networks (CNNs) to train the separator in an end-to-end manner. CNNs eliminate the need for manually designing features and separation rules, but require large amount of annotated training data. We overcome this challenge using transfer learning as well as automatically synthesizing training exemplars. We evaluate our technique on the ImageCLEF Medical dataset, achieving 85.9% accuracy and outperforming manually engineered previous techniques. We made the resulting approach available as an easy-to-use Python library, aiming to promote further research in scientific figure mining.

Resources

– Github code and pre-trained model: https://github.com/apple2373/figure-separator
— easy to use figure separator!
– Preprint of the paper: https://arxiv.org/abs/1703.05105
– Citation: If you use the figure separator in your research or found it useful, please consider to cite:
@inproceedings{figure-separate,
title={{A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks}},
author={Satoshi Tsutsui, and David Crandall},
booktitle={{The IAPR International Conference on Document Analysis and Recognition (ICDAR)}},
year={2017}
}

Examples

Example of algorithm output, showing figure components with red bounding boxes around them. Example of algorithm output, showing figure components with red bounding boxes around them.
Example of algorithm output, showing figure components with red bounding boxes around them.Example of algorithm output, showing figure components with red bounding boxes around them.

The IU Computer Vision Lab's projects and activities have been funded, in part, by grants and contracts from the Air Force Office of Scientific Research (AFOSR), the Defense Threat Reduction Agency (DTRA), Dzyne Technologies, EgoVid, Inc., ETRI, Facebook, Google, Grant Thornton LLP, IARPA, the Indiana Innovation Institute (IN3), the IU Data to Insight Center, the IU Office of the Vice Provost for Research through an Emerging Areas of Research grant, the IU Social Sciences Research Commons, the Lilly Endowment, NASA, National Science Foundation (IIS-1253549, CNS-1834899, CNS-1408730, BCS-1842817, CNS-1744748, IIS-1257141, IIS-1852294), NVidia, ObjectVideo, Office of Naval Research (ONR), Pixm, Inc., and the U.S. Navy. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.