Every day, millions of people across the world take photos and upload them to social media websites. Their goal is to share photos with friends and others, but collectively they are creating vast repositories of visual information about the world. Each photo is an observation of how the world looked at a particular point in time and space. Aggregated together, these photos could provide new sources of observational data for use in disciplines like biology, earth science, social science or history. This project is investigating the algorithms and technologies needed for mining these large collections of photographs and noisy metadata to draw inferences about the physical world. The project has four research thrusts: (1) investigating techniques for identifying and correcting noise in metadata like geo-tags and timestamps, (2) developing algorithms for extracting semantic information from images and metadata, (3) creating methods for robust aggregation of noisy evidence from multiple photos, (4) validating these techniques on interdisciplinary applications in biology, sociology, and earth science.
This project is funded by a National Science Foundation CAREER award, “CAREER: Observing the world through the lenses of social media,” IIS-1253549, 3/1/2013-2/28/2018.
- David Crandall, Indiana University
- Sven Bambach, Computer Science Ph.D. student
- Ramya Rao, Computer Science M.S. student
- Jingya Wang, Computer Science Ph.D. student
- Ishtiak Zaman, Computer Science Ph.D. student
- Dennis Chen, Summer 2014 REU participant from Olin College
- Demetris Coleman, Summer 2015 REU participant from Auburn University
- Kun Duan, Ph.D. (2014), now at GE Global Research
- Tayla Frizell, Summer 2015 REU participant from Mississippi Valley State University
- Gustavo Goncalves, Summer 2014 REU participant from Dillard University
- Mohammed Korayem, Ph.D. (2015), now at CareerBuilder
- Stefan Lee, Computer Science Ph.D. student
- Alex Seewald, Summer 2014 REU participant from Earlham College, now Ph.D. student at Indiana University
- Joshua Sherfield, Summer 2013 REU participant from Norfolk State University
- Haipeng Zhang, Ph.D. (2014), now at IBM Research China
The following publications are directly related to this project:
- Mingze Xu, Chenyou Fan, John Paden, Geoffrey Fox, David Crandall, "Multi-Task Spatiotemporal Neural Networks for Structured Surface Reconstruction," IEEE Winter Conference on Applications of Computer Vision (WACV), 2018. (45% acceptance rate) [PDF]
- Noam Levin, Saleem Ali, David Crandall, "Utilizing remote sensing and big data to quantify conflict intensity: The Arab Spring as a case study," Applied Geography, 2018. (impact factor = 2.56) [PDF]
- Mingze Xu, David Crandall, Geoffrey Fox, John Paden, "Automatic estimation of ice bottom surfaces from radar imagery," IEEE International Conference on Image Processing (ICIP), 2017. (Oral, 45.0% acceptance rate) [PDF]
- Wen Chen, David Crandall, Norman Su, "Understanding the Aesthetic Evolution of Websites: Towards a Notion of Design Periods," ACM CHI Conference on Human Factors in Computing Systems (CHI), 2017. (Oral, 25.0% acceptance rate) [PDF]
- Sven Bambach, Zehua Zhang, David Crandall, Chen Yu, "Exploring inter-observer differences in first-person object views using deep learning models," IEEE International Conference on Computer Vision Workshop on Mutual Benefits of Cognitive and Computer Vision, 2017. [PDF]
- Sven Bambach, David Crandall, Linda Smith, Chen Yu, "An Egocentric Perspective on Active Vision and Visual Object Learning in Toddlers," IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), 2017. (Oral, 37.1% acceptance rate) [PDF]
- Chenyou Fan, Jangwon Lee, Mingze Xu, Krishna Kumar Singh, Yong Jae Lee, David Crandall, Michael Ryoo, "Identifying first-person camera wearers in third-person videos," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. (Poster, 29.0% acceptance rate) [PDF] [Project page]
- Bardia Doosti, David Crandall, Norman Makoto Su, "A deep study into the history of web design," ACM International Web Science Conference (WebSci), 2017. (Oral, 35.3% acceptance rate) [PDF] [Project page]
- Scott Workman, Menghua Zhai, David Crandall, Nathan Jacobs, "A unified model for near and remote sensing," IEEE International Conference on Computer Vision (ICCV), 2017. (Poster, 28% acceptance rate) [PDF] [Project page]
- Satoshi Tsutsui, David Crandall, "A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks," IAPR International Conference on Document Analysis and Recognition (ICDAR), 2017. [PDF] [Project page]
- Eman Hassan, Rakibul Hasan, Patrick Shaffer, David Crandall, Apu Kapadia, "Cartooning for enhanced privacy in lifelogging and streaming video," IEEE Conference on Computer Vision and Pattern Recognition Workshop on Computer Vision Challenges and Opportunities for Privacy and Security (CVPR CV-COPS), 2017. [PDF]
- Sven Bambach, David Crandall, Linda Smith, Chen Yu, "Active Viewing in Toddlers Facilitates Visual Object Learning: An Egocentric Vision Approach," Annual Conference of the Cognitive Science Society (CogSci), 2016. (Oral, 34% acceptance rate) [PDF]
- Mohammed Korayem, Robert Templeman, Dennis Chen, David Crandall, Apu Kapadia, "Enhancing Lifelogging Privacy by Detecting Screens," ACM CHI Conference on Human Factors in Computing Systems (CHI), 2016. Honorable Mention Award. (CHI Note, 23.4% acceptance rate) Honorable Mention Award! [PDF] [Project page]
- Sven Bambach, Linda Smith, David Crandall, Chen Yu, "Objects in the Center: How the Infant's Body Constrains Infant Scenes," IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), 2016. Best paper award. (Oral, 33.7% acceptance rate) Best paper award! [PDF]
- Tousif Ahmed, Patrick Shaffer, Kay Connelly, David Crandall, Apu Kapadia, "Addressing Physical Safety, Security, and Privacy for People with Visual Impairments," USENIX Symposium on Usable Privacy and Security (SOUPS), 2016. (Oral, 27.8% acceptance rate) [PDF]
- Jingya Wang, Mohammed Korayem, Saul Blanco, David Crandall, "Tracking Natural Events through Social Media and Computer Vision," ACM International Conference on Multimedia (MM), 2016. [PDF] [Project page]
- Stefan Lee, Senthil Purushwalkam, Michael Cogswell, Viresh Ranjan, David Crandall, Dhruv Batra, "Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles," Advances in Neural Information Processing Systems (NIPS), 2016. (Poster, 22.7% acceptance rate) [PDF]
- Stefan Lee, Haipeng Zhang, David Crandall, "Predicting Geo-informative Attributes in Large-scale Image Collections using Convolutional Neural Networks," IEEE Winter Conference on Applications of Computer Vision (WACV), 2015. (Oral and poster, 36.7% acceptance rate) [PDF] [Project page]
- Tousif Ahmed, Roberto Hoyle, Kay Connelly, David Crandall, Apu Kapadia, "Privacy Concerns and Behaviors of People with Visual Impairments," ACM CHI Conference on Human Factors in Computing Systems (CHI), 2015. (Full paper, 23% acceptance rate) [PDF]
- Sven Bambach, Stefan Lee, David Crandall, Chen Yu, "Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions," IEEE International Conference on Computer Vision (ICCV), 2015. (Poster, 30.3% acceptance rate) [PDF] [Project page]
- Sven Bambach, David Crandall, Chen Yu, "Viewpoint Integration for Hand-Based Recognition of Social Interactions from a First-Person View," ACM International Conference on Multimodal Interaction (ICMI), 2015. (Poster, 41% acceptance rate) [PDF]
- Roberto Hoyle, Robert Templeman, Denise Anthony, David Crandall, Apu Kapadia, "Sensitive Lifelogs: A Privacy Analysis of Photos from Wearable Cameras," ACM CHI Conference on Human Factors in Computing Systems (CHI), 2015. (CHI Note) [PDF]
- Noam Levin, Salit Kark, David Crandall, "Where have all the people gone? Enhancing global conservation using night lights and social media," Ecological Applications, 2015. (impact factor = 4.126) [PDF]
- Stefan Lee, Nicolas Maisonneuve, David Crandall, Alexei Efros, Josef Sivic, "Linking past to present: Discovering style in two centuries of architecture," IEEE International Conference on Computational Photography (ICCP), 2015. (Oral, 24% acceptance rate) [PDF] [Project page]
- Robert Templeman, Mohammed Korayem, David Crandall, Apu Kapadia, "PlaceAvoider: Steering First-Person Cameras away from Sensitive Spaces," Network and Distributed System Security Symposium (NDSS), 2014. (Oral, 18.6% acceptance rate) [PDF]
- Kun Duan, Luca Marchesotti, David Crandall, "Vehicle Recognition with Constrained Multiple Instance SVMs," IEEE Winter Conference on Applications of Computer Vision (WACV), 2014. (Oral and poster, 40% acceptance rate) [PDF]
- Stefan Lee, Jerome Mitchell, David Crandall, Geoffrey Fox, "Estimating Bedrock and Surface Layer Boundaries and Confidence Intervals in Ice Sheet Radar Imagery using MCMC," IEEE International Conference on Image Processing (ICIP), 2014. (Oral, 44% acceptance rate) [PDF] [Project page]
- Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, Apu Kapadia, "Privacy Behaviors of Lifeloggers using Wearable Cameras," ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2014. (Oral, 20.7% acceptance rate) [PDF]
- Kun Duan, David Crandall, Dhruv Batra, "Multimodal Learning in Loosely-organized Web Images," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. (Poster, 29.9% acceptance rate) [PDF]
- Sven Bambach, John Franchak, David Crandall, Chen Yu, "Detecting Hands in Children's Egocentric Views to Understand Embodied Attention during Social Interaction," Annual Conference of the Cognitive Science Society (CogSci), 2014. (Oral, 41.0% acceptance rate) [PDF]
- Sven Bambach, David Crandall, Chen Yu, "Understanding Embodied Visual Attention in Child-Parent Interaction," IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), 2013. (Oral, 33% acceptance rate) [PDF]
- David Crandall, Andrew Owens, Noah Snavely, Daniel Huttenlocher, "SfM with MRFs: Discrete-Continuous Optimization for Large-Scale Structure from Motion," IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2013. (impact factor = 4.795) [PDF] [Project page]
- Jingya Wang, Mohammed Korayem, David Crandall, "Observing the natural world with Flickr," International Conference on Computer Vision Workshop on Computer Vision for Converging Perspectives, 2013. Best paper award winner. (Oral) Best paper award! [PDF]
- Haipeng Zhang, Mohammed Korayem, David Crandall, Gretchen LeBuhn, "Mining Photo-sharing Websites to Study Ecological Phenomena," International World Wide Web Conference (WWW), 2012. (Oral, 12% acceptance rate) [PDF] [Project page]
A full list of our lab’s publications is also available.
Courses and educational materials
So far, the PI has offered five courses that are related to this project. We make lecture notes and other materials available here with the hope that they may be useful to others.
- Probabilistic Approaches to Artificial Intelligence (CS B553, Spring 2013).
Uncertainty is a fact of everyday life, caused in part by incomplete and noisy observations, imperfect models, and (apparent) nondeterminism of the social and physical world. Much (and in some cases most) recent work across a range of computing disciplines (including artificial intelligence, robotics, computer vision, natural language processing, data mining, information retrieval, bioinformatics, etc.) has used probabilistic frameworks to explicitly address this uncertainty. This course will introduce the statistical, mathematical, and computational foundations of these frameworks, with a particular focus on a popular and very general framework called Probabilistic Graphical Models. We will also cover related topics in optimization and probability theory. We will study applications of these techniques across a range of AI disciplines, with perhaps a bias towards computer vision, and students will be encouraged to choose a final project that aligns with their own research interests.
- Search Informatics (Info I427, Fall 2013, Fall 2014, Fall 2015).
Web search is not only one of the hottest areas in IT, but it has also become an integral part of our society. For example, in the span of about a decade Google has transformed from a small academic research project run by two graduate students to become the 14th largest public corporation in the world, with a market capitalization of nearly $200 billion, and whose search engine receives nearly 34,000 visits per *second*. How does Google manage to search though over a trillion pages and return a result to your query in a fraction of a second? How does it find new webpages as people create them? How does it decide how to rank the pages it returns? How do the choices that Google
makes impact all of our lives? What is the future of web search? This course studies how Google, Yahoo and Bing work “under the hood”. Students master techniques and tools to automatically crawl, parse, index, store and search Web information, organizing knowledge that can help meet the needs of organizations, communities and individual users. We also discuss the social, security, and business impact of search engine technology. As a project, students build a functioning search engine and compare it with Google or Yahoo. Along the way, students learn how to hack Web applications in Perl.
- Informatics Research Methods for Undergraduates (Info I399, Fall 2013).
This course is designed to introduce students to scientific research, specifically in the fields of Informatics. The course will introduce important concepts, methods, and techniques in performing research, including identifying research questions, conducting literature reviews, designing surveys and experiments, performing statistical analysis, managing team projects, using collaborative tools, and reporting results to the public through papers, posters, and videos. The course will also introduce students to various research areas in informatics, information science, and computer science. Class sessions will consist of a mixture of lectures, guest presentations, discussions and activities, and group work time. The class is centered around a semester-long team research project, in which students go through all stages of research from defining a concrete research question all the way to reporting results to the public. Each team will be assigned a graduate student mentor who will serve as a consultant to assist the team throughout the course of the research project. Teams will be required to submit a number of deliverables throughout the course of the semester, including: (1) a project proposal and presentation, (2) a midterm project review report and presentation, and (3) a final project report, presentation, poster, and video. Please check out the gallery of finished projects!
- Computer Vision (CS B657, Spring 2014, 2016, 2017). This is an introductory-level graduate course that broadly introduces the field of computer vision. The course covers vision broadly, from fundamental signal processing through the latest approaches to key problems like recognition, stereo, and tracking. The course has also evolved to focus more on deep learning and machine learning in general, since these are the core interests of most students and the community at large. The course centers around 4 challenging, open-ended programming projects, and a final project in which students complete a research project of their choice and presente it at a public poster session. A gallery of the projects from this class is available.
- Elements of Artificial Intelligence (CS B551, Fall 2015 and 2016, Spring 2017). This is a graduate-level course that covers both AI fundamentals and cutting-edge techniques in machine learning, probabilistic inference, deep learning, and applications including computer vision and natural language processing. In addition to the residential section, the PI developed a new version of B551 for online data science students, including redesigned lectures, projects, and activities for the online format. The new version features video lectures organized in 5-10 minute segments, separated by short activities, in an attempt to replicate an in-person active learning classroom environment.
Posters and slides
Downloads and demos
- EgoHands, a large-scale dataset of egocentric video from interacting people, including annotated fine-grained hand segmentation maps.
- An interactive demo of our project on architectural element discovery from large-scale collections of StreetView images.
- See our Discrete-Continuous Optimization for Large-scale Structure from Motion project page to download source code and datasets for our 3d reconstruction papers.
- Trained CNN object models for detecting private content in lifelogging images is available at our ScreenAvoider project page.
- Source code for our ice sheet layer-finding code.
The project is laying the foundation for using visual social media as a new source of observational data for a variety of scientific disciplines. The educational component is preparing students for the next generation of “big data” jobs through new undergraduate and graduate courses and online instructional materials. Undergraduate students (particularly from under-represented groups) are recruited to participate in the research program and encouraged to pursue scientific careers. An annual workshop is planned to educate general audiences, particularly senior citizens, about data mining and social media. Source code, datasets, course materials, and other results of the project will be disseminated to the public via this website.
Examples of outreach and dissemination activities organized or conducted by the PI include:
- The IU Intelligent and Interactive Systems Seminar, a new weekly seminar for topics related to computer vision, artificial intelligence, machine learning, data mining, and other broadly-related areas. Videos of talks are on made publicly available on the web to enhance the reach of the seminar.
- The 1st International Workshop on the Social Web for Environmental and Ecological Monitoring, held in conjunction with ICWSM 2016.
- Talks and workshops at IU Mini University, a summer returning education program for alumni and community members.
Awards and press
- Our paper Observing the natural world through Flickr won best paper at the IEEE International Conference on Computer Vision Workshop on Computer Vision for Converging Perspectives.
- Our paper This Hand is My Hand: A Probabilistic Approach to Hand Disambiguation in Egocentric Video won best paper at the IEEE International Conference on Computer Vision and Pattern Recognition EgoVision workshop.
- Our work on snow detection was covered in brief articles in New Scientist and CACM.
- Our work on lifelogging privacy was covered in MIT Tech Review, Fast Company (and again), and Claims Journal.
- Our paper “Enhancing lifelogging privacy by detecting screens” won an honorable mention award at CHI 2016.
- Our paper “Objects in the center: How the infant’s body constrains infant scenes” won best paper at ICDL-EpiRob 2016.
- Please see our lab’s in-the-media page for more articles and interviews.
Point of contact
For more information, please contact PI David Crandall.
This material is based upon work supported by the National Science Foundation under Grant No. 1253549. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.Last updated: January 30, 2017 at 1:53 am