Observing the world through the lenses of social media

Every day, millions of people across the world take photos and upload them to social media websites. Their goal is to share photos with friends and others, but collectively they are creating vast repositories of visual information about the world. Each photo is an observation of how the world looked at a particular point in time and space. Aggregated together, these photos could provide new sources of observational data for use in disciplines like biology, earth science, social science or history. This project is investigating the algorithms and technologies needed for mining these large collections of photographs and noisy metadata to draw inferences about the physical world. The project has four research thrusts: (1) investigating techniques for identifying and correcting noise in metadata like geo-tags and timestamps, (2) developing algorithms for extracting semantic information from images and metadata, (3) creating methods for robust aggregation of noisy evidence from multiple photos, (4) validating these techniques on interdisciplinary applications in biology, sociology, and earth science.

This project is funded by a National Science Foundation CAREER award, “CAREER: Observing the world through the lenses of social media,” IIS-1253549, 3/1/2013-2/28/2020.

People

Principal Investigator:

Current students:

  • Shujan Naha, Computer Science Ph.D. student
  • Satoshi Tsutsui, Informatics Ph.D. student
  • Jingya Wang, Computer Science Ph.D. student
  • Violet Xiang, Computer Science M.S. student
  • Ishtiak Zaman, Computer Science Ph.D. student
  • Zehua Zhang, Computer Science Ph.D. student

Alumni:

  • Sven Bambach, Ph.D. (2016), now Research Scientist at Nationwide Children’s Hospital
  • Dennis Chen, Summer 2014 REU participant from Olin College, now at Google
  • Demetris Coleman, Summer 2015 REU participant from Auburn University, now Ph.D. student at Michigan State
  • Kun Duan, Ph.D. (2014), now at Quibi
  • Chenyou Fan, Ph.D. (2019), now at Google
  • Tayla Frizell, Summer 2015 REU participant from Mississippi Valley State University
  • Gustavo Goncalves, Summer 2014 REU participant from Dillard University
  • Emmanuel Klutse, Summer REU participant from Tougaloo College
  • Mohammed Korayem, Ph.D. (2015), now at CareerBuilder
  • Stefan Lee, Ph.D. (2016), now Assistant Professor at Oregon State University
  • Alan Lu, Summer REU participant from UIUC
  • Benjamin Newman, Computer Science and Cognitive Science B.S. 2016, now Ph.D. student at CMU
  • Ethan Petersen, Summer 2017 REU participant from Rose-Hulman, now at start-up smileML
  • Gerald Pineda, Summer REU participant from Depauw
  • Ramya Rao, M.S. (2018), now at Amazon
  • Tyler Rarick, Summer 2017 REU participant from Rose-Hulman
  • Alex Seewald, Summer 2014 REU participant from Earlham College
  • Joshua Sherfield, Summer 2013 REU participant from Norfolk State University
  • Ali Varamesh, now at KU Leuven
  • Dylan Vener, Summer 2017 REU participant from Rose-Hulman
  • Mingze Xu, Ph.D. (2020), now at Amazon
  • Haipeng Zhang, Ph.D. (2014), now at Shanghai Tech

Collaborators:

Publications

The following publications are directly related to this project:

Courses and educational materials

So far, the PI has offered five courses that are related to this project. We make lecture notes and other materials available here with the hope that they may be useful to others.

    • Probabilistic Approaches to Artificial Intelligence (CS B553, Spring 2013).
      Uncertainty is a fact of everyday life, caused in part by incomplete and noisy observations, imperfect models, and (apparent) nondeterminism of the social and physical world. Much (and in some cases most) recent work across a range of computing disciplines (including artificial intelligence, robotics, computer vision, natural language processing, data mining, information retrieval, bioinformatics, etc.) has used probabilistic frameworks to explicitly address this uncertainty. This course will introduce the statistical, mathematical, and computational foundations of these frameworks, with a particular focus on a popular and very general framework called Probabilistic Graphical Models. We will also cover related topics in optimization and probability theory. We will study applications of these techniques across a range of AI disciplines, with perhaps a bias towards computer vision, and students will be encouraged to choose a final project that aligns with their own research interests.
    • Search Informatics (Info I427, Fall 2013, Fall 2014, Fall 2015).
      Web search is not only one of the hottest areas in IT, but it has also become an integral part of our society. For example, in the span of about a decade Google has transformed from a small academic research project run by two graduate students to become the 14th largest public corporation in the world, with a market capitalization of nearly $200 billion, and whose search engine receives nearly 34,000 visits per *second*. How does Google manage to search though over a trillion pages and return a result to your query in a fraction of a second? How does it find new webpages as people create them? How does it decide how to rank the pages it returns? How do the choices that Google
      makes impact all of our lives? What is the future of web search? This course studies how Google, Yahoo and Bing work “under the hood”. Students master techniques and tools to automatically crawl, parse, index, store and search Web information, organizing knowledge that can help meet the needs of organizations, communities and individual users. We also discuss the social, security, and business impact of search engine technology. As a project, students build a functioning search engine and compare it with Google or Yahoo. Along the way, students learn how to hack Web applications in Perl.
    • Informatics Research Methods for Undergraduates (Info I399, Fall 2013).
      This course is designed to introduce students to scientific research, specifically in the fields of Informatics. The course will introduce important concepts, methods, and techniques in performing research, including identifying research questions, conducting literature reviews, designing surveys and experiments, performing statistical analysis, managing team projects, using collaborative tools, and reporting results to the public through papers, posters, and videos. The course will also introduce students to various research areas in informatics, information science, and computer science. Class sessions will consist of a mixture of lectures, guest presentations, discussions and activities, and group work time. The class is centered around a semester-long team research project, in which students go through all stages of research from defining a concrete research question all the way to reporting results to the public. Each team will be assigned a graduate student mentor who will serve as a consultant to assist the team throughout the course of the research project. Teams will be required to submit a number of deliverables throughout the course of the semester, including: (1) a project proposal and presentation, (2) a midterm project review report and presentation, and (3) a final project report, presentation, poster, and video. Please check out the gallery of finished projects!
    • Computer Vision (CS B657, Spring 2014, 2016, 2017, 2018, 2019). This is an introductory-level graduate course that broadly introduces the field of computer vision. The course covers vision broadly, from fundamental signal processing through the latest approaches to key problems like recognition, stereo, and tracking. The course has also evolved to focus more on deep learning and machine learning in general, since these are the core interests of most students and the community at large. The course centers around 4 challenging, open-ended programming projects, and a final project in which students complete a research project of their choice and presente it at a public poster session. A gallery of the projects from this class is available.
  • Elements of Artificial Intelligence (CS B551, Fall 2015 and 2016, Spring 2017, Fall 2018). This is a graduate-level course that covers both AI fundamentals and cutting-edge techniques in machine learning, probabilistic inference, deep learning, and applications including computer vision and natural language processing. In addition to the residential section, the PI developed a new version of B551 for online data science students, including redesigned lectures, projects, and activities for the online format. The new version features video lectures organized in 5-10 minute segments, separated by short activities, in an attempt to replicate an in-person active learning classroom environment.

Videos


Posters and slides


Downloads and demos

Broader Impacts

The project is laying the foundation for using visual social media as a new source of observational data for a variety of scientific disciplines. The educational component is preparing students for the next generation of “big data” jobs through new undergraduate and graduate courses and online instructional materials. Undergraduate students (particularly from under-represented groups) are recruited to participate in the research program and encouraged to pursue scientific careers. An annual workshop is planned to educate general audiences, particularly senior citizens, about data mining and social media. Source code, datasets, course materials, and other results of the project will be disseminated to the public via this website.

Examples of outreach and dissemination activities organized or conducted by the PI include:

Awards and press

  • Our work on lifelogging privacy was covered in MIT Tech Review, Fast Company (and again), and Claims Journal.
  • Our paper “Enhancing lifelogging privacy by detecting screens” won an honorable mention award at CHI 2016.
  • Our paper “Objects in the center: How the infant’s body constrains infant scenes” won best paper at ICDL-EpiRob 2016.
  • Please see our lab’s in-the-media page for more articles and interviews.

Point of contact

For more information, please contact PI David Crandall.

Acknowledgement

This material is based upon work supported by the National Science Foundation under Grant No. 1253549. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

[lastupdated]