Mining Photo-sharing Websites to Study Ecological Phenomena

Haipeng Zhang, Mohammed Korayem, David CrandallGretchen LeBuhn

The popularity of social media websites like Flickr and Twitter has created enormous collections of user-generated content online. Latent in these content collections are observations of the world: each photo is a visual snapshot of what the world looked like at a particular point in time and space, for example, while each tweet is a textual expression of the state of a person and his or her environment. Aggregating these observations across millions of social sharing users could lead to new techniques for large-scale monitoring of the state of the world and how it is changing over time. In this paper we step towards that goal, showing that by analyzing the tags and image features of geo-tagged, time-stamped photos we can measure and quantify the occurrence of ecological phenomena including ground snow cover, snow fall and vegetation density. We compare several techniques for dealing with the large degree of noise in the dataset, and show how machine learning can be used to reduce errors caused by misleading tags and ambiguous visual content. We evaluate the accuracy of these techniques by comparing to ground truth data collected both by surface stations and by Earth observing satellites. Besides the immediate application to ecology, our study gives insight into how to accurately crowd-source other types of information from large, noisy social sharing datasets.

For more details, please see our WWW 2012 paper and slides from our WWW talk.

Videos

Sample videos showing appearance estimates produced by the Flickr photo analysis described in this paper. Green indicates high probability of appearances, and grey and black indicate low-confidence areas (with few photos or ambiguous evidence).

Butterfly Leaves Dragonfly
Snow

Media coverage

Snow Snaps Give You a Better Weather Picture:

Results

Photos by different people are (almost) independent observations, with uncorrelated noise

Estimate daily snow cover for individual cities

Estimate daily snow quantity for individual cities

Philadelphia Boston New York Chicago
RMS error (inch) 1.44 1.26 1.15 1.06

 

Estimate snow cover on each day at each place in North America

  • For each geographic bin of size 1° x 1° (~35 million total decisions)
  • Use ground truth data from Terra satellite

 

Papers and presentations

BibTeX entries:

@inproceedings{snow2013iccvw,
    author = {Jingya Wang and Mohammed Korayem and David Crandall},
    title = {Observing the natural world with {F}lickr},
    booktitle = {International Conference on Computer Vision Workshop on Computer Vision for Converging Perspectives},
    year = {2013}
}

@inproceedings{ecology2012www,
    author = {Haipeng Zhang and Mohammed Korayem and David Crandall and Gretchen LeBuhn},
    title = {Mining Photo-sharing Websites to Study Ecological Phenomena},
    booktitle = {International World Wide Web Conference (WWW)},
    year = {2012}
}

Acknowledgements

We thank Professor Michael Trosset for discussions on the linear regression models. We also gratefully acknowledge the support of the following:

Lilly Endowment National Science Foundation IBM
Lilly Endowment IU Data to Insight Center National Science Foundation IBM
The IU Computer Vision Lab's projects and activities have been funded, in part, by grants and contracts from the Air Force Office of Scientific Research (AFOSR), the Defense Threat Reduction Agency (DTRA), Dzyne Technologies, EgoVid, Inc., ETRI, Facebook, Google, Grant Thornton LLP, IARPA, the Indiana Innovation Institute (IN3), the IU Data to Insight Center, the IU Office of the Vice Provost for Research through an Emerging Areas of Research grant, the IU Social Sciences Research Commons, the Lilly Endowment, NASA, National Science Foundation (IIS-1253549, CNS-1834899, CNS-1408730, BCS-1842817, CNS-1744748, IIS-1257141, IIS-1852294), NVidia, ObjectVideo, Office of Naval Research (ONR), Pixm, Inc., and the U.S. Navy. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.