Lessons learned from USAID’s first crowdsourced project

[Guest post by Timo Luege – I’m passionate about information, communication and how they can be used to make the world a better place.
My two main areas of expertise are:
• Communication through digital media
• Media relations during disasters
Over the last thirteen years I have worked for the Red Cross Red Crescent Movement, the UN, German national public radio and a wire agency Social Media 4 Good] on June 29 – 2012

On June 1st, USAID launched its first ever crowdsourcing project. Yesterday, they shared their lessons learned during a webcast, in a power-point and a case study. Here are the main takeaways.

The USAID project was a little different from what many people have in mind when they think about crowdsourcing; it did not involve Ushahidi, nor did it have anything to do mapping data submitted by beneficiaries. Instead it was a clean-up operation for data that was too messy for algorithms to comprehend.

USAID wanted to share the locations of regions around the world where it had made loans available. The problem was that these locations were not captured in a uniform format. Instead, different partners had submitted the information in different formats. In order to show all loans on a map, USAID needed a uniform data structure for all locations.

Combined human-machine approach

Before sharing the data with the volunteers, USAID had already tried to clean the data with scripts. This meant that only the datasets remained, that were too difficult to be treated automatically.

I  like that USAID did not simply outsource all the data to the crowd, but used human intelligence only for the cases that were too hard for the algorithm. This demonstrates that human capacity is seen as a valuable resource that should only be requested and used where it can have the highest impact.

Locations before and after the cleanupLocations before and after the clean-up


Humans more accurate than algorithms

After the project, USAID asked the GISCorps to take a random sample of the machine-generated records as well as the human-generated records and compare their accuracy. According to analysis, the volunteers were more accurate than the machines, even though most volunteers weren’t GIS experts:

While 85 per cent of the records cleaned up by the volunteers were accurate, only 64 per cent of the records treated by the algorithm were correct. The volunteers were also much faster than expected – instead of the predicted three days, it only took the volunteers 16 hours to go through the data.

Comparatively little room for “creativity”

As one of the volunteers involved in the clean-up operation, I think that one of the reasons for the high accuracy rate was that the project was very focused and didn’t leave the volunteers a lot of room to be “creative”. USAID asked us to do something very specific and gave us a tool that only allowed us to operate within very restrictive parameters: during the exercise, each volunteer requested five or ten datasets that were shown in a mask where he could only add the requested information. This left very little room for potential destructive errors by the users. If USAID had done this through a Google Spreadsheet instead, I’m sure the accuracy would have been lower.

My takeaway from this is that crowdsourced tasks have to be as narrow as possible and need to use tools that help maintain data integrity.

Walk, crawl, run

Prior to the project launch, USAID ran incrementally larger tests that allowed them to improve the workflow, the instructions (yes, you need to test your instructions!) and the application itself.

Tech support! 

If you ask people in 24 time zones to contribute to a project, you also need to have 24 hour tech support. It is very frustrating for volunteers if they cannot participate because of technical glitches.

Volunteers in the Washington DC area could come to the USAID office to take part in the crowdsourcing project (Photo: Shadrock Roberts)Volunteers in the Washington DC area could come to the USAID office to take part in the crowdsourcing project (Photo: Shadrock Roberts)

It’s a social experience

This was emphasized a few times during the webcast and I think it’s an extremely important point: people volunteer their time and skills because they enjoy the experienceof working on a joint project together. That means you also have to nurture and create this feeling of belonging to a community. During the project duration, multiple Skype channels were run by volunteer managers where people could ask questions, exchange information or simply share their excitement.

In addition, USAID also invited volunteers from the Washington DC area to come to their office and work from there. All of this added to making the comparatively boring task of cleaning up data a fun, shared experience.

You need time, project managers and a communications plan

During the call USAID’s Shadrock Roberts said that he “couldn’t be happier” with the results, particularly since the costs of the whole project to the agency were “zero Dollars”. But he also emphasized that three staff members had to be dedicated full time to the project. So while USAID didn’t need a specific budget to run the project, it certainly wasn’t free.

To successfully complete a crowdsourcing project, many elements need to come together and you need a dedicated project manager to pull and hold it all together.

In addition to time needed to organize and refine the technical components of the project, you also need time to motivate people to join your project. USAID reached out to existing volunteer and tech communities, wrote blog post and generated a buzz about the project on social media –  in a way they needed to execute a whole communications plan.

Case study and presentation

USAID published a very good case study on the project which can be downloaded here. It is a very practical document and should be read by anyone who intends to run a crowdsourced project.

In addition, here is the presentation from yesterday’s call:

PPT credit Shadrock Roberts/Stephanie Grosser – USAID

The entire case study was presented by Roberts, Grosser and Swartley at the Wilson Center 7/28/2012. The event was livestreamed:
Event video courtesy of the Wilson Center

Beyond Brute Force: Unexpected Lessons from Crowdsourcing Satellite Imagery Analysis for UNHCR in Somalia

The recent SBTF effort to identify IDP shelters in Somalia for the UNHCR has been notable for several reasons beyond the fantastic work by our very, very hard working volunteers; some of whom may now need an eye exam and glasses… And I feel that what I’m seeing is an inflection point in the development of crisis mapping (or indeed any form of “live” or “crowdsourced” mapping). It’s the point at which we move beyond the “brute force” method of chopping large tasks into little pieces and disseminating them among a distributed human network and begin reaping the rewards of the process itself, as a collaborative space for learning and outreach. For me, this has been the most unexpected dimension of this project so far and I wanted to share my thoughts here for feedback. 

I am always skeptical of crowdsourced data or, indeed, any data. As a geographer and remote sensor whose focus is enumerating displaced populations, I have to be: skepticism is part of my job. All data contain error, so best to acknowledge it and decide what that error means. There is still a lot of uncertainty around these types of volunteered geographic information; specifically questions over the positional accuracy, precision, and validity of these data among a wide variety of other issues. These quantitative issues are important because the general assumption is that these data will be operationalized somehow and it is, therefore, imperative that they add value to already confusing situations if this enterprise is to be taken seriously in an operational sense . The good news is that research so far show that these “asserted” data are not – a priori – necessarily any worse than “authoritative” data and can be quite good due to the greater number of individuals to correct error.

It was with this thinking in mind that I joined the current SBTF effort and I very much appreciate the willingness of our great colleagues at Tomnod, DigitalGlobe, JRC, and UNHCR to treat this as an experiment to see how well a very large amount of very specific imagery analysis could be performed with crowdsourcing. We are beginning to analyze the data now and will likely be doing so for the next month or so. What has been surprising, however, have been a few new twists along the way that I feel probably are lost in the exclusively quantitative concerns that so many (myself included) focus on.

  1. There is huge potential here for stakeholder engagement and broadening your outreach: In a time of plummeting budgets, building a constituency for what your organization does is paramount. Efforts such as this give the public and chance to get engaged, to take part in your mission in a fairly easy way. Speaking about the involvement of students from her Air Photo classes at the University of Georgia, Dr. Marguerite Madden said that the engagement, “raised awareness of this grave situation and many [students] got online to find out more information about why this is happening and what is being done to help…” Today there are almost 200 more people who are familiar with the UNHCR and its mission in Somalia than there was two weeks ago. That’s one heck of an ancillary benefit, especially considering that a vast majority of the volunteers are students with the energy and the desire to contribute to a project such as this. Which brings me to the second point…
  2. The collaboration may be as important as the data. I have been consistently (and pleasantly) surprised by the rich discussion among the volunteers about virtually every aspect of this project. We specifically set out to include the academic community and, especially, the remote sensing community by engaging with the student chapters of the American Society for Photogrammetry and Remote Sensing (ASPRS) due to their higher level of familiarity with imagery analysis. Columbia University’s New Media Taskforce and the University of Madison-Wisconsin’s Department of Geography were major contributors and geography departments at both George Mason University and The University of Georgia hosted mapping events (tip of the hat to Lawrence Jefferson and Chris Strother for making those happen). As a result, we created a very rich environment for exchange and learning. Dr. Madden jumped at the chance to use her class as an opportunity to introduce un-orthodox platforms for imagery analysis to her students and everyone benefited. They learned how crowdsourcing for imagery analysis could work in a live environment and we got tons of good feedback on everything from our rule-set to the platform from her very bright students. It’s this meeting of “professional science” and “citizen science” that helps foster new developments in how we approach these emerging practices.
  3. It’s not always about fixes, it’s about concepts: while I believe that Linus’ Law is a powerful argument for crowdsourcing, it’s important to note that this not only applied to technical bugs, but conceptual ones. Part of remote sensing involves creating a rule-set to aggregate features on the surface of the Earth into meaningful classes that allow you to say something about how the world is or works. While there are robust, scientific, ways to go about this it is worth emphasizing the fact that every classification scheme (in any science) is situated within a specific context and point of view. It’s entirely possible to have very well thought out classification schemes that have little to do with the lived reality on the ground. It was with profound humility that I read the very insightful questions posted to our working Google Doc by volunteers, some of whom had zero experience with remote sensing and, yet, had very perceptive insights regarding the assumptions made by our classification scheme. For more than just a steady workforce to place dots on a map, the volunteers really put their thinking caps on to get under the hood of both the technical aspects of the effort but also the conceptual ones. It was precisely their perspective as non-experts that gave them the ability to see things in a new way.

We remain committed to a critical analysis regarding the substantive contribution of our effort to UNHCR operations, but the sense of community in our dedicated channels of communication that allowed for such vibrant discussion should also be understood as valuable. While the operational use of projects like this cannot go unexamined, it bears repeating these types of projects offer much more than just an additional set of data but present a unique forum and opportunity for creative collaboration, engagement, and learning.

By keeping these thoughts in mind we can begin to move beyond the “brute force” period in crisis mapping, in which complex and – generally – machine-driven functions are simply distributed to a human network and, instead, expand the meaning of geographic data in these new spaces of engagement.

Many thanks to all who have participated in the project thus far. As always, you are fantastic teachers.

A Thank You Note from AI-USA for the Syria Satellite Imagery Project

We are very grateful to Amnesty International USA (AI-USA) for their support and partnership on the recent Syria Satellite Imagery project. This initiative leveraged high resolution satellite imagery kindly provided by DigitalGlobe, the advanced Tomnod platform and over 70 volunteers from the SBTF Satellite Team to crowdsource evidence of mass atrocities in three key Syrian cities. Here is a very kind thank you note from our counter-part at AI-USA who spearheaded the project with us. We look forward to continuing our collaboration and support of AI-USA’s important work moving forward.


Dear SBTF volunteers–

Though there is much left on our end to be done in relation to the Syria pilot project, I wanted to take a moment and write to express deep gratitude.

The SBTF and Amnesty International are natural partners. Amnesty operates under the principle that—given the tools—people everywhere can act in concert to protect the fundamental rights and inherent dignity of each of us.

In AI’s 50 year history, the methods and means by which we agitate as a crowd has evolved beyond community-based Amnesty chapters, to a truly global movement that is no longer artificially separated into groups of activists/advocates and the people whose rights are at risk.

Article 27 of the Universal Declaration of Human Rights guarantees all people the right to share in scientific advancement, and its benefits. Technology has always changed our world. But represented in your work and efforts is a truly new paradigm of social action for social good…one that at once transcends the structural, geographic, and economic barriers that have segmented the human family for too long, and also leverages the power of Article 27 back onto itself in a reinforcing model of technological innovation and group action.

Sadly, as we know, our efforts in this project will not bring about the end of the widespread and systematic abuses occurring in Syria. Indeed, over the course of the project, the situation on the ground as evolved for the worse, and large swaths of the country are effectively crimes scenes. The collection of evidence—and the path to justice—will be a long term endeavor.

But the fruits of the time and energy you committed to in the Syria pilot will have lasting and permanent implications for how AI and other human rights watchdogs approach documentation of war crimes and crimes against humanity. Through this pilot, we have already learned a great deal about the immense leverage social computation can have in the fight for human rights. I look forward to working with you and our other partners on this pilot to incorporate those valuable lessons into future plans.

And above all else, I look forward to the opportunity to work with you in the very near future, and to great effect.

On behalf of Amnesty International—our staff, our volunteers, our activists, and our partners in Syria and everywhere else we can and must collectively make a difference with this approach—I want to express profound gratitude. And I personally and humbly give thanks.

In Solidarity,
Scott Edwards
Advocacy, Policy, and Research Department
Amnesty International, US

Testing new crowd sourcing software and micro-tasking workflows


Haiti SMS Exercise Briefing from notgeorge on Vimeo.

You can view here the video and audio of the pre-exercise briefing for the Camp Roberts / RELIEF Humanitarian Technology event, which some SBTF volunteers were involved in. The SMS exercise brought together volunteers from around the world to process text messages that were received during the Haiti earthquake in order to test out new crowd sourcing software and micro-tasking workflows.

Connecting the crowd: notes on Sudan Vote Monitor

Here’s one thing I’ve learned from working to support Sudan Vote Monitor (SVM): the crowd is always there, but it needs to be connected to be a source. Sounds simple, but things would have gone very differently if more had been done to connect the crowd to SVM.

A few weeks ago, the SBTF mobilised to provide geo-referencing, media monitoring and technical support to Sudan Vote Monitor. Sudan Vote Monitor is an initiative of the Sudan Institute for Policy Research to use connection technologies in support of the independent monitoring of the referendum by local Civil Society Organizations, local media and the general public. SVM aims to support these groups by deploying an Ushahidi platform that receives reports via email, sms and web. All reports are mapped by volunteers and posted to the SVM website in real time. SVM also produces summary blog posts of reports received.

The potential for a citizen reporting initiative in Sudan is strong. In a country with large distances, a system that allows for local monitors and the general public to get their story out can be very powerful. And yet as polls closed for the referendum, the platform had received less than 100 reports. Perhaps this is not a tool that is well suited to the Sudanese context after all? And yet SVM also deployed during the elections in Sudan in April and received over 200 reports from the ground, despite having its shortcode blocked for a number of days. So what was different this time round?

Few reports could be a reflection of a calm referendum with little to report, but that’s probably not the whole story. Lack of preparedness on the ground was a big problem. There was no shortcode this time, only an international long code. The local CSOs and local media whose monitoring work SVM planned to support were not contacted until very close to the referendum. Local CSOs had an established workflow by this point, and integrating SVM’s support at such a late stage proved difficult. Likewise, the local media already had a full program for referendum coverage, and was not given enough time to plan coverage of the SVM message.

This doesn’t mean we should give up on SVM. The referendum vote may be over, but there is still the referendum results, the (separate) Abyei referendum, popular consultations in Blue Nile and Southern Kordofan and state elections in Southern Kordofan to come. If separation is the result of the referendum, there will also be an important transition period. In short, over the next six months Sudanese civil society would benefit from a platform that enables citizen reporting from the ground. In fact, as individuals and organizations have found out about SVM, the team has received very positive feedback on the potential for this initiative.

Over the past week, the SVM team has built up connections with a number of local CSOs.  One great outcome of deploying for the referendum is that if reports begin to arrive as a result of these new connections, there will already be a tested platform and a group of online volunteers to support it. In the past week, SBTF volunteers have shown an incredible capacity to mobilize quickly and respond flexibly to SVM’s request for support. The GPS team responded to the mapping request from SVM efficiently, but more importantly when workload was low, looked for other ways to enhance the mapping interface – adding boundary layers, mapping out of country polling stations. The media monitoring team organized itself within hours to respond to a last minute request from SVM and are responsible for about half the reports entered in the platform.

This is the SBTF’s first live deployment, and it’s been a great learning experience. Deployment of SBTF teams in a relatively quiet operation gave us a chance to further test the workflows and protocols, and to train more volunteers. The lessons on the importance of preparedness on the ground will carry through to other deployments. SBTF volunteers will most often be working to support locally driven projects. Our focus will remain on providing online support to cover certain tasks – technical set up, geo-referencing, translation, analysis and media monitoring. However, as we learn from deployments, we will be able to discern whether the key steps to preparing for effective crowdsourced reporting of a poll have been fulfilled. We hope that SVM and other similar initiatives will also learn from these experiences.