Ideas

Mobile Text Surveys: A Smarter Way to Measure Conservation’s Impacts on People?

September 30, 2013

Follow Eddie
Samburu woman with her cell phone. Image credit: Eddie Game/TNC.
Samburu woman with her cell phone. Image credit: Eddie Game/TNC.

Conservation today has a lot riding on being able to demonstrate the positive impact of its work not just on nature, but also on human well-being, especially in developing countries. Donors, other funders, governments and potential partners are demanding that our work be sensitive to people’s welfare and even enhance it if possible. If we don’t prove we’re having that positive impact, we risk being shut out of many global conversations and dynamics.

One problem: The way we’ve been measuring the impact of our work on people — through in-person household surveys — can often be intrusive, expensive and unreliable. It can take hours to complete one of these surveys. Subjects sometimes tell surveyors what the subjects think they want to hear. And it’s impractical to survey people frequently.

Enter the mobile phone — especially the SMS (short message service) text.

Mobile phone use in developing countries has exploded at a rate unmatched by any other technology in history. A person in a developing nation is more likely to have access to a mobile phone than clean water or electricity. 77% of the world’s 6 billion active mobile phone subscriptions are in the developing world.

They also use SMS in overwhelming numbers. About 96% of Indonesia’s mobile owners use SMS regularly. In Kenya, it’s 89%, Mexico 82%, and China 80% (World Bank 2012).SMS is the lowest common denominator in phone technology — all phones are capable of it.

The availability of mobile phones in the developing world is radically changing the approach to data collection and information delivery in fields like health and agriculture. We believe mobile phones can do the same for conservation.

That’s why The Nature Conservancy is piloting the use of mobile phone text surveys to collect data for human well-being indicators in Kenya’s Northern Rangelands — one of the poorest regions of the country and one historically beset by tensions between nomadic pastoralists and wildlife.

If we’re successful, the approach could spark a revolution in how conservation tracks its impact on people.

Why Collecting Data Via Mobile Phones Can Be Better Than In-Person Surveys

This isn’t to dismiss the value of in-person household surveys. But in many developing country situations like the Northern Rangelands, collecting data via mobile phones can be a superior approach for these reasons:

Quick. Compared with in-person surveys, answering a text message is quick and relatively unobtrusive. When was the last time you were enthusiastic about spending an hour to take a survey?

Accurate. If someone asked you how many time have you been sick in that past two years? You’d take a wild guess and probably forget a bunch of times — unless you happened to be sick in the past couple of weeks, in which case you’d probably overestimate. (Psychologists know this as availability bias.) But if asked whether you have been sick in the past week, you should be able to answer it pretty accurately. Text message questions can be asked in near real-time, thus collecting more accurate answers.

Frequent. The environment changes continuously and understanding how these changes influence human well-being and behavior is difficult with surveys conducted once every few years. Mobile phones will allow for nearly continuous data collection, giving the power to relate changes in human well-being to important environmental or even social events.

Cheap. For many conservation projects, household surveys can be prohibitively expensive. A World Bank study found that getting the same information through in-person surveys cost more than 6 times as much as via text message.

How it Works: The Mobile Data Collection Pilot in Northern Kenya

Kenya is a nearly ideal place to test the mobile data-collection approach. Nairobi is fast becoming the tech hub of Africa. Mobile phone and text message use is high in Northern Kenya, even among the widely dispersed pastoral communities of Samburu, Turkana, Borana, and Somaili who live here. And the need is great: Increasing population and aridity in the region have put enormous pressure on both its people and its wildlife.

We’re working on the pilot with our local partner, the Northern Rangelands Trust, and our initial focus is the Kalama community conservancy — which, like other community conservancies in northern Kenya, has both wildlife conservation objectives (for species such as the endangered Grevy’s zebra) and a range of objectives concerned with improving the lives of the communities there.

But while Kalama has a good system in place to monitor wildlife populations and rangeland ecosystems, that system doesn’t yet have a good counterpoint for tracking human well-being. So we are setting up a monitoring system based on SMS text messages that will tell us how investments in actions such as creating grass banks, improving pastoral management and establishing alternative livelihoods are improving people’s lives.

Participants at a project site register into a database and are then periodically sent a small number of SMS questions regarding aspects of human well-being. As an incentive, a small amount of airtime credit — 10 Kenya shillings in the Kalama pilot — is offered when people answer questions. (As an indication of relative value, sending a single SMS in Kenya typically costs 1 shilling.)

We license a free number or “short code” with each of the mobile networks so that participants are not charged for the SMS messages when they register or answer questions; we also have partnered with the web mobile platform provider EchoMobile to establish a system that automatically transfers airtime credit to a respondent’s account when an answer is received. By repeatedly asking the same question through time, we can use the answers to look for changes in well-being linked to the conservation project.

What are some of the issues our questions to the Kalama participants touch on? For instance, we want to know about milk yields from their cattle, which we hope will improve as a result of better grazing management and healthier rangelands. Or incidents of banditry or other violence that the participants might perceive to diminish as a result of community cooperation and the presence of wildlife scouts.

Because there’s no chance for follow-up, SMS survey questions need to be tailored for the purpose. Ideal questions reflect real-time or very recent experiences, such as “Have you worked for cash in the past two weeks?” Response rates to questions can help us fine tune the way we ask them.

And just like the many a/b tests that Google does to improve its service, sending out large number of questions via SMS allows us allows to ask the same question in different ways and look at the response rate.

For example, many people in northern Kenya do not know their birth by a date, but relative to events such as droughts, floods, elections, etc. We tried asking participants their age, both directly and in broad age categories to see if they gave us the same demographic, and whether one would receive more responses than another. The answer? No difference.

Mind the Gap: Are Those Without Phones and the Illiterate Excluded?

Ownership: An obvious risk of our survey technique is that it might exclude those people who don’t have access to a phone — a real problem if, for instance, the poorest section of society didn’t own phones.

To find out whether there are any demographic blind spots for this approach in our pilot site, we undertook a thorough survey of mobile phone ownership and use amongst Kalama communities. The results surprised even our local partners: over 90% of adults owned a mobile phone. Importantly for the proposed monitoring approach, whether someone owned a phone or not was unrelated to age, gender or wealth.

Literacy: Another challenge — and one that I was more worried about in northern Kenya — is relatively low literacy rates. Reading and responding to an SMS survey question requires basic literacy, and literacy rates among adults in Northern Rangelands communities are well below the Kenyan national average of around 90%.

Overall in Kalama, 59% of those surveys were comfortable using SMS, and another 26% said that someone in their household helped them to read and respond to SMS. As expected based on literacy rates, however, older community members, and especially older women, were much less likely to use SMS — only 45% of elders said they were able to use SMS themselves, with a further 25% saying someone in their household could help. Although we feel these numbers will not invalidate the survey results, it will be important that we find a way to hear the voices of those people unable to participate in the mobile survey.

To ensure the literacy barrier is no bigger than necessary, we are able to send SMS questions in the most locally appropriate language. In northern Kenya, the most widely spoken language is Samburu, but few people see it written so we decided to send the messages in Swahili. However, text-book Swahili is long way from its colloquial use in the rangelands, so each message needs to be crafted to be both simply and consistent with local dialect.

‘How Will This Help Us?’

This was the first question I was asked when I met with the Kalama Conservancy board of elders in June to seek their approval to pilot mobile phone monitoring in Kalama.

I described two tangible benefits: First, knowing whether projects in Kalama were delivering the desired outcomes or not would offer a chance to improve those projects. Second, being able to firmly demonstrate the impact of projects would help make the case for more resources to expand successful activities.

In addition to these, the elders offered two other benefits I hadn’t considered: the system would provide a way for community members to communicate directly with NRT about experiences or concerns. It would also provide a mechanism for the Kalama board to send out community announcements about events or warnings.

Very reasonably, I fielded numerous questions about data security and access. Who would have access to their numbers and their responses? Was there a risk the government might use their number to monitor their activities? Despite my unsatisfying answer that governments appear keen on monitoring mobile phones irrespective of projects like these, it was a good litmus test of the security protocols we had established.

And — not to be underestimated — there was a palpable sense of novelty as a number of the elders sent a registration text message, answered some questions, and received a small airtime transfer and thank you message.

The Community Response

Once we secured the approval and support of the Kalama board, we shared the project more broadly with Kalama residents — and the response far exceeded my expectations.

In a few days, about 370 people voluntarily registered and answered a series of initial demographic questions via mobile phone. Importantly, these people included folks from all ages, both genders, and a range of occupations.

Buoyed by this response, I now want the pilot not simply to reach a representative sample, but most Kalama’s 2,000 or so adults with this approach. While I’m sure the airtime incentive helped, it was immediately clear that respect for the Northern Rangelands Trust was instrumental in encouraging participation.

It remains to be seen what sort of response rate will be maintained through time. Previous mobile phone survey work has indicated a long-run response rate of around 50%, and our early results are slightly above this. As this is not intended to be a panel survey that dedicatedly follows the same person through time, it is fine if participants miss some questions and respond to others, provided there is a critical mass of responses to each question.

What’s New Here? And What’s Next?

Of course, collecting data via mobile phones is hardly a new idea. Substantial efforts have been made — particularly in the fields of development, health and advertising — to use SMS as an avenue for data collection.

But our pilot takes a novel approach. Most SMS data collection efforts fall into one of three types: 1) an aggregator (such as a community nurse) who collects local statistics and shares them via SMS; 2) panel surveys that follows a small number of individuals through time; or 3) one-time surveys asking for opinions on a product or event.

In contrast, our pilot is based on looking at trends in response through time by asking questions directly to a large number of participants in a semi-randomized fashion.

The statistical power of our approach comes through the volume of responses and randomization rather than trying to control other variables. In genesis, this paradigm owes more to the methods developed by Harvard psychologists to study happiness than it does to other SMS data collection approaches.

And we are cautiously optimistic that the approach will prove useful, replicable and scalable. In addition to extending our pilot to other community conservancies in northern Kenya, we are also starting a pilot in Colombia later this year to track well-being impacts of the Agua Por La Vida water fund — water funds having rapidly become a flagship conservation strategy throughout Latin America, but one with little data to date on impacts on people.

With a little support, conservation’s ability to document and improve our impact on people’s lives could be transformed.

 

Eddie Game

Eddie Game is the lead scientist for The Nature Conservancy, Asia Pacific Region. He is responsible for ensuring that the Conservancy remains a world leader in making science based conservation decisions, can robustly report on our impact, and that we get the greatest return for our conservation investments. Eddie has worked on conservation projects in over 15 countries, helping to apply innovative methods and analyses to projects as diverse as community protected areas in the Solomon Islands, grazing management in northern Kenya, and catchment restoration in Colombia. More from Eddie

Follow Eddie

Join the Discussion

3 comments

  1. […] Purchasing text/ airtime is a challenge for people surviving on little means and asking them to use up those precious minutes might be a stretch.  Companies like Echo Mobile are partnering with NRT & the Nature Conservancy to conduct surveys via SMS. […]

  2. […] Community outreach has long been an activity of extractive industries operating in developing countries. For companies, community outreach can improve the acceptance of their operations on the ground and keeps them in touch with problems that communities might be facing due to their operations, while communities gain access to services that their government or NGOs might not be able to provide. However, in order to be effective in leading to better development outcomes for communities and operational risk management for companies, community outreach has to be done right. Key challenges here involve including marginalized groups and strengthening participation and voice of citizens on issues that affect their lives. This makes outreach to communities a cumbersome and expensive process: site managers spend between a third and half of their time on community outreach.  It is here that mobile technologies can provide real value for companies and communities. Simple text- and voice-based approaches allow closing the feedback loop for continuous, cost-effective, and more accurate community engagement. […]