I have reconnected with a former senior Monitoring and Evaluation manager at my former employer’s regional office. He was our unit’s boss’ boss actually hence our regular interaction with him. Despite his achievements internationally, he has remained grounded, humble, and true to the development ideals.
He shared me the compendium of his development “adventures” and “misadventures” which I immediately devoured. The story on the first page, I’m sharing here below, is distinctly him, reminding me of his wit, as we in the M&E Regional Network which he headed tried to make common meaning of development results. The Filipino, in his stories, often comes out the hero simply because he was Filipino. The other nationals took this good-humoredly. In exchange for their sportsmanship, other nationals are by turns made the heroes.
How we become development workers
There was an international workshop on development at a riverside hotel in Bangkok, Thailand. One of the topics was how to attract the best and the brightest to the development field. The workshop went very well. The resource persons were development specialists (an American, an Englishman and a Filipino). They were excited to write the workshop report and send it out. At the end of the workshop, they went out for dinner. They were tired of the hotel food like prawns, crabs and lobsters.
After dinner, they decided to walk back to the hotel. They would like to have a closer look at the Chao Phraya River at high tide. At one point, they were convinced it was deep. They then heard people shouting. They found out that a little girl fell into the river. It was clear nobody was about to rescue her. Suddenly, the Filipino was swimming in the direction of the hapless little girl. After a few minutes, both of them were on the riverbank. He was mobbed by cheering spectators. They profusely thanked and congratulated him.
Upon seeing his colleagues who were smiling widely, the Filipino quietly asked them, “Who pushed me into the waters?”
I agree with items 1, 2, and 6 in this order: the demand for evaluators will continue to grow; the number of training programs in evaluation will increase; a national registry of evaluations will be created.
The reason: as traditional economies, such as partly the system in the Philippines, move toward market economies so will demand for transparency and accountability that in turn increases demand for evaluation services. This then increases demand for knowledge and skills in evaluation.
Logic says increased demand for transparency and accountability (especially in public institutions) should lead to creation of a national registry of evaluation findings and reports, but in reality this would depend on so-called political will or level of political maturity. Are owners of information willing to make these available in the public domain? To what extent are they willing to share? What information are they willing to disseminate?
Internal evaluators have it worse as friction at the office, usually with colleagues in Operations, is a constant. It’s expected considering that asking tough questions is part of their job description although this doesn’t mean that they’re in evaluation mode 24/7!
During my time as one, the only person at the office who was relatively at ease in my company was our General Services Staff because his work – cleaning the comfort rooms, for one – wasn’t in my list of things to evaluate.
On coffee breaks, after the usual greetings to colleagues already inside the room a few of whom I may have had some vigorous exchange with only moments before, he was the one I chatted with because I supposed he felt he could freely talk to me. By this I mean he didn’t worry that I was checking on his every word for like Freudian slips that if he was in Program or Operations he’s afraid would expose his inefficiency or ineffectiveness. Also, the guy’s knowledgeable on a lot of topics so that our exchanges even if for a few minutes were gratifying. In fact I’ll go further and say such brief respites helped me maintain a positive outlook of my work.
Deadly disease outbreaks brings to mind Newton’s Third Law of Motion i.e. for every action, there is an equal and opposite reaction in so far as it applies to the downside of mobility which in the age of globalization is taken for granted. Facilitated by technology, epidemics could now cross national borders and potentially spread onto the international arena. Then, again, following Newton’s law, countries can react back.
The Independent Evaluation Group (IEG) at the World Bank did a review of the World Bank Group’s interventions to the different global crises in past years, and found key lessons (10) for an effective response to the Ebola epidemic:
Weaknesses in health systems are a major contributing factor to disease risk, especially from the lack of trained and equipped medical personnel, contract tracing capacity, sample collection and transport capacity, laboratory diagnostic capacity, and intensive care units with isolation capacity. The crisis response should support not only immediate emergency interventions but also medium-term risk reduction through public health system strengthening, recognizing that future opportunities to engage may be limited once the crisis has passed.
Capacity building efforts should be done in a way that are relevant to more than just a single disease, and should consider from the outset means to build sustainable systems that last beyond the current emergency. This would likely include support for animal health and veterinary systems, in addition to public health systems, and for managing other zoonoses and infectious diseases.
Preventative epidemiology, including effective disease reporting systems, is a necessary means of identifying and tracking disease outbreaks. Contact tracing of infected individuals is urgent to guide responses to outbreaks. Many serious zoonotic diseases have significant reservoirs in wild animals, such as bats or birds, and the threat level posed by these diseases may be poorly understood without surveys of wild populations in the medium term.
Jane Davidson, at the international conference Evaluation Revisited: Improving the Quality of Evaluative Practice by Embracing Complexity, said of evaluation:
It’s not just measuring outcomes; it’s saying how substantial, how valuable, how equitable those outcomes are;
It’s not just reporting on implementation fidelity (did it follow the plan?); it’s saying how well, how effectively, how appropriate the implementation was;
It’s not just reporting whether the project was delivered within budget; it’s asking how reasonable the cost was, how cost-effective it was, and so forth.
But what if the commissioning agency shelves the value judgments into obscurity? The conference report cites an article in Roger’s blog which says insincerity (commissioners of evaluation defaulting on their commitment to respond to information about both success and failure) is “perhaps the greatest threat to the success of individual evaluations and to the whole enterprise of evaluation”.
“Implementing development programs effectively, efficiently and sustainably can’t happen without transparent aid information,” Samantha Custer, policy outreach and communications director at AidData, told Devex. “Governments and donors use this information to improve coordination, reduce duplication and maximize the impact of their investments.”
Custer added: “It’s not enough to publish vast amounts of aid information on the Web — the quality of that information matters and local actors often lack the capacity to use this in their daily work.”
Part of the problem why aid, despite numerous studies and initiatives, is not as effective as donor agencies would hope, is the numerous gaps not only in publishing data, but also in operationalizing these pieces of information.
One of my tasks when I was employed was to produce briefs, maximum of five pages, out of 100-plus or so pages of full reports of research and evaluation my organization had funded. The aim was to improve organizational learning and evidence-based decision-making. The organization’s senior management team was the primary audience, with sector advisors kept in the loop of these.
On the whole, the task can be likened to eating pizza. I ate pretty much entire pans of; bits of toppings and crusts, by the rest. At some point, you’d ask, is it possible for a brain to be obese?
But yes evidence must move out of the pages of reports into discussions and eventually operations, for the desired change to happen. Strategy making and operationalization are out of the hands of data producers and communicators. People who do the next phase of the work must take over.
In the case of the Philippine government, evaluation of national strategies and public programs (not just projects) must first be made a national policy, integral to broader goals of accountability and managing for results.
Still on the temporary bunkhouses in Tacloban, it is a matter of course that an evaluator assessing the efficiency of a project makes a comparison of models. The bunkhouse project as it is looks like this
Now, there is, apart from the ones featured in previous posts, a DIY model that is the brainchild of Alastair Parvin, the man behind Wikihouse, an “open source construction set that lets anyone download, design, and print CNC-milled, easy to assemble houses”.
Parvain explains his idea in his talk at TED, Architecture for the People by the People
I believe, along with Parvain and the others, a belief based on a decade of field experience and observations, that, given the resources and supervision, carpenters and even builder-hobbyists in the villages, young and old alike, can put up their own houses as well as the needed number of houses in post-disaster areas around the country. This essentially is the practical meaning of the right to housing: the right to build the roof I need over my own head.
There was enthusiastic talk in the aftermath of Haiyan about work-pay schemes for displaced persons which was on hindsight merely talk as evidenced by DPWH’ outsourcing of the bunkhouse project to a private firm (that in turn hired locals only as laborers which is the status quo with private builders). The Parvain model provides an alternative way of doing the work which fits in with the envisioned work-pay and empowerment arrangement for affected locals. The model, on the whole, adds up to big savings not to mention the nonmonetary benefits of joy, improved confidence, and community building among the participants. The DPWH bunkhouse project model, as it is currently constructed, is, on the other hand, the result of a traditional imagination that is not anymore relevant in present times. This is all water under the bridge, but nonetheless one can’t say enough of the very unfortunate circumstance that the affected who have already transferred to the bunkhouses find themselves in on a daily basis as a result of that imagination.
The NDRRMC is commended for its two projects–Project DINA and National Cell Broadcasting System–which will be launched soon according to the Philippine Star. These projects are designed to improve the dissemination of disaster information and warning.
In its future projects, however, NDRRMC could reach out to more by looking at results of studies, usually commissioned by the I/NGO community, on communication pathways that locals utilize relative to early warning and disaster information. I was part of a few of these commissioned studies and results of these point to some interesting facts, one in particular:
Radio is consistently the top source for disaster/emergency information of both adults and children. The medium is followed by word of mouth, that is, through local officials (i.e. the Barangay Captain and Councilors, or the Mayor) and neighbors. Information from these two sources are delivered in the vernacular. (I think it’s in this blog where I wrote about the use of high falluting language of PAGASA weather reports, which is like talking Russian to locals, hence do not serve its purpose in the localities.)
Text or mobile messaging and the Internet are not popular sources for this kind of information. This is partly due to location (i.e. there is no available service in so-called far flung barrios or villages) and of many villages being off the grid (i.e. electricity has not reached these areas yet). But it is these very facts that contribute to these folks’ vulnerability to disasters. Of warning messages from local officials, locals cited inconsistency in content (i.e. the Barangay Captain’s information differs from the Mayor’s or vice-versa). Also, broadband speed is not fast enough as marketed by the telecoms (on your unlucky day accessing the Net can take you hours and you may end up flinging the thing out your window in sheer frustration; or worse, if ever there was a flash flood again you’d have already drowned and died while the thing is still trying to log you onto the Net. I’ve been in areas where you have to get onto a tree branch in order to receive at least one bar of network availability–this means you can send and receive messages but calls are choppy; climb a hill, and perhaps one more bar and maybe improved calling service.). It means that information from the radio and local officials need to be accurate at all times otherwise the consequence may be dire. In fact, one reason why families were caught off guard in the recent disasters in the country is attributed in part to inaccurate information, delayed information, or not having received warning at all.
This reiterates the nature of ‘communication’, which is, it is a two-way street. While it is true that the Philippines is a top user of social media, the question is who are the users? What’s their profile? You’d be surprised that even in Metro Manila, not everyone has a 24/7 Internet service or an Internet-capable phone (you’d be even more surprised that there is no electricity supply in certain areas, the slums which ironically are more at risk). If this is the scenario in the mega city, what’s in the provinces, many of which have weak or entirely no Internet and/or mobile phone service yet?
Planning using the lens of inclusivity should change the current imbalance of things. A project costing, say, PHP500M should not cater to or benefit, say, only 500 people. Internet-based platforms may be relevant to urban areas but not to the many off the grid towns and villages. Project planners and designers should therefore look into, among other things, where are the areas more at risk vis-a-vis the different types of disasters (we’ve focused on natural hazards and merely on the elements but what about pandemics, floods as resulting from environmental degradation, civil unrest, terrorism, war, chemical spillages, etc.?) What’s the profile of these areas? the residents, in terms of their communication patterns and access to communication channels? And be conscious of the fact that data collected now is entirely not the same after 3 to 5 five years, meaning, an M&E system should be in place, preferably one that can be run by locals themselves and capable of vertical and horizontal integration with other but similar databases. Resilience of the project (or, system) is correlated with the quality of M&E systems; without needed data anytime and anywhere it is needed, planners and implementers are walking blind. You could rely on your guts perhaps in love (although what may be termed ‘guts’ is actually your subconscious seeing patterns in the behavior – which is a scientific procedure – and the familiar act is the one you often respond positively toward) but it is science that will get you to the moon and back.
Regardless, the launch of Project DINA and the NCBS should be accompanied, where these are needed, with increased access to improved utilities and telecommunication services and lines.
Among the challenges facing the DRR or Humanitarian Assistance community, quality of emergency shelters is probably the hardest to fulfill despite there being clear-cut standards (Sphere Handbook) available to planners. One reason is the inherent difficulties in inter-agency coordination and collaboration. In many instances, international standards set in the Handbook need localization, in consideration of local needs and culture and availability of materials. The inability to establish coherent inter-agency coordination and collaboration regarding the matter is manifested in the disorderliness and substandard shelters that we often come across in the affected areas.
I’m particular about sanitation thus this is what I check out first. What I’ve seen so far, the facilities are far from meeting minimum standards in Sphere. Usually, there’s a hundred or so persons packed into a given shelter, often the open gym of the city or municipality or the public school. And usually there’d only be three portable toilet/bath to the population. From what I’ve seen so far of this portalet/bath, it is still substandard. Quality bath/shower room would look like this
Just looking at the sample facility uplifts the spirit — and that is one of the aims of emergency assistance, to protect the right to life with dignity which has already received quite a bashing from the disaster. The point of the visualization is that the design of existing portalets/baths in emergency shelters could do much better.
Another shelter facility that is sorely in need of upgrade is the cooking station. The usual scenario is, each family has its own portable stove. A good thing when the name of the game is independent living, but when a relatively small space is commonly shared with 99 or more others, the overall picture created by 100 stoves being used all at the same time is chaos. In this case, common cooking stations–one serving xx families–is the better shelter management option. The portable cooking station would look like this
The samples above are merely to provide visualization of what quality facilities look like. These facilities or equipment ought to be made or built during lag periods, that is, as preparatory activity for when the next disaster or emergency happens, and using collaborative means (e.g. bringing in experts in industrial design, temporary agreements with local water providers or cooperatives relative to water supply for portable washrooms and city or municipal governments relative to waste disposal and collection).
I’m sensitive to emotions conveyed on people’s faces. I can see the range of emotions flitting across people’s faces, like the play of light and shadow brought on by the early morning and late afternoon sun on a building facade. Always, I am affected. Mostly, fascinated. And often, I haven’t misread. The corollary of this is that I am disturbed – scared – when I see no emotions on faces – I feel I’ve come across a blank wall and I have to blindly make my way through.
Which is why when I’m face to face with a subject in a situation in which I am the researcher, I feel for the subject, meaning, I’m anxious over how I’m influencing my subject via emotions or messages I may unconsciously convey on my face. The anxiety can of course be picked up along with the others by the subject. Or maybe not, depending on how keen the subject is to these things. (In such a setting, I see in my subject myself being mirrored back to me regardless of whether or not she is conscious of the effect, so that, at the end of the session I have a momentary blurring with regards to whether I’m the researcher or the researched, but this is another topic altogether.) At best, I try to be as detached as possible to the topics I put forward and the feedback or reactions I receive. By detached I mean to have set aside emotions, positive and negative. I try to accomplish this by visualization before the session – imagining myself taking out of myself a bundle (these should be my emotions) and leaving them at a corner behind me.
Over the holidays in between celebrations I caught up on movies I missed (before this, I only managed to watch two movies in 2012), one was A Dangerous Method (I’ve had the DVD for eons). The story centers on Carl Jung, Sabina Spielrein, and Sigmund Freud. Spielrein who later became a leading psychoanalyst in her time was Jung’s first patient on which he used Freud’s psychoanalysis or more popularly “the talking cure”. In one such “the talking cure” sessions, Jung sat behind Spielrein and conducted the session from there. According to the review on Psychology Today, “This orientation was suggested by Freud, so the psychologist wouldn’t have to worry about the patient monitoring his/her reactions as potential judgments.” And I thought, omigod, of course!
But who nowadays conducts psychotherapy or research from behind his or her subject? And wouldn’t the subject’s facial and frontal non-verbal expressions important to the analyst’ or researcher’s overall understanding of the subject? What if in the context of trauma the subject’s oral recount is in monotone – wouldn’t the analyst want to check on the subject’s face or expression for discrepancy or consistency? Is voice all that mattered?
In past development researches and evaluations done for my employers, I’ve gone against the ethic of not influencing your subject – development workers in the context of meetings call it ‘facipulation‘, had attacks of conscience because of it but took refuge in the fact that it was made without malice. The setting in localities called for certain rules not necessarily in line with clinic-based research. Villagers and even public officials – the research and evaluation subjects – would’ve walked out on me leaving me without a coherent set of information hence without an output or a report if I spoke in unattached monotone to them or deliberately detached myself from the spirit of the discussion. They would’ve stamped me a snob and snobs on locals’ lists are not found in their sphere of confidence. This, plus the fact that the organizations’ – my employers’ – mantra and dictum to its employees is “go to the people. live with them. learn from them. love them.” Love them! The exact opposite of detachment because if you love you’re somehow attached. And if the researcher’s attached to the subject, it means the relationship between researcher and subject even prior to the actual research is already contaminated. In these cases, what I did was to review my research notes and draft report and flesh out – that’s how excruciating it is – my involvement or extent of influence in the proceeding; where it’s impossible for me to do so without changing a major part of the report, I report it as it has transpired giving the reader a feel of the interaction; and reserved my thoughts as a researcher in the Conclusions. As a result, at one time, a colleague having read a report joked that I was being anthropological (although my undergraduate is economics not anthropology) instead of rational which is what rational organizations are understandably after. I took that as an indication that I’ve driven home the point – I wanted the subjects (not me the researcher) to speak to the reader.
Working as a development researcher in an organization can bring some surprising challenges to the profession but my experience is that as long as you stick to the bottomline you can with some creativity and presence of mind work through the established organizational process. And maybe with a smattering of luck sow seeds of positive change into it, which is after all what research – knowledge – is for.