Advantages of an external evaluator

Read the article at Brad Rose Consulting, Inc.

Advertisements

Evaluating with children Part 2

Children are also the most enthusiastic group, once they’ve warmed up. I once visited a school in Davao City for a lesson learning study for a client organization. The school was among the beneficiaries under the client organization’s technical support for the national K12 program. It had augmented the school’s inventory of learning aids as an integral component of a teacher training series.

After introductions in which I felt welcomed by their smiles and easy manner, the class proceeded to demonstrate the use of these materials in their subject lessons. I could see that the children have mastered the games and are ready to learn new ones. Their eagerness and zest soon made me forgot my discomfort from the morning heat. At intervals, the teacher, using a portable microphone, had to bring back order to the group.

In the afternoon, I interviewed three pupils from the class. Observing and listening to them, I’m once again reminded of the fact that children, when allowed to be themselves and supported in their growth, are really smart and have naturally sharp intellect.

Evaluating with children

I’ll leave national affairs awhile and write something for the International Year of Evaluation.

Of the focus group discussions I’ve done in connection with evaluation studies, those with children were the most challenging because of the concern for age appropriate communication approaches for this age group.

The first 30 minutes of encounter is the most crucial part, wherein the facilitator/evaluator needs to establish rapport with and among the participants. External consultants are oft-criticized as “helicopters” in that they go in to the project or program area for a day, interact with different groups of locals for at most two hours each, then leave, and voila! an expert report – findings and recommendations – on the local situation. Hence the constant challenge to build and facilitate meaningful connections albeit in a very short span of time.

With children and young people, I find that they become at ease hence are open with their thoughts and experiences when I relate to them as peers, starting with my voice — friendly and non-condescending but firm. It’s important to create a respectful and non-judgmental space and communicate to them that they are the experts, that is, they’re the ones sharing what they know from their experiences. With younger children, especially those in the lower primary grades, action songs and non-structured group work such as drawings of, for instance, how they perceive the world around them, are effective starter activities. The facilitator/evaluator then direct the discussion based on these group outputs.

Also, days before such encounters, I’d prepare myself by reading up on the project or program literature provided by the client (e.g. approved project proposal, variance reports, monitoring and assessment reports, profiles of project beneficiaries) as these will form the bases of pertinent decisions (e.g. should girls/women and boys/men mix?) and the latest trends fashionable to age groups.

Another crucial element of the FGD is documentation or recording of the discussion. It’s actually more difficult to find a good documenter (than an FGD moderator/facilitator). But this is a work area in evaluation in which skills and knowledge transfer can happen, that is, the moderator/facilitator, who’s primarily in charge of drawing up the FGD Guide, would know what sort of documentation output is needed for the study hence she needs to communicate that to the documenter, or lacking experience, provide the documenter with the documentation template and guide him how to use it. There is always the recorder but for me I only listen to the recording when the written documentation is lacking in certain parts. Besides, after doing FGDs with 300 participants, listening to the recordings is the worst thing that could happen– it’s as if you’re conducting another round of FGDs all over again!

Another key element is selection of participants. Once, due to misunderstanding with the client over instructions in identifying members of the groups, we decided to cancel the scheduled discussions for the day, apologize to those who came, and draw up another set of groupings and schedule. The mix of participants affects the outcome of the study. You’d want a robust discussion, or in other words, you’d want participants with knowledge and experience of the project or of the context in which the project took place.

While management of FGD activities follow standard rules, the facilitator/evaluator needs to be flexible because group personality and dynamic differ. I’d be handling four FGDs (of 8 to 10 members each) in one day, for instance, but each group would be different. One group may be more reticent, another would be like a runaway train, etc.; the challenge is to be able to handle each group and discussion effectively. I’ve encountered various groups since my first day in development work and these serve as templates in mixing and matching techniques appropriate to each group. Allowing enough transition time between meeting with the groups to prepare oneself for the next facilitation, also helps.

Another thing in the conduct of FGDs is that surprises are commonplace hence the need to work out beforehand a Plan B or even a Plan C. Once, because of misunderstanding over instructions relayed to local field coordinators via the client (i.e. the specific instruction was to identify 8 to 10 participants), we arrived at the venue to find 50 or so participants who as the coordinators informed us comprised the FGD group. It was two o’clock in the afternoon and the group was supposed to be our last FGD in the province (Quezon) before going back to Manila. I was the only facilitator then (the other had gone back earlier), and with me was the documenter (who wasn’t capable of doing FGDs) and the driver. We were supposed to be in another project area (Batangas) the following week meaning there was no more time to break up the big group, and for security reasons, we needed to leave the area before sundown. After ranting at the field coordinators in the backroom, I went out to meet the 50 eager-looking young people who understandably didn’t have a clue about the situation. I asked one of the coordinators to lead in the introductions which took up a good 20 minutes so I could hash up an alternative approach.

In another FGD, in a school, I was in the middle of facilitating a discussion with a group of students when a series of blares made us jumped out of our seats. There was apparently a flooding and everybody needed to leave. So we did, immediately. We went back at another time to finish the discussion.

Equally important too is for facilitators to prepare themselves for the rigorous field schedule. A mixed methodology of interviews and FGDs with, say, a total of 300 participants would take more or less two weeks, Monday to Saturday or even Sunday. It’s crucial that energy levels are maintained throughout, morning til sundown, day after day. For me, I get re-energized by the sessions themselves– the flurry of activity, discussions, people I’m meeting. A sunny weather, I find, also helps me maintain my mood. I only have to look at the blue sky. Also, physical exercise, like taking a walk at the end of the day to explore the CBD with members of the team works as a de-stressor and refreshes the senses. Then, at the end of the engagement, or after two weeks of nonstop talking and hyper alertness in the field (e.g. keeping tabs of what participants are saying during discussions, one FGD to the next) and afterward the pressure from defending findings, rest, for me, comes in the form of clamming up, mentally shutting down, and focusing on things other than evaluation (some I know tend to their gardens).

Each session presents an opportunity to learn. The evaluation team needs to make time to reflect on and identify, sooner than later, what these learnings are. For example, at the end of the day, the documenter/observer and I would get together, usually at dinner and discuss, among other things, how the sessions went, because the observer may have noticed things that I haven’t while facilitating the sessions. The feedback are then utilized to improve the next day’s sessions.

On the threat of insincerity in evaluation

Jane Davidson, at the international conference Evaluation Revisited:  Improving the Quality of Evaluative Practice by Embracing Complexity, said of evaluation:

  1. It’s not just measuring outcomes; it’s saying how substantial, how valuable, how equitable those outcomes are;
  2. It’s not just reporting on implementation fidelity (did it follow the plan?); it’s saying how well, how effectively, how appropriate the implementation was;
  3. It’s not just reporting whether the project was delivered within budget; it’s asking how reasonable the cost was, how cost-effective it was, and so forth.

But what if the commissioning agency shelves the value judgments into obscurity?  The conference report cites an article in Roger’s blog which says insincerity (commissioners of evaluation defaulting on their commitment to respond to information about both success and failure) is “perhaps the greatest threat to the success of individual evaluations and to the whole enterprise of evaluation”.

On the satisfaction survey of barangay services

happy with barangay services

Whoa! If this is the case, national and local media are from outer space. And all the talk about corruption, poverty, hunger, conflicts, political dynasties, disaster risk reduction, citizenship, low ratings concerning water and sanitation, ease of doing business, competitiveness, regional development, transparency, etc. are nonsense. And we’re all Alices seriously considering the blather of Mr. Mad Hatter.

But these things do not happen in outer space. These are happening, first, on the ground, in the barangays (villages) where people are and going about their lives.

Or.

The survey result is suggestive of the state of awareness of those surveyed. How?

Every business man or woman knows this by heart: that a luxury product, say, Vuitton, won’t sell in a location where people living there or nearby are earning not more than USD2 daily. Unless the LV store is intended as a museum with free passage, yes it could happen. Similarly, in a barangay where residents have always thrown their waste into the creeks, merely shrug their shoulders when the Barangay Local Government Unit does not do its job, and live day in and out year in and out on convictions like “bahala sila” or “wala akong pakialam, basta ako —“, everything or anything is good and satisfactory. Because that’s the way they’re wired, the way it is, has always been, and as far as their horizon allows them to see, always will be. They wouldn’t demand for more because they don’t know ‘more’ or what is ‘more’. To them, this is the world they know.

In my previous work, my employer organizes and sponsors out of town trainings for village (barangay) residents (not necessarily barangay officials). Some of these trainings were in hotels and resorts in the big cities – Cebu, Tagaytay, and of course Metro Manila. A few were abroad. Returning to their villages, these folks – farmers, fishers, homemakers, young people and yes barangay officials who’ve never stepped out of their towns – wear down their families, relatives, and neighbors with stories of what they refer to as “luxurious” living. Their amusement and joy burst out of them like grand fireworks. Think The Gods Must Be Crazy. They’re like “sirang plaka” about these particular episodes in their lives. Listening to them talk about these, you can either laugh or weep. Or both.

To many of us, elevators, escalators, generators or 24/7 electricity supply, hot showers, air-conditioning, hair dryers, five-inch hard mattresses, clean linens, toilets (proper and clean), 24/7 water supply, 24/7 coffee and tea service, polite behavior, are behavior we live by and ordinary everyday things and experiences.

After many years, colleagues and I went back to some of these areas to do post-intervention or impact studies. And people identified that it is the knowledge and learning they’ve gained about themselves, others, and the world through training and development and exposure to worlds outside their own that have and continue to stay with them (even after our organization left their areas) and which were used and led to significant changes in their lives. They now are more, know more, and are demanding more, from themselves, their families, communities, and others. (What I mean by ‘more’ here is, what it should be, not some snobbish desires of service, goods, and treatment.) Ironically, because they’re demanding more, they’re now seen as having inflated heads from people who did not know more. (Funny world, but yes, the circle of life.)

Maslow is right. All the time. One cannot be an adult with matured tastes, coming out of the womb. One must first have well-grown and developed teeth and digestive system before one is able to know, corn, and chew a cow.

Within the research community, ethical behavior is the norm. And an important ethical question that researchers especially those dealing with human participants must ask is, what do I want to contribute toward out of the research? what is the intention in conducting the research?

This survey has somehow left me hanging, like it’s telling something and then not. What were the key questions of the study? In what service areas does it pertain to? Where was the study conducted? Urban? Rural? Mixed? What is the profile of the 376 sampled? (incidentally, I’m confounded that 376 is the sample used for a national survey, given there are 42,027 barangays in the country. Researchers out there will protest! I’m doing surveys and 500 for a mega city is either a typo error or gross ineptitude. Unless. The 376 are from the posh villages.)

Researchers are responsible for the information and knowledge we impart out there. Plus, if you ask me, local government services, an intricate and complicated subject in itself, cannot be bounded and understood by ‘I’m not satisfied’ to ‘I’m satisfied’ ratings alone. Please. The subject is not up for popularity contest.

Pigstatistics

There was a pig who wanted to be a scientist. He was not interested in models. When asked how he planned on making sense of the world, the pig would say in a deep mysterious voice, “I don’t do models: the world is my model” and then with a twinkle in his eyes, look at his interlocutor smugly.

Around that time, some dogs opened a pub called, “Doogle” which was visited by all animals in the jungle. The wine was delicious and the traffic at the pub was unprecedented. The dogs became rich and famous; they also obtained a lot of data from the visiting animals. They bought even more pubs and collected even more data about their customers.

Now, they wanted to analyze this data to attract even more customers towards Doogle. The pig saw this as a big opportunity and gathered other like-minded pigs. The drove of pigs helped Doogle in applying pigstatistical methods (ham-correlation formulation etc), to predict various things including: kinds of animals attracted to the kinds of beverages; drinking patterns of different animals; the kinds of tables liked by classes of animals; arrival times; number of glasses Doogle would need in the near future, etc, etc, etc. To an astonishing degree, the pigs made quite accurate predictions using their pigstatistics.

The services of our pigs were acquired by other entities including FaceSlap, Barker, and Snorter, among others. Our heroic pigs helped their clients in outshining the competition. In fact the pigs method of collecting huge amounts of data and then applying pigstatistics on it came to be known as “Pig Data” in their honor.

Read entire article at Realm of the Scensci

Eliminating researcher’s influence on subject

I’m sensitive to emotions conveyed on people’s faces. I can see the range of emotions flitting across people’s faces, like the play of light and shadow brought on by the early morning and late afternoon sun on a building facade. Always, I am affected. Mostly, fascinated. And often, I haven’t misread. The corollary of this is that I am disturbed – scared – when I see no emotions on faces – I feel I’ve come across a blank wall and I have to blindly make my way through.

Which is why when I’m face to face with a subject in a situation in which I am the researcher, I feel for the subject, meaning, I’m anxious over how I’m influencing my subject via emotions or messages I may unconsciously convey on my face. The anxiety can of course be picked up along with the others by the subject. Or maybe not, depending on how keen the subject is to these things. (In such a setting, I see in my subject myself being mirrored back to me regardless of whether or not she is conscious of the effect, so that, at the end of the session I have a momentary blurring with regards to whether I’m the researcher or the researched, but this is another topic altogether.) At best, I try to be as detached as possible to the topics I put forward and the feedback or reactions I receive. By detached I mean to have set aside emotions, positive and negative. I try to accomplish this by visualization before the session – imagining myself taking out of myself a bundle (these should be my emotions) and leaving them at a corner behind me.

Over the holidays in between celebrations I caught up on movies I missed (before this, I only managed to watch two movies in 2012), one was A Dangerous Method (I’ve had the DVD for eons). The story centers on Carl Jung, Sabina Spielrein, and Sigmund Freud. Spielrein who later became a leading psychoanalyst in her time was Jung’s first patient on which he used Freud’s psychoanalysis or more popularly “the talking cure”. In one such “the talking cure” sessions, Jung sat behind Spielrein and conducted the session from there. According to the review on Psychology Today, “This orientation was suggested by Freud, so the psychologist wouldn’t have to worry about the patient monitoring his/her reactions as potential judgments.” And I thought, omigod, of course!

But who nowadays conducts psychotherapy or research from behind his or her subject? And wouldn’t the subject’s facial and frontal non-verbal expressions important to the analyst’ or researcher’s overall understanding of the subject? What if in the context of trauma the subject’s oral recount is in monotone – wouldn’t the analyst want to check on the subject’s face or expression for discrepancy or consistency? Is voice all that mattered?

In past development researches and evaluations done for my employers, I’ve gone against the ethic of not influencing your subject – development workers in the context of meetings call it ‘facipulation‘, had attacks of conscience because of it but took refuge in the fact that it was made without malice. The setting in localities called for certain rules not necessarily in line with clinic-based research. Villagers and even public officials – the research and evaluation subjects – would’ve walked out on me leaving me without a coherent set of information hence without an output or a report if I spoke in unattached monotone to them or deliberately detached myself from the spirit of the discussion. They would’ve stamped me a snob and snobs on locals’ lists are not found in their sphere of confidence. This, plus the fact that the organizations’ – my employers’ – mantra and dictum to its employees is “go to the people. live with them. learn from them. love them.” Love them! The exact opposite of detachment because if you love you’re somehow attached. And if the researcher’s attached to the subject, it means the relationship between researcher and subject even prior to the actual research is already contaminated. In these cases, what I did was to review my research notes and draft report and flesh out – that’s how excruciating it is – my involvement or extent of influence in the proceeding; where it’s impossible for me to do so without changing a major part of the report, I report it as it has transpired giving the reader a feel of the interaction; and reserved my thoughts as a researcher in the Conclusions. As a result, at one time, a colleague having read a report joked that I was being anthropological (although my undergraduate is economics not anthropology) instead of rational which is what rational organizations are understandably after. I took that as an indication that I’ve driven home the point – I wanted the subjects (not me the researcher) to speak to the reader.

Working as a development researcher in an organization can bring some surprising challenges to the profession but my experience is that as long as you stick to the bottomline you can with some creativity and presence of mind work through the established organizational process. And maybe with a smattering of luck sow seeds of positive change into it, which is after all what research – knowledge – is for.

Development for whom? (commentary by Nono Felix)

It appears that child sponsorship and development organizations have yet to report an impact on children’s wellbeing with solid evidence. In the Web, the available reports are mainly on good practices and lessons learned, which, by the way, include the observation that the rights-based approach will only succeed if justice, peace and freedom are pursued.

Read the commentary in today’s Philippine Inquirer.

Critique of a sectoral development plan – Part II

If and when development planning is done a bit more rigorously (without being rigid), strategies and projects and programs will manifest themselves clearly.

Like this CCT (4Ps) which has one of the biggest allocation (perhaps because GOP counterpart to the WB loan has been raised). The Program contends it is targeting the poor. Well and good because that’s what it’s supposed to do. But those of us who are or have worked with the poor know that within the poor population are gradient levels of poverty. There are the poor who are credit-worthy according to assessments by microfinance institutions, the poor who are skilled and are or could capitalize on that, etc. and the poor who are absolute poor or the so-called the poorest of the poor, those at the very bottom. But what’s happening is all the poor are prioritized. This implies that we are not again learning the lesson from studies of subsidies. Findings from these studies show that subsidies do not benefit or hardly benefit the poorest of the poor – those who really need subsidies – because there was no proper targeting made in beneficiary coverage. It was assumed that all poor are the poorest of the poor.

Let’s take another example just to expound on the blanket approach: school supplies given to school children. The sponsoring agency, organization, group, or individual gets the enrollment statistics for the current or previous school year from the school, buys the supplies based on the number, and distributes them. There’s nothing wrong with giving. But if you want to make an impact or a real difference and narrow down or eliminate the gap between the haves and have nots, you have to do a bit more research and tougher decision making than just getting the enrollment record. Your research should answer the questions: if your target are schools, how many among the enrolled school children are with and without school supplies (disaggregate); if your target are children not in schools or the out-of-school children and youth, how many among are the OSYCs in the barangay (village) and of this how many would have enrolled in school if given school supplies (because there are those whose “problem” for being out of school is not answerable by provision of school supplies). And do your buying (targeting) of school supplies accordingly. In fact, if you want the school supplies to be of real value to the users/recipients, you’d ask around from the targeted users what school supplies they really need for the school year (since there are those who only need, say, paper and pencils, while there are those who only need school bag, or even school shoes, and there are those who need the whole gamut of supplies) and buy according to the needs of the users.

In Part I of this article, there is mention of working girls (children) the majority of whom did not complete primary education even at 15 years old. And why are they working? Because of dysfunctions or perhaps debilitating illnesses in the family these children are made to work to feed both their siblings and adults in their families. Among poor children, they are among the more disadvantaged. To make an impact and create real change in the lives of the bottom poor, the 4Ps should include these working children and their families in its priority list. And to make the priority list of value, design a graduated scale of cash transfer as opposed to the current PhP2,000 monthly for everyone. I mean, what’s the use of doing a baseline and maintaining a database and a separate system if you end up giving everyone the same amount? It’s like giving all your students the same grade regardless of each student’s performance.

Evaluations the world over show that CCT programs made considerable impact on children’s schooling and health. For me, one doesn’t need an evaluation to tell you these because common sense will tell you that of course the more children are chivied up into schools and health centers (as a result of the conditions to get to the carrot) statistics will record that increase; of course, food poverty within the program period will go down at least for families who utilize the transfer to buy more food or nutritious food; but no you can’t expect 24,000 a year (or, 2,000 monthly) for 5 years to pull you out of poverty in the economic sense.*

So what I’d like to know in terms of impact from the CCT program is the extent to which it changed the behavior and attitude of recipient families toward health and education / are recipient families putting their children to school and the health center for primary health care even after the phase out of conditions and cash transfers? This is what CCT is really for in the long term, the conditions are really to change the behavior and attitude of families toward health and education and the cash transfers are the incentives to “push” them toward realizing that: the carrot at the end of the stick. Impact evaluations of CCTs should therefore look at this and these can be studied perhaps five years after the phase out of the program (enough time to have everything from the program seep in and be actualized into practice).

Anyway my point really is, sound evidence and analysis will give you a plan (strategies, projects and programs) responsive to realities. And because realities change relative to time research and evaluation to capture that change or development should be done at regular intervals. This is how a plan is said to be a living document (as opposed to one that is cast in stone; because it’s lifeless it loses usefulness).

* I’ve come across quite a number of families (who haven’t reached maximum number of children allowed in the program) who say the cash transfer is actually an incentive to have more children within the program period it’ll mean an additional amount. This means CCT is incentivizing more children among already poor families who are cashing in on the program / population growth in aggregate – is this a good thing?

Systems Thinking in Early Education – Peter Senge

Peter Senge humors his audience to imagine the young child’s innate ability to control his or her family as an example of systems thinking.  As with other experts in his field and in others, Senge laments that a young child’s innate abilities are forgotten once he or she enters formal school.  What could early education learn from systems thinking to improve children’s learning?