Evaluating with children

I’ll leave national affairs awhile and write something for the International Year of Evaluation.

Of the focus group discussions I’ve done in connection with evaluation studies, those with children were the most challenging because of the concern for age appropriate communication approaches for this age group.

The first 30 minutes of encounter is the most crucial part, wherein the facilitator/evaluator needs to establish rapport with and among the participants. External consultants are oft-criticized as “helicopters” in that they go in to the project or program area for a day, interact with different groups of locals for at most two hours each, then leave, and voila! an expert report – findings and recommendations – on the local situation. Hence the constant challenge to build and facilitate meaningful connections albeit in a very short span of time.

With children and young people, I find that they become at ease hence are open with their thoughts and experiences when I relate to them as peers, starting with my voice — friendly and non-condescending but firm. It’s important to create a respectful and non-judgmental space and communicate to them that they are the experts, that is, they’re the ones sharing what they know from their experiences. With younger children, especially those in the lower primary grades, action songs and non-structured group work such as drawings of, for instance, how they perceive the world around them, are effective starter activities. The facilitator/evaluator then direct the discussion based on these group outputs.

Also, days before such encounters, I’d prepare myself by reading up on the project or program literature provided by the client (e.g. approved project proposal, variance reports, monitoring and assessment reports, profiles of project beneficiaries) as these will form the bases of pertinent decisions (e.g. should girls/women and boys/men mix?) and the latest trends fashionable to age groups.

Another crucial element of the FGD is documentation or recording of the discussion. It’s actually more difficult to find a good documenter (than an FGD moderator/facilitator). But this is a work area in evaluation in which skills and knowledge transfer can happen, that is, the moderator/facilitator, who’s primarily in charge of drawing up the FGD Guide, would know what sort of documentation output is needed for the study hence she needs to communicate that to the documenter, or lacking experience, provide the documenter with the documentation template and guide him how to use it. There is always the recorder but for me I only listen to the recording when the written documentation is lacking in certain parts. Besides, after doing FGDs with 300 participants, listening to the recordings is the worst thing that could happen– it’s as if you’re conducting another round of FGDs all over again!

Another key element is selection of participants. Once, due to misunderstanding with the client over instructions in identifying members of the groups, we decided to cancel the scheduled discussions for the day, apologize to those who came, and draw up another set of groupings and schedule. The mix of participants affects the outcome of the study. You’d want a robust discussion, or in other words, you’d want participants with knowledge and experience of the project or of the context in which the project took place.

While management of FGD activities follow standard rules, the facilitator/evaluator needs to be flexible because group personality and dynamic differ. I’d be handling four FGDs (of 8 to 10 members each) in one day, for instance, but each group would be different. One group may be more reticent, another would be like a runaway train, etc.; the challenge is to be able to handle each group and discussion effectively. I’ve encountered various groups since my first day in development work and these serve as templates in mixing and matching techniques appropriate to each group. Allowing enough transition time between meeting with the groups to prepare oneself for the next facilitation, also helps.

Another thing in the conduct of FGDs is that surprises are commonplace hence the need to work out beforehand a Plan B or even a Plan C. Once, because of misunderstanding over instructions relayed to local field coordinators via the client (i.e. the specific instruction was to identify 8 to 10 participants), we arrived at the venue to find 50 or so participants who as the coordinators informed us comprised the FGD group. It was two o’clock in the afternoon and the group was supposed to be our last FGD in the province (Quezon) before going back to Manila. I was the only facilitator then (the other had gone back earlier), and with me was the documenter (who wasn’t capable of doing FGDs) and the driver. We were supposed to be in another project area (Batangas) the following week meaning there was no more time to break up the big group, and for security reasons, we needed to leave the area before sundown. After ranting at the field coordinators in the backroom, I went out to meet the 50 eager-looking young people who understandably didn’t have a clue about the situation. I asked one of the coordinators to lead in the introductions which took up a good 20 minutes so I could hash up an alternative approach.

In another FGD, in a school, I was in the middle of facilitating a discussion with a group of students when a series of blares made us jumped out of our seats. There was apparently a flooding and everybody needed to leave. So we did, immediately. We went back at another time to finish the discussion.

Equally important too is for facilitators to prepare themselves for the rigorous field schedule. A mixed methodology of interviews and FGDs with, say, a total of 300 participants would take more or less two weeks, Monday to Saturday or even Sunday. It’s crucial that energy levels are maintained throughout, morning til sundown, day after day. For me, I get re-energized by the sessions themselves– the flurry of activity, discussions, people I’m meeting. A sunny weather, I find, also helps me maintain my mood. I only have to look at the blue sky. Also, physical exercise, like taking a walk at the end of the day to explore the CBD with members of the team works as a de-stressor and refreshes the senses. Then, at the end of the engagement, or after two weeks of nonstop talking and hyper alertness in the field (e.g. keeping tabs of what participants are saying during discussions, one FGD to the next) and afterward the pressure from defending findings, rest, for me, comes in the form of clamming up, mentally shutting down, and focusing on things other than evaluation (some I know tend to their gardens).

Each session presents an opportunity to learn. The evaluation team needs to make time to reflect on and identify, sooner than later, what these learnings are. For example, at the end of the day, the documenter/observer and I would get together, usually at dinner and discuss, among other things, how the sessions went, because the observer may have noticed things that I haven’t while facilitating the sessions. The feedback are then utilized to improve the next day’s sessions.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s