From Data Overload to Real Impact: A Case Study in Monitoring, Evaluation, Research and Learning
by LSIG MERL Team
12/5/20255 min read
“Thank you for the outstanding report you produced, which allows us to connect more deeply to our mission. Notably, this marks the first time a comprehensive report has been developed, whereas in previous years we relied solely on primary data compiled in Excel"
— Program Director
MERL Frameworks: Why Even the Most Robust Systems Fall Short of Expectations
The Client's Challenge: Good Intentions, Broken Process
The client was an organization genuinely committed to the communities they serve. They had survey instruments in place, a team in the field, and a process for submitting evaluation reports. On paper, the monitoring and evaluation system existed. In practice, it was not working, and no one could agree on why.
When our Consultant began the engagement, the first thing we did was look at the full process from end to end: from the survey instruments themselves, through the quality control mechanisms, all the way to how the final evaluation report was produced and used.
What we found was not a single problem. It was a chain of disconnected ones.
The Client's Challenge: Good Intentions, Broken Process
The Question That Started Everything
How do you effectively evaluate the performance of your programs and the impact they have on the communities you serve, when you are not entirely sure what data you should be collecting in the first place?
It is a question more organizations face than are willing to admit. Some do not yet recognize the strategic importance of data collection in monitoring and evaluation. Others understand it in principle but have collected data without a clear plan for what to do with it next. And others have collected so much data that the evaluation process has become overwhelming. In the field of project and program management, we call this infobesity: an excess of information that paradoxically leaves decision-makers with less clarity, not more.
None of these situations must be permanent. And that is exactly what one organization discovered when they approached LSIG.
What We Found: A System Under Strain
Supervisors and enumerators were talking past each other. Supervisors attributed data quality issues to enumerators not following the established guidelines. Enumerators, on the other hand, raised concerns about questions and instructions that were not clear enough to execute consistently in the field and felt that those concerns were not being heard or taken seriously. Both sides had legitimate points. Neither had a forum to resolve them together.
The survey instruments themselves needed work. When LSIG facilitated a joint review session with directors, supervisors, and field teams, it became clear that some of the issues attributed to fieldwork execution originated in the questionnaire design. This insight allowed us to pivot, adding critical variables and sharpening technical guidance to ensure the instrument was perfectly calibrated for the operational reality. No amount of field training could compensate for instruments that presented weaknesses. It is important to note that a key aspect of monitoring and evaluation is first identifying the most relevant indicators to track and determining the most effective methods for collecting the data. In many cases, a mixed-method approach combining quantitative and qualitative data can provide more meaningful insights.
Fieldwork supervision lacked the right tools and structure. Beyond the questionnaire, the mechanisms for supervising data collection in real time, in the field were insufficient, creating inconsistency in how data was gathered, where data was gathered, and how data was uploaded in the system hence reducing confidence in the reliability of what had been collected.
The data existed, but its potential was untapped. Perhaps most significantly, the organization had accumulated data from previous cycles that had never been properly analyzed. They had the raw material for meaningful insight. They simply did not yet have the framework to extract it.
What LSIG Did
The engagement unfolded in four connected stages.
Stage 1 — Instrument and process review.
We began by conducting a thorough review of the existing survey instruments, assessing quality control mechanisms, and mapping every team and handoff point involved in the monitoring, evaluation and data collection process. This gave us a complete picture of where the system was working and where it was breaking down.
Stage 2 — Unified training across all levels. Rather than addressing supervisor concerns and enumerator concerns separately, which would have reinforced the disconnect between them, we designed and delivered training modules that brought all parties together. The sessions drew on past experiences from both supervisors and enumerators, establishing shared best practices and creating a space where concerns on both sides could be heard, validated, and resolved. Leadership was part of this process: a focused session with the organization's directors and managers enabled to scrutinize survey instruments and pre-fieldwork protocols. This proactive approach ensured that the team was fully equipped and the methodology fully vetted, before the next phase of data collection began, not just ensuring that all pre-fieldwork preparations were just completed but they were strategically aligned with the project’s long-term objectives.
Stage 3 — Instrument revision and supervision framework. Based on the findings from Stage 1 and the input gathered in Stage 2, we revised the survey instruments to eliminate ambiguity and improve consistency. We also established a structured supervision framework with the right tools for field management hence giving supervisors what they needed to support enumerators effectively rather than manage them reactively.
Stage 4 — Statistical analysis and deep insight reporting. With a cleaner process and revised instruments in place, we conducted a rigorous statistical analysis of the data that had already been collected. The resulting report went well beyond a standard evaluation document. It delivered deep, specific insights that showed exactly where the organization stood in relation to its stated objectives and the commitments it had made to its stakeholders and the communities it serves.
The Outcome
The organization's response to the final report was telling. It was the first time, they said, that they had received insights at that level of depth and specificity from an evaluation exercise. Not because the data had not existed before — it had. But because no previous process had been designed to extract what the data was actually capable of showing.
That distinction matters. There is a significant difference between having data and knowing how to use it. There is an equally significant difference between producing an evaluation report and delivering insights that genuinely inform strategy, strengthen accountability to stakeholders, and demonstrate impact to the communities an organization serves.
About the Practice
LSIG’s Monitoring, Evaluation, Research and Learning practice supports organizations in building evaluation systems that are rigorous, sustainable, and genuinely useful.
The Broader Lesson
This engagement reflects a challenge LSIG observes across many organizations in the nonprofit, community development, and public sector and even private sector space. Monitoring and evaluation systems are often built around compliance, the need to produce a report, rather than around learning and decision-making. When that is the case, the evaluation process feels burdensome rather than valuable, and the insights it could generate are left on the table.
The path forward is not necessarily more data. In many cases, it is collecting better data — by clearly identifying the right indicators to track, designing clearer survey instruments, ensuring stronger supervision, and conducting deeper analysis to connect what was done to what actually changed in the lives of the people being served.
That is what effective monitoring, evaluation, research and learning looks like in practice. And it is what every organization committed to real impact deserves to have.
LSIG's Monitoring, Evaluation, Research and Learning practice supports organizations in building evaluation systems that are rigorous, sustainable, and genuinely useful, from instrument design and field supervision to statistical analysis and strategic reporting. Contact us to discuss how we can support your next evaluation cycle.
CONNECT:
© 2026. All rights reserved.
LSIG
HELPFUL LINKS
TERMS OF USE
COMPANY