Introductory programming courses are known worldwide to pose challenges for both students and educators. A recent meta-review of research in the area has indicated something in the order of a sixty six percent pass rate globally. Yet the New Zealand Government has asked institutions to set high and increasing targets as a goal for student pass rates in its educational performance indicators. Increasingly these metrics are being used to shape the behaviour and educational outcomes sought from educational institutions, with the threat of penalties by way of loss of funding for supposedly “poorly performing courses”. Yet while focused at the institutional level, how do these indicators really meet the needs of all the stakeholders in the tertiary education system? To what extent do they distort and create incentives for perverse behaviours? This review assesses the dilemmas such measurement systems pose to educators using the case of introductory programming as an example.