If you are interested in adapting your education to improve student autonomy, you may – like us – be curious to see the impacts of your efforts. Below are the main outcome variables we have studied so far - and if you want to read more, see our Concept Note.
1. Uptake: do students use our tools?
Definition. The extent to which learners exposed to a new teaching tool actually use it in the way that is intended.
Why study this? Understanding uptake is essential to understanding the impact of any product or service.
How to measure it? We use different ways of measuring uptake, depending on the tool you use and the way you implement it in your courses. It is most easily measured if the tool is implemented in a digital (online) way, for example a Teams quiz or similar, as student activity is automatically logged. If you use other implementation approaches, you can still easily measure uptake by asking students to submit their work (e.g. for a pass/fail grade), or by sending a survey in which you ask about uptake directly (could be easily combined with an evaluation of user experience, see below).
2. User experience: how do students experience our tools?
Definition. User experience can be divided into a wide range of dimensions (cf. Zarour et al. 2017), but we choose to focus simply on the extent to which users like different aspects of our teaching tools.
Why study this? Although learning does not always have to be a pleasant experience, we think that a good user experience can enhance the uptake and impact of our teaching tools.
How to measure it? Although you can measure this in any number of ways, including with smiley- or star-ratings, we prefer to ask two simple, open questions in an online survey (e.g. Qualtrics):
-What did you like about the intervention/tool?
-What improvements would you suggest?
References. Zarour, M., Alharbi, M., & Park, E. (2017). User experience framework that combines aspects, dimensions, and measurement methods. Cogent Engineering, 4(1).
3. Learner autonomy: to what extent do our tools help learners become more autonomous and self-regulated?
Definition. We understand autonomy as the ability to direct one’s own learning (cf. Benson 2011). This requires developing an understanding of your goals as well as the ability to regulate your learning behaviour, which involves an iterative process from goal setting to taking action (practicing deliberately) and seeking and using feedback to set new goals.
Why study this? Autonomy is intrinsically important because it helps people give meaning to their lives. Moreover, it helps people shape their life in the way they want and engage with it intentionally and with volition. We believe it is arguably the key learning objective of higher education and will only grow in importance as information sources multiply, the labour market becomes more dynamic, and the world continues to globalize. For learning specifically, autonomy is widely thought to make learning both more meaningful and more effective.
How to measure it. Before engaging with the myriad self-reported measurements of learner autonomy, we prefer two relatively straightforward measures:
1. Recording the responses of the learners to the prompts provided in our educational tools. This provides a direct, objective measure of learner autonomy: to what extent, and in what ways, do students engage in goal setting, deliberate practice, and feedback collection and processing? This rich, qualitative data can be coded for quantitative analysis. It can also be related to the many other, self-reported measures of autonomy.
2. A direct question about the subjective experience of the learner, e.g. “To what extent do you feel this tool helped you to direct your learning process?”
Reference. Benson, P. (2011). Teaching and researching autonomy. Longman.
4. Motivation: do our interventions change student motivation for learning?
Definition. We view motivation as the reasons underlying behaviour (cf. Lai 2011). We are specifically interested in strengthening motivations for learning through the implementation of our tools.
Why study this? Motivation is a necessary condition for meaningful and effective learning. We have therefore designed our teaching tools to enhance learner autonomy as well as their motivation for learning and are curious to evaluate and understand their impacts.
How to measure it. To measure motivation, we use two simple approaches:
1. Ask students directly in a brief survey, before and after our intervention, to rate their motivation for their course(s), research project, or other learning opportunity. Where ethically acceptable, such a study can be set up as an experiment, with a control group without exposure to the prompted reflection tool and a treatment group that does engage with it.
2. Ask students to reflect directly about what happens to their motivation during the intervention, either in a (repeated) survey or live interview.
Reference. Lai, E. R. (2011). Motivation: A literature review. Person Research’s Report, 6.
Here we have described our approach to measuring these outcomes. If you would like us to share a full version of our online surveys that you can use for your students, please get in touch.

Commenti