Are We Measuring Student Engagement or Just Student Compliance?

 I've recently begun toying with the notion that many of the learning analytics metrics that we measure are likely to be telling us more about student compliance with course requirements than student engagement per se. This is particularly true when looking at data from an LMS (learning management system). Unlike an online tutoring system, where learners are actively engaging in instruction and practice activities while they are using the tool, an LMS is primarily just a repository of course materials. This is especially likely if the course is being taught face-to-face (ah, those were the days). Thus, metrics like time spent in the LMS are, at best, measures of "engagement" with the LMS itself, not necessarily the learning materials contained therein.

 Of course, there are "interactive" elements to LMS courses, such as discussion boards and quizzes, and the shift to remote learning has created a greater reliance on these tools. Even so, students are not likely to make use of them unless they are required elements of the course. If a course requirement is that students respond to at least three discussion posts by other students, most will dutifully do so. In a course where the discussion board is optional or not graded, there will be very little usage.

One could instead use very broad measures of engagement, such as whether or not a student visits the course site at all during the week.While such measures might be better correlated with ultimate attainment in the course, we are still essentially measuring as aspect of compliance, e.g., attendance. 

The challenge is not just that we are using proxy measures, which we are, but that we use "engagement" as a shorthand for a positive or desirable attribute without defining exactly what we mean. And, if we are using course outcome as the dependent variable for the effect of engagement, what does that mean? Are "easy A" courses by definition the most engaging? Are students who get higher scores in harder courses "more engaged" than students with lower scores?

Like so many things in life, the more I ponder, the less I know.