Of X-Rays and Learning Analytics

 Unless you have been asleep for the past several years, you will have no doubt heard of the machine learning and artificial intelligence revolution. Even before ChatGPT made headlines as having the fastest growing user-base in history, most large organizations were already looking to leverage machine learning and AI models in their business to take advantage of the ever expanding amount of data collected by digital systems -- "Data is the new oil" as they say.

The advances in AI and machine learning in recent years have indeed been quite astounding. With a large enough data set and sufficient computing power, tasks once thought to be the exclusive domain of human experts are now being performed by digital systems with accuracy that matches or even exceeds human benchmarks. Machine learning expert Andrew Ng predicted in 2017 that if you are an expert radiologist (someone who scans X-Ray images for evidence of disease), you are more in danger of being replaced by an artificial system than by your best lab assistant, a sentiment that has been widely echoed in various media outlets ever since. 

Given this state of affairs, it seems obvious that institutions of higher education, if they want to survive in the AI-dominated world of the future, need to embrace the digital revolution, and that educational leaders who continue to cling to outdated modes of interaction will see their institutions falter and perish in the not too distant future. By taking full advantage of the copious amounts of data collected by educational technology systems and judiciously applying machine learning algorithms, educators will be better able to serve students where they are, improve outcomes and retention, and identify potential student mental health issues in a timely fashion leading to more effective intervention. 

Not so fast. While it is true that there is a lot of room for innovative thinking in higher education, one needs to proceed with caution. In a more recent interview in 2021, Andrew Ng admitted that the early exuberance over AI's capabilities was slightly overstated. The issue is not that the models were not good -- the AI models as created did indeed outperform the human baseline  -- but rather that they were not generalizable. Move the trained model to a different hospital, which uses a slightly different X-Ray machine with slightly different resolution, and you are back to square one. (The expert human, of course, has no problem reading X-Rays from different machines.) To be effective, models need to be able to differentiate the signal from the noise. To do this, there needs to be consistency in the signal.

Now think about the incredibly wide range of interactions captured as data in educational technology systems. Think about all of the different ways that instructors can run their courses, that students can interact with the materials, that classroom and interpersonal dynamics can impact communication styles. Even the "signals" that we have in the data are contested ("Grades don't tell you about effort or learning," "Attendance does not mean engagement," etc.). Is it really reasonable to think that algorithms will be able to solve the educational challenges that educational professionals have not been able to solve themselves?