The Ice-cream Problem

As one of several people responsible for data-related requests about our enterprise Learning Management System (LMS), I was recently involved in several meetings relating to a seemingly simple proposition -- incoming students to the university would self-register for a course in the LMS (using their newly acquired university credentials), and upon successful completion of all of the course modules, they would be eligible for a free scoop of ice-cream at a shop in the student union on campus.

At first blush, it is not surprising that this would be seen as essentially a "data" issue. Our team regularly pulls data from the LMS to investigate trends in LMS usage (number of published courses, assignments per course, etc.), so one could imagine that the core issue to solve is how to query the data to identify the number of completed students for that particular course. And, to be sure, the ability to retrieve that course completion data from the LMS is indeed a prerequisite for the "complete the course, get some ice-cream" idea. 

However, the ability to retrieve the data is only part of the challenge. The data must be retrieved in a timely fashion. Our normal data process is very much retroactive, e.g., "What did LMS activity look like last semester compared to the semester before?". As such, the LMS information in our reporting database lags the LMS system by at least a couple of days. It defeats the purpose of the ice-cream incentive if a student who completed the course has to wait several weeks before that completion is verified in order to get their free ice-cream.

Luckily, the LMS also includes an API, and potentially relevant endpoints seemed to exist for the data that was needed. (The challenge of confirming that the API results did indeed include the data we need without having a sample of the course to use as a test case is an issue for another post.) We do occasionally use the API to get targeted data that is not readily available from our database.  

Another challenge is where the data needed to flow and how frequently. The "free" ice-cream is actually provided in the form of a credit in the amount of one scoop of ice-cream associated with the student's university ID card which can be swiped to be redeemed. This means that the data need to end up in a place where that card swipe system can take over the process. In other words, this is not only a "data" issue, but also an "integration" issue.

Lots of the pain points in integration are hidden if you never have to deal with them. For most of our daily data work, we explicitly control the scripts that we run and have access to the locations where the data are stored, with most of the authentication processes happening seamlessly behind-the-scenes. (And we are there to enter our credentials if prompted.) We don't think twice about manually running a script that points to a local folder or shared drive. But we are logged into the system what happens. To automate the ice-cream data pipeline the script has to hit the API (with the appropriate LMS credentials to do so) and save the data in the appropriate place (with the appropriate file system permissions to do so) via an automatic scheduler that has been provisioned (by someone with the appropriate permissions to do so), and then populate the swipe system (with appropriate permissions to do so), all without human intervention.

As for my part, I thought through the logic that one would need to implement in order to query the API for the current student roster (as student could self-enroll, the roster is always increasing), check the current roster against a list of completed students and filter them out, and then hit the API again to check for completion status of those remaining. I created a proof-of-concept script which implemented the basic logic for the LMS data side of the problem and shared that with others on the team (who then engaged their developers). I'm not sure what, if any, of my proof-of-concept script made it into the final version, but I don't feel slighted. For anything involving non-trivial integration, it makes much more sense to give the appropriate LMS credentials to a software developer who can put all the pieces together rather than trying to make a "data person" into an integration specialist.