Request Table, Finally!

This post will only be meaningful for those who deal with Canvas data from the Canvas LMS (learning management system).

Our institution has been using Canvas since around 2013 (first as a pilot, then as the official LMS). One of the data tables available to us through download is the request table -- essentially a record of all of the web requests in our Canvas instance since we began using it many years ago. Given that we have tens of thousands of people using the system each day, this represents a lot of data.

I've recently blogged about the difficulties in extracting data from the request table. For the past year or so, we have been unable to unzip the data from the request table as it has been too big to fit on the team laptop.

Yesterday, I was finally able to extract all of the data onto a networked server. Or, to be more accurate, I was able to unzip the individual files (over 10,000 currently) that make up the request table data. I have yet to successfully run the canvas-data-cli script on the request data, though it works fine for everything else.

At the end of the day, we have 2.4 TB of request table data when unpacked.  Now that we know what we are dealing with, we'll be able to work towards building a pipeline moving forward. .