Several months back we created an epic on our waffle board that charted the flow from first encountering our site to taking part in a pairing session. We identified the following key components:
- Signing Up
- Browsing existing pairing events
- Creating pairing events
- Taking the pairing event live
- Event notifications
- Event details (or show) page
- Editing existing events
There are still other points that could be smoothed out, but we've done a good portion of the epic. The question that partly troubles me now is how to "put it to bed". A new epic that contains only the remaining issues is probably the way to go, but finally we've got to the point to start analysing some data, since we've got the notifications for the edX MOOC pairing activity flowing to the MOOC Gitter chat fairly reliably and we've just broken through on removing key confusions about joining an event, and working out some problems about the event displaying whether it is live.
This last element is worth looking at in a little more detail as it strongly affects the type of data we are gathering. Creating (and tracking) Google Hangouts for pairing from the AgileVentures site involves creating a Google Hangout that has a particular plugin, called HangoutConnection, that knows the server side event it is associated with. This was originally designed by Yaro Apletov and is written in CoffeeScript. It gets loaded when the hangout starts and attempts a connection back to the main AgileVentures site. Given successful contact an EventInstance object is created in association with the event. This EventInstance includes information about the hangout such as the URL, so that other people browsing the site can also join the hangout without being specifically invited. The HangoutConnection continues to ping the site every two minutes assuming the hangout is live, the plugin hasn't crashed and so on.
What Michael and I identified on Wednesday was that only the first of these pings actually maintained the live status, making it look like all our pairing hangouts were going offline after about 4 minutes. This had been evidenced by the "live now" display disappearing from events somewhat sooner than appropriate. This might seem obvious, but the focus has been on fixing many other pressing issues and usability concerns from the rest of the epic. Now that they are largely completed this particular problem has become much clearer (also it was obscured for the more regular scrums which use a different mechanism for indicating live status). One might ask why our acceptance tests weren't catching this issue. The problem here was that the acceptance tests were not simulating the hit of the HangoutConnection to our site. They were manipulating the database directly, thus as is often the case, the place where the bug occurs is just in that bit that wasn't covered by a test. Adjusting the tests to expose the problem made the fix was relatively straightforward.
This is an important usability fix that will hopefully create better awareness that hangouts are live (with people present in them), and increase the chances of people finding each other for pairing. There's a lot more work to do however, because at the moment the data about hangout participants that is sent back from HangoutConnection gets overwritten at each ping. The Hangout data being sent back from HangoutConnection looks like this:
{
"0" => {
"id" => "hangout2750757B_ephemeral.id.google.com^a85dcb4670",
"hasMicrophone" => "true",
"hasCamera" => "true",
"hasAppEnabled" => "true",
"isBroadcaster" => "true",
"isInBroadcast" => "true",
"displayIndex" => "0",
"person" => {
"id" => "123456",
"displayName" => "Alejandro Babio",
"image" => {
"url" => "https://lh4.googleusercontent.com/-p4ahDFi9my0/AAAAAAAAAAI/AAAAAAAAAAA/n-WK7pTcJa0/s96-c/photo.jpg"
},
"na" => "false"
},
"locale" => "en",
"na" => "false"
}
}
Basically the current EventInstance will only store a snapshot of who was present in the hangout the last time the HangoutConnection pinged back; and data from pings after the first two minute update has been being discarded. We're about to fix that, but here's the kind of data we can now see about participation in hangouts:
#participants #hangouts
Monday:
1 *
Tuesday:
1 *
Wednesday:
1 ****
2 *
3 **
Thursday:
1 ****
Friday:
1 *
2 *
3 *
Saturday:
1 *
Sunday:
1 ***************
2 *
3 *
Monday:
1 ******************************
2 ****
3
4 **
The above is just a snapshot that corresponds to the MOOC getting started; we're working on a better visualisation for the larger data set. We can see a clear spike in the number of hangouts being started, and a gradually increase in the number of hangouts with more than one participant, remembering that the participant data is purely based on who was present at two minutes into the hangout.
If the above data was reliable we might be saying, wow we have a lot of people starting hangouts and not getting a pair partner. That might be the case, but it would be foolish to intervene on that basis using inaccurate data. Following the MOOC chat room I noticed some students at the beginning of the course mentioning finding hangouts empty, but the mood music seems to have moved towards people saying they are finding partners; and this is against the backdrop of all the usability fixes we've pushed out.
To grab more accurate participation data we would need to do one or more of the following:
- adjust the EventInstance data model so that it had many participants, and store every participant that gets sent back from the HangoutConnection
- store the full data sent back from every HangoutConnection ping
- switch the HangoutConnection to ping on participant joining and leaving hangouts rather than just every two minutes
- ruthlessly investigate crashes of the HangoutConnection
Personally I find this all intensely fascinating to the exclusion of almost everything else. There's a real chance here to use the data to help adjust the usability of the system to deliver more value and more positive learning experiences.
No comments:
Post a Comment