Friday, June 13, 2014

Thoughts about pair roulette, pairing in MOOC

One pedagogical argument is that pairing should be restricted to just pairs.  By having only two people in the pairing session we increase the likelihood that both are doing active work.  Others may benefit from observing, but they are also reducing the pool of other possible pair partners, and in the case where students are pairing on homeworks that are being assessed, there might be an argument to suggest that observers are freeloading.

However, taking an active part in a pair is very intimidating for many and so it's a big advantage in some ways if random pairing sessions are visible for observation and/or some degree of interaction with a wider group of learners.

There is a tension between the desire for each individual learner to operate in the manner that feels most comfortable at any given time (working solo, working actively in a pair, observing a pair etc.) and the desire to assess learner's abilities.  If we were only concerned with promoting learning, then we would not necessarily impose restrictions on observers, although there is the further case of where learners would like to pair, but would like to restrict who is observing for privacy reasons.  Privacy reasons are unclear to the current author, but seem to revolve around the fear of aspects of one's personality being displayed for others to judge?  Online remote pair programming does not require that people display video of themselves, or even necessarily any audio, but still one's typing is exposed in real time, and one's decisions about what to type next, what code to create, are being exposed, in the much the same way that they might be in an oral exam, and many people are intimidated by interviews and oral exams.  Superficially this is related to a fear that one will be judged as not having made the grade, but the author would love to hear insights from others on this.

Ideally learning situations such as remote pairing should not revolve around judgement of each others abilities, and provide a supportative environment for learning, but naturally all learners will be making judgements of each other (and themselves), such as relating to their partners language ability, coding ability, and so forth.  Some learners are in a hurry and may feel that they don't have time to be pairing with someone they consider as being less skilled in areas they want to improve in.

The author would suggest that the idea of placing learners on a linear spectrum of ability does not make much sense.  Everyone's understanding and skills are a complex multidimensional entity.  For example one individual may be very confident in Ruby String manipulation, but much less confident as regards OOP and another learner's understanding/confident may be exactly the opposite.  As regards coding one's number of years programming is often considered a guide towards "ability level" but still there is no clear linear relationship, and learners might well be advised to be patient and discover what they can learn from every possible pair partner.

Having all pairing sessions recorded is arguably a good move since it allows analysis of the pairing, ensure that any disputes about behaviour can be resolved with relation to a video of what actually happened.  Knowledge that one is being recorded (if not directly observed) serves as an incentive to behave more congenially.  The downside of recording is that it will increase the nervousness of some, and might prevent them from participating in the pairing session as actively as they would like.

Assuming that one can host pairing sessions and drop 2 or more people into them, and record them; the question arises about the best mechanism of pairing people up.  One of the key issues seems to be people's shifting schedules and being ready to pair.  If you add your name to a list of people wanting to pair, will you be available at the time someone else is ready?

If you have some mechanism for checking that someone is actually looking at a given page you could indeed have a list of people ready to go, although people may still be AFK (away from keyboard).  It's an interesting question about whether people should be asked to register their interests for a pairing session, e.g. Homework 2, Project X, or open to suggestions.  Should you have a single list of everyone ready for impromptu pairing, or should their be some set of lists divided up at some granularity - it seems like that depends on the numbers involved.  In the first instance perhaps one should just have a single list, until one demonstrates the need for more ...

There is also the question about whether one should have a list of ongoing sessions that others can browse and then join.  In the open source project case it seems the answer should be a resounding yes.  In the student case it seems we are caught between the desire to showcase the system running in progress (and allowing others to learn by observing), and the desire to afford students with additional privacy, and reduce perceptions of freeloading.

If one is offering a service for free then one can argue that participants have a degree of obligation to share what they are doing, i.e. contributing back, although of course the flip side is that if participants feel uncomfortable then they will just not participate.  It seems reasonable to suggest that a premium version of the system would allow privacy to be ensured; since participants are contributing to the system by paying for it.

Finally there is the question of how to effectively scaffold pairing sessions with novice pair partners.  One can imagine step by step pairing walkthroughs, but it is currently unclear what the best mechanism would be to insert these into a pairing session?  Perhaps a cloud9/nitrous session where the computer would prompt the pair partners to switch roles and have certain sorts of discussions at certain times.  A human coach can scaffold that with some degree of skill - how much of it can we automate?

Monday, July 29, 2013

Sam Joseph on MOOCs for Hawaii Business Magazine

I was recently interviewed by Hawaii Business Magazine's Pavel Stankov on the subject of Massive Open Online Classes or MOOCs

Pavel: So first off, tell me about the public class that you taught online at HPU? What class was it? How long was it taught?

Sam: The class was the combined CSCI 4702 Mobile programming and MULT 4702 Mobile Design classes, and was taught over the usual 14 weeks of the HPU Spring semester this year.  It's a class that I've been teaching in one form or another at HPU and UHM for about 8 years, and focuses on the design and programming of applications for mobile devices such as tablets and smartphones.

Pavel: Who pitched the idea for that and how successful was the course? Did anybody drop out?

Sam: I designed the course myself and I pitched the idea that HPU should trial it as a public beta.  I would say the course was moderately successful :-)  I continue to get the positive feedback that I have gotten from students on this course as I have over the years.  All of the official HPU students enrolled in the course completed it, and the public format enabled some who had previously failed the course to
retake and successfully complete the course.  The majority of the non HPU students taking the course did not complete it, but I still got positive feedback from them.  I don't know that "dropping out" as a concept is particularly useful for students who are not studying for credit.  If for-credit students "drop out" it is clearly a negative event where a student has paid for support in their learning process, and for whatever reason, feels that they are not getting what they expect from a course.

When casual "MOOC" students are taking a course it is much more like they are receiving an encyclopedia.  They are gaining access to a set of materials that they can take and pick from as they please.  Since in the HPU public course trial we were not certifying their abilities I don't believe that their "drop out" rate of non-HPU students indicates the relative success or failure of the course.

Pavel: Would it be considered a MOOC, if it's not delivered through Coursera, edX, or Udacity? Was there a third party at all, or was it offered directly from HPU, just open to the public?

Sam: The first MOOCs were offered before Coursera, EdX or Udacity existed so I don't think that
who provides a course says anything about whether it should be considered a MOOC or not.
Whether we consider something a MOOC or not depends on four things, specifically it being
a course, it being available online, it being available to the public, and it being taken
by a large number of students.  Our online public course trial had around 20 students combined so I think it qualifies more as a SOOC (Small Open Online Course) than anything else :-)  I used the free and open Google Sites framework to host and deliver the course along with other free open source tools and Google App Scripts that I programmed myself.

Pavel: What class are you currently teach through edX? What is the turnout? What are the expectations and what do you hope to achieve through offering it? Have you offered a course of such a scale before? How do you feel about it?

Sam: In collaboration with UC Berkeley's Professors Dave Patterson and Armando Fox I am facilitating CS169X Software as a Service through EdX.  The current instance of the class
has over 13000 students enrolled and close to 250 teaching assistants.  The expectation
is that we can spread as widely as possible the concepts of software engineering craftsmanship.  My personal hopes are that by being closely involved in this course that my own HPU Software Engineering course will become even more valuable to the students taking it, in terms of the quality of the curricular materials they have access to, and the range of other learners they can interact with.

Sam: One of the key values of delivering a for-credit class publicly with a mix of for-credit and casual students is that the students get to mix with a much wider range of learners.  Students can take part in collaborative learning with people from all over the world, who in many cases bring fantastic industry experience with them to the class.  My involvement in this summer's EdX course is the first time that I have taken a major role in a class of this scale and I am extremely excited about it.  I see the combination of MOOC delivery systems such as EdX with personal scaffolded collaborative learning experiences such as pair programming and group projects revolutionizing the nature of the educational experience.

Pavel: What are the challenges for a MOOC instructor? What is the hardest part? The easiest part?

Sam: The challenges are having a continous flow of information with questions from students coming in constantly 24/7.  All materials have to be of exceptionally high quality. The hardest part is often just switching off for a moment to refresh yourself.  Delegation is the combined hardest and easiest part.  Given the 250 or so teaching assistants I have to control myself not to dive too deep on problems from individual students the moment they come up, leaving the teaching assistants to triage the challenges the students are facing.  The MOOC instructor must listen carefully to their teaching assistants balancing when to jump in with their expertise so as to benefit the maximum number of students possible.

Pavel:  If you were to start over, and change something in the way you approach your MOOCs, would you do so?

Sam: I am making an ongoing effort to change MOOCs so that they move away from the one size
fits all mode of learning, and focus on the individual.  I think real time interaction
with fellow MOOC students and open-ended projects of consequence to the individual
students is the key.  Certain Stanford Coursera classes have trail-blazed in this regard such as Scott Klemmer's HCI course, however I think we can go a lot lot further.

Pavel: How do you address the issues of academic honesty?

Sam: Academic honesty is an issue in a class that tries to offer credit for a largely one size fits all method of assessment.  My personal approach to academic honesty is to award credit for unique individual contributions.  To the extent possible within the contexts of HPU, Berkeley and EdX I make all my classes dependent on a students ability to offer unique individual project work.  In a class where projects are developed incrementally it quickly becomes clear if an individual is trying to attribute the work of others to themselves, even in the online context.   This is of course more complicated at large scales, but I believe we have the seeds in place to make academic honesty effectively a non-issue, and you are likely to see some very exciting developments in this regard over the next 24 months or so.

Pavel:  What is your take on the peer-grading approaches offered by some MOOC providers? Do you suppose there might be a better technology to control for plagiarizing?

Sam: I think peer-grading is an interesting approach.  I've used it in some of my classes, and I've taken MOOC classes in which it has been used.  I don't think it is yet quite in a form that delivers an ideal learning experience but it's an excellent start.  Since I teach programming and design, plagiarising is not quite the issue that is in other classes such as English and History, however it is still a concern.  I have also been taking a very interesting MOOC on cheating in online classes, and I think it's a mistake to focus on plagiarism and how to control it.  I believe the focus should be on igniting the imagination of individual students.  The key is for the instructor or teaching assistant to get to know the individual students personally, and ask them what it is they are really excited about doing.  The focus should be on providing a framework, scaffolding if you will, that enables the student to do something that they are excited about, giving them access to the tools that allow them to achieve their dreams.  I would argue that plagiarism and academic dishonesty comes largely from students not being interested in performing the academic exercises they are being set.  The solution is not to ask how to control plagiarism through technology, but what is it that interests an individual student?  Unlock the interest of the individual student and they will have no incentive to be academically dishonest; they will be truly motivated to create something of quality, and developing the necessary skills to support that.

Pavel: Do you think some classes are better suited to be taught through a MOOC than others? Which ones?

Sam: Given the form of the current MOOCs from providers such as Coursera, EdX and Udacity, I would say that computer science courses are particularly well suited to being taught through a MOOC.  One might argue that introductory courses are also well suited to MOOCs, however I am not sure I agree.  I think the being suited to a MOOC depends on who's perspective we are talking about, e.g. educator, student, institution etc.  However with the new technologies that are rolling out in terms of remote collaboration software such as Google hangouts, Multiway Skype screenshare etc. the majority of classes can be taught just as well in MOOC format, if not better than at a physical institution.  Although a clear exception would be those courses that are attempting to train or instruct in the use of expensive specialist equipment that is not available to the individual at their home.

Pavel:  Under what circumstances should colleges and universities award academic credit for third-party MOOC providers?

Sam: At the moment I think it's not in colleges and universities interest to award academic credit for third-party MOOC providers unless assessment has been verified through proctored examinations.

Pavel: How do you see the development of MOOCs in the observable future?

Sam: Difficult to say although my personal hope and plan is that MOOCs will become a lot more accessible and personal with MOOCs supporting students to talk in real time with their peers and instructors.

Pavel: Is there anything in particular about Hawaii that makes our location different when it comes to online higher education?

Sam: Hawaii of course has a large military presence; and military students really must have online education due to the nature of their work.  Of course Hawaii is also geographically isolated meaning that the range of educational choices is not what it is in other areas, making online education particularly valuable to Hawaii residents.

Pavel: How do you react to comments that MOOC are undermining higher education? For instance, some people are inclined to say that if awarded with academic credit, they would have no incentive to physically go to school when they can take everything online for free or a symbolic fee? Do you think this is a valid argument? Do you think this is a problem for faculty?

Sam: I guess the argument is that as MOOCs start awarding academic credit then many students might not attend a bricks and mortar "academy", thus undermining higher education institutions?  I think it's entirely possible that MOOCs may undermine higher education institutions that are not offering the highest quality of courses and value for money to their students.  I don't think that anything that itself promotes education can undermine education unless one posits that students receive some particular benefit from attending a physical institution that they cannot receive online.  Individuals have been taking distance and online courses around the world for many many years.  Here in the UK the Open University has been delivering higher education without physical institutions for some 30 odd years, and the UK still has a very strong set of higher education institutions in the physical domain.  I think it is an open question as to the value that students receive from attending a physical institution, and the most important thing in a free market is that of choice.  Students should have a free choice as to where they look for support to achieve their learning goals, and should not be paying over the odds for the support they receive.  I think the only challenge that MOOCs and other educational technology developments present for faculty is for those faculty who are not delivering the highest quality educational experience possible, and for those institutions who are charging over the odds.

Pavel: Finally, is there anything that you want to add, or a question that i should have asked you?

A general comment would be that globally I think we have education back to front.  Rather than asking what skills we should be teaching students and what subjects they should be learning, we should be asking our students what excites them, what they want to learn and what they want to build.

Friday, May 24, 2013

Scheduling Remote Pair Programming Sessions

So maybe it's just me, but it feels like there has been a recent explosion in interest in remote pair programming. Maybe it's being going on for ages, but I recently discovered great resources such as Joe Moore's and

I've now added the PairWithMe badge to all my projects on Github, and I follow the #pairwithme hashtag on twitter, which is exposing me to things like this remote pair programming meetup group:

@batarski and @eee_c were saying on twitter that the #pairwithme website should coordinate and maybe host the sessions.  I gave a lame tweet reply that that would be cool, and that some other sites were doing that, but didn't really have space to list the, so here they are:

And there was another one I signed up with recently, but I lost my Chrome history in a recent crash, so can't find it at the moment :-(

However although I've made contact with a few people through these (mainly just on twitter) I think I've only got a couple of remote pair programming sessions going from them.  Far more intensive has been the now 30+ and counting hours I've been remote pair programming on our EdX SaaS LocalSupport project which we are mainly co-ordinating through Skype chat rooms.  Here's a playlist of those videos

I'm looking forward to seeing more solutions to the challenge of remote pair programming scheduling.

Saturday, April 27, 2013

HCI Wireframing Assignment: Pair Programming Scheduler

These prototypes *very* loosely based on my earlier storyboards - basically I switched away from interventions during actual pairing, to think about scheduling pairing sessions.

Prototype 1 where the user indicates their availability first:

Prototype 2 where the user browsers who is available first. (more like the tweet aggregator)

Thursday, April 25, 2013

What I'd do with a ScreenHero API

So Screenhero rocks, but I was thinking it would be really cool if it had an API so you could scaffold the interaction between pair programmers collaborating on code.

Imagine that you've got a couple of relative novice programmers who are new to pair programming and to related concepts like TDD.  Think about being able to have suggestions pop up once they had completed certain parts of the task.  You could do this without plugging directly into ScreenHero, but what would be nice would be to ensure that one person did one portion of coding, followed by the other person doing another.

I guess what I should really do first is work out the precise flow of a scaffolded pairing session and then think more carefully about what prompts the users would need, and how to keep track of who's doing the work.  Ultimately we want to be able to have a reliable automated check that actual pair programming is happening rather than just one person doing all the work.

Perhaps having each person check their code into github separately - but that then pulls the pair apart.  I guess it raises the question of how to attribute code changes in a version repository when two people are working on the same file at the same time on one computer ...

Follow up - just watched great talk by Sam Livingston-Gray, which mentioned a pair programming protocols "Ping Pong" and some variations called "Change the Message" and "one-undermanship":

Looks like ping pong protocol goes way back.  I think this is what I'd really like to be scaffolding.

I wonder if the best way to scaffold would be through checkins to a Github repo - then at least you could check in retrospect how people coded ...

More HCI Bloopers

So I do appreciate the folks at Cengage Learning providing me online access to a textbook via the coursesmart system.

I was less excited about getting locked out of my account for skimming quickly through the book.  Then to unlock it there was a helpful chat window, but the multiply nested form I had to fill in to get into the chat itself was a great HCI blooper in my opinion.  It took me three or four attempts to work out which category "account locked" should go under and in the end it was none of them, so I just wasted 3 extra minutes, and eventually chose some random sub-categories:

The guy in the chat did help me out pretty promptly, but didn't seem very interested in my suggestions regarding the lack of skimming functionality or the problems with the form.  I did recommend that he direct their interface designers to the free Stanford HCI course but he didn't seem particularly interested ... :-)

Monday, April 22, 2013

Remote Pair Programming for the Masses

I am submitting this project for funding from my University.  It concerns creating an online calendar in which students from around the world can register their interest in remote pair programming.  Evidence is mounting that pair programming is the most effective way to learn how to program, and also generates higher quality code faster.  One of the key challenges is finding someone to pair with; however an appropriately structured calendar that took into account timezone differences, programming skill level etc., could create an environment where there was always someone ready to pair program with you day or night.

Project Description

Pair programming is the process whereby two programmers work together on the same coding problem.  Usually one will act as a navigator and the other as a driver with control of the keyboard.  The central idea is that there is an immediate feedback loop whereby the navigator can check the code the driver creates and the two can discuss possible alternatives.  This is similar to the kind of feedback that may come from code reviews and project retrospectives, but is on a much smaller timescale.
Pair programming is being increasingly adopted in industry although there is a general resistance based on the fear productivity will be lower given that two programmers could be working on two different tasks.  However Cockburn & Williams (2000) demonstrated the following significant benefits of pair programming:
  • many mistakes get caught as they are being typed in rather than in QA test or in the field (continuous code reviews)
  • the end defect content is statistically lower (continuous code reviews)
  • the designs are better and code length shorter (ongoing brainstorming and pair relaying)
  • the team solves problems faster (pair relaying)
  • the people learn significantly more, about the system and about software development (line-of-sight learning)
  • the project ends up with multiple people understanding each piece of the system
  • the people learn to work together and talk more often together, giving better information flow and team dynamics
  • people enjoy their work more

Cockburn & Williams go on to point out that the development cost for these benefits is not the 100% that might be expected, but is around 15%; which is quickly paid back for in shorter and less expensive testing, quality assurance, and field support.  Of particular note are the learning benefits, which have also been noted by Williams & Kessler (2003) who indicate that:
"Knowledge is constantly being passed between partners, from tool usage tips to design and programming idioms. The partners take turns being the teacher and the student. Even unspoken skills and habits cross partners."
Canfora et al (2007) found a reduction in defect rates of 15% to 50%, varying depending on programmer experience and task complexity, while a number of other studies found similar improvements (Lui, 2006; Arisholm et al., 2007; Lui et al., 2008), however meta-analyses have cautioned that this all depends on a number of factors and that you cannot expect “faster and better and cheaper” (Hannay et al., 2009).

Independently of whether pair programming is more effective than solo programming, the educational benefits seem clear.  McDowell et al (2002) and Salleh et al. (2011) both found that students’ grades and satisfaction improved with pair programming as compared to programming solo.  Furthermore McDowell et al. (2002) saw increases in retention indicating that the use of pair programming could prevent students from dropping out.

Schümmer  & Lukosch (2009) describe some of the additional difficulties faced when pair programming remotely, and there are also concerns that productivity and satisfaction can drop during novice-novice pairing is used without sufficient availability of a mentor for coaching (Stephens & Rosenberg, 2003).  All of this indicates that there is a huge pedagogical potential for remote pair programming, but that the process needs to be carefully scaffolded to ensure successful outcomes.

Specific Aims
  • Create online calendar for remote pair programming (RPP)
  • Provide support for editing, discovery and matchmaking
  • Scaffolding successful RPP sessions
  • Recording RPP sessions for analysis
  • Using results of analysis to improve the RPP experience


The significance of this work is that it is becoming clear that pair programming is the fastest and most effective way to learn how to program, and yet in general it can be very challenging to find pair programming opportunities.  A well organized calendar that allowed students from around the world to discover other students available to pair program could be invaluable in increasing student satisfaction and minimizing the amount of time required to become a confident programmer.

Work so far

We have created a shared Google calendar and sign up form that now has 30 students signed up.  Remote pair programming sessions have been recorded with at least 10 of these students and a preliminary analysis of one session has been conducted.

  • Arisholm, Erik; Hans Gallis, Tore Dybå, Dag I.K. Sjøberg (February 2007). "Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise". IEEE Transactions on Software Engineering 33 (2): 65–86.
  • Canfora, Gerardo; Aniello Cimitile, Felix Garcia, Mario Piattini, Corrado Aaron Visaggio (2007). "Evaluating performances of pair designing in industry". The Journal of Systems and Software 80 (80): 1317–1327.
  • Cockburn, A. & Williams, L. (2000). "The Costs and Benefits of Pair Programming". Proceedings of the First International Conference on Extreme Programming and Flexible Processes in Software Engineering (XP2000).
  • Hannay, J. E.; Dybå T., Arisholm E., Sjøberg D. I. K.  (2009). "The Effectiveness of Pair Programming: A Meta-Analysis". Information and Software Technology 51 (7): 1110–1122.
  • Lui, K. M. (2006). "Pair programming productivity: Novice-novice vs. expert-expert". International Journal of Human-Computer Studies 64 (9): 915–925
  • Lui, K. M., Chan K. C. C., Nosek J. T. (2008). "The Effect of Pairs in Program Design Tasks". IEEE Transactions on Software Engineering 32 (2): 197–211.
  • McDowell, C., Werner, L., Bullock, H., & Fernald, J. (2002). The effects of pair-programming on performance in an introductory programming course. ACM SIGCSE Bulletin, 34(1), 38-42.
  • Salleh, N., Mendes, E., Grundy, J.C. (2011) Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review. IEEE Trans. Software Eng. 509-525.
  • Schümmer, T. & Lukosch S. (2009) "Understanding Tools and Practices for Distributed Pair Programming". Journal of Universal Computer Science 15 (16): 3101–3125.
  • Stephens M. & Rosenberg D. "Extreme Programming Refactored: The Case Against XP", Berkeley, CA: Apress, 2003.
  • Williams, Laurie; Kessler, Robert (2003). Pair Programming Illuminated. Addison-Wesley. ISBN 0-201-74576-3.