Automated Text Chat Assistants for Online Classes

I'm proposing this research project at my University in the hope of getting some course releases to spend more time on it. The proposed project concerns analyzing the text and email logs from a number of online computer science and multimedia communication classes in order to identify common students questions and concerns.  This data will be used to improve course design and automate the generation of frequently asked questions (FAQs) web pages.  Longer-term goals include the developing an instant messenger (IM) chatbot that can interact with students online when instructors are not available.

Background:

Researchers have studied the effects of making synchronous chat available in online classes such as Spencer & Hiltz (2003) who showed that students found classes with synchronous chat significantly more ‘rewarding’ and less ‘complex’ than classes with only asynchronous communication. This is not to say that asynchronous interaction such as that facilitated by blogs, emails, bulletin boards, wikis and so forth it not valuable.  In fact according to Johnson (2006) both synchronous and asynchronous forms of online discussion have advantages and there is evidence that both contribute to student cognitive and affective outcomes.

While the benefits of rich interactive tools for online learning are not in question, there have been relatively few assessments (although see Tomei, 2006) of the time taken by the online instructor to manage these systems.  The principle investigator of this project argues that much instructor time could be saved by the automated generation of class frequently asked question (FAQ) pages, as well as by providing multiple means to access this information, possibly including chatbot technology to interact with students over instant messenger (IM) systems such as Skype.

There have been a number of attempts to integrate chatbot technology into the online classroom over the years.  For example Heller et al. (2005) demonstrated that famous person chatbots (e.g. FreudBot) are promising as a teaching and learning tool in online education; while Mikic et al. (2009) developed the CHARLIE chatbot which can maintain a general conversation with students, showing them contents of courses, and asking them questions about learning material.  Even without a working chatbot there are studies (Kerly et al., 2006) using Wizard-of-Oz experiments that indicate this technology, if successful, could have widespread application in schools, universities and other training scenarios.

There are some positive results at the K-12 level (Kerly et al., 2007) however other authors (e.g. Knill et al., 2004) have cautioned that:

The teacher is the backbone in the teaching process. Technology like computer algebra systems, multimedia presentations or ‘chatbots’ can serve as amplifiers but not replace a good guide.

Despite these concerns recent researchers have experimented with humor enabled chatbots (Augello, 2008) and chatbots that support second language learning (De Gasparis & Florio, 2012; Jiyou, 2009; Sha, 2009); while yet others have demonstrated chatbot effectiveness in supporting network management training (Leonhardt et al., 2007) and security training (Kowalski et al., 2009). Furthermore in a detailed study Alencar & Netto (2011) developed an online education portal chatbot that successfully answered 69% of the students’ general questions about distance education.

In summary it seems that there is a great potential for the use of chatbot technology in the online classroom, but that one should be careful not to try and replace the human instructor entirely.  The proposed project is focused on providing automated online teaching assistants that will help the human online instructor manage their online classes. The principle investigators experience indicates that synchronous chat helps with student engagement and retention although as Wilging & Johnson (2004) point out students’ reasons for dropping out of an online program are varied and unique to each individual.  It is the principle investigators hope that well designed automated FAQs and IM chatbots can server to amplify the retention-enhancing effect of synchronous chat in online classes.

Clearly a cautious approach is required to develop systems that can successfully support online learning success.  This project plans to analyze the logs from several online classes using the ethnographic approach demonstrated by authors such Zembylas & Vrasidas (2007).

Work Plan:

Jan/Feb 2013: Analysis of CSCI 4702, 4705 and MULT 4702 chat logs
Feb/Mar 2013: Generation of 1^st version of FAQs
Mar/Apr 2013: Automation of FAQ generation
Apr/May 2013: Link existing chatbot to FAQs to support usability testing over summer 2013

Work and Funding so far:

This project has not previously received any funding, but the principal investigator conducted an exploratory analysis of the chat logs during summer 2012.  A basic chatbot that answers simple questions has been created and deployed and further analysis and development is required to make something that can be of use in an online course.

No funding has been received or applied for from other sources.  The project is ongoing and will continue as long as the principal investigator continues to teach online classes.  However in the first instance a single course release is being requested to help work through the large volume of text chat that needs analyzing

Budget Justification:

The budget is for a single course release to provide the principal investigator with time to conduct the chat-log analysis and do code development based on the results.

Access:

No specific facilities are required.  The only requirement is IRB approval which is pending from the HPU CHS.

Dissemination:

Dissemination includes presenting the work at professional conferences such as the World Congress on Education or the International Conference on Advanced Learning Technologies, and ultimately in journals such as Knowledge-Based Systems and Distance Education.  The work would also be disseminated as an open source package and be promoted through the “Funniest Computer Ever” annual computer comedy contest. 

In the long term it would be good to see many educators making use of the techniques and tools developed from this project, and the plan is to disseminate the work as widely as possible through both academic, i.e. peer-reviewed publications and technical circles, i.e. an open source toolkit for online educators.

Deliverables:

Two conference paper submissions and working FAQ and chatbot prototypes by end of Summer 2013, for deployment in Fall 2013 classes.

Bibliography

• Alencar, M. & Netto, J. (2011). Developing a 3D Conversation Agent Talking About Online Courses. In T. Bastiaens & M. Ebner (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2011 (pp. 1713-1719). Chesapeake, VA: AACE
• Augello, A. (2008)Humorist Bot: Bringing Computational Humour in a Chat-Bot System. International Conference on Complex, Intelligent and Software Intensive Systems, 2008. CISIS 2008.
• De Gasperis G.  & Florio N. (2012) Learning to Read/Type a Second Language in a Chatbot Enhanced Environment. International Workshop on Evidence Based Technology Enhanced Learning. Advances in Intelligent and Soft Computing, 2012, Volume 152/2012, 47-56
• Heller, B., Proctor, M., Mah, D., Jewell, L. & Cheung, B. (2005). Freudbot: An Investigation of Chatbot Technology in Distance Education. In P. Kommers & G. Richards (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2005 (pp. 3913-3918). Chesapeake, VA: AACE.
• Jiyou J. (2009) CSIEC: A computer assisted English learning chatbot based on textual knowledge and reasoning. Knowledge-Based Systems. Volume 22, Issue 4, May 2009, Pages 249–255
• Kerly, A., Ellis, R. & Bull, S. (2007). CALMsystem: A Conversational Agent for Learner Modelling, in R. Ellis, T. Allen & M. Petridis (eds), Applications and Innovations in Intelligent Systems XV – Proceedings of AI-2007, 27^th SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence, Springer Verlag 89-102.
• Knill, O., Carlsson, J., Chi, A., and Lezama, M. (2004). An artificial intelligence experiment in college math education. Preprint available at http://www.math.harvard.edu/∼knill/preprints/sofia.pdf.
• Kowalski S., Pavlovska K. & Goldstein M. (2009) Two Case Studies for Using Chatbots for Security Training. World Conference on Information Security Education 2009.
• Leonhardt, M.D. Tarouco, L., Vicari, R.M.,Santos, E.R. & dos Santos da Silva, M. (2007) Using Chatbots for Network Management Training through Problem-based Oriented Education. Seventh IEEE International Conference on Advanced Learning Technologies, 2007. ICALT 2007.
• Shaa G. (2009) AI-based chatterbots and spoken English teaching: a critical analysis. Computer Assisted Language Learning, Volume 22, Issue 3.
• Spencer, D. H., & Hiltz S. R.  (2003) A field study of use of synchronous chat in online courses. Paper presented at the 36th Hawaii International Conference on System Sciences, Big Island, HI, January. http://csdl2.computer.org/comp/proceedings/hicss/2003/1874/01/187410036.pdf
• Tomei, D.L. (2006). The Impact of Online Teaching on Faculty Load: Computing the Ideal Class Size for Online Courses. Journal of Technology and Teacher Education, 14(3), 531-541.
• Zembylas, M & Vrasidas C. (2007) Listening for Silence in Text-Based, Online Encounters. Distance Education; May 2007; 28, 1.