A Preventive Tutoring System for Beginning Programming

INTRODUCTION:
My dissertation work is a direct result of my teaching experiences with beginning programmers.  Like many introductory computer science instructors, I found myself consistently surprised by the intense struggles of some students, even with problems that seemed to me at the time (and to many students) to be "easy."  After seeing many poorly conceived and poorly implemented attempts at solving these problems, it became clear that some novices needed a different kind of help.

So what sort of help is needed?  Obviously, there are many answers to this question.  Guzdial [8] points out that each answer to the question "What makes programming hard?" implies a family of environments providing assistance based on that answer.  For me, the answer was that novices lacked the ability to plan (or perhaps simply chose not to).  My conviction was (and still is) that novices need help earlier in the process programming to prevent misconceptions that could later balloon into major difficulties.

With this goal of early intervention, I built a dialogue-based tutoring system named ProPl ("pro-PELL", from "Program Planner") that uses natural language interaction to help students understand and plan their programs before they attempt to write them.  The aim is to model and support the cognitive problem solving activities that novice programmers are known to generally underestimate or even bypass altogether.  Specific tutoring strategies are used when answers are partially correct or incorrect, all with the aim of eliciting the ideal answers from the student.  The purpose is both to confirm understanding of the problem as well as to help the student generate a plan for use during a traditional, independent implementation phase. 

The ProPl interface consists of a simple browser, a dialogue window, and a window displaying notes and pseudocode (screenshot). Dialogue is driven by a reactive planner and understanding accomplished via an augmented semantic grammar (details in [10]).  During a tutoring session, students are presented a problem (in the browser), and asked to identify programming goals and speculate on possible ways to achieve those goals (in a dialogue).  Sub-dialogues are sometimes launched that present examples, point the student back to the problem statement, or try to elicit answers through deeper lines of questioning.  As the dialogue proceeds, goals and important observations are posted (screenshot).  Concomitantly, pseudocode for each goal is also maintained (screenshot).  Both of these windows are maintained entirely by ProPl.

BACKGROUND & MOTIVATION:
Novices struggle with just about every aspect of programming and adopt behaviors that frequently worsen the situation for them [5].  Pennington [11] found that a large number of mistakes stemmed from flawed understandings of goal and subgoal structure. Similarly, Soloway and Spohrer discovered that many of the bugs present in novice programs reflect misconceptions about the underlying algorithm rather than the language being used [14].  It is also known that novices tend to begin the programming process by keying in code [12], thinking of programming in terms of small, syntactic units. This is not peculiar to programming:  novices in other domains (like physics) display similar behaviors [3].  In general, it seems that novices enter into a programming task without a global picture of their program. 

Given these tendencies, it is unlikely that novices will ask, or even know to ask, many important program design questions until implementation time. Thus, they are forced into "on-the-fly" planning.  It is clear that if any of these detrimental novice behaviors can be averted, intervention must take place early.  ProPl attempts to do this by engaging the student in discussions about programming goals (the "what?") and strategies to achieve them (the "how").  The knowledge underlying how to achieve goals may best be viewed as Soloway-style programming plans [15].

RELATED RESEARCH:
Dialogue-based tutoring systems and are now emerging and being shown to be pedagogically effective in many domains: conceptual physics, electricity, computer fundamentals, and physiology [6]. The technology to understand and classify natural language input from students has become powerful enough to drive tutorial and dialogue management modules of systems [1,6].

A large number of tutoring systems for programming exist, most of which tutor specific languages and do not make obvious distinctions between the planning and implementation phases of programming.  A portion of these do support the phases of problem understanding and program design, however.  For example, Bridge [2] walks the student through a series of translations from English-like phrases down to Pascal code.  Two other more recent systems include DISCOVER [13] and SolveIt [4], both of which draw clear lines between planning and implementation.  Many systems, such as GPC-Editor [7], do not employ techniques of Artificial Intelligence, but rather provide an environment and tools that make it easier for novices to engage in planning activities.

RESEARCH GOALS:
My hope with ProPl is that novices will adopt a more global and goal-oriented perspective of the task of programming.  I have built a system that promotes a particular cognitive model of planning simple programs - it engages students and asks them to identify programming goals and how to achieve them.  As such, my primary research goal is to determine the efficacy of this pedagogical approach to improve programming skills.  Indirectly, I also hope to evaluate the ability of a machine to elicit a kind of perspective and behavioral change that is suggestive of a more mature programming approach.

My evaluation plan calls for three conditions:  a reading group (control), a human tutored group, and a ProPl group, along with both quantitative and qualititave analysis of the resulting data.  The reading group is simply presented with a goal analysis and pseudocode for a program.  The human group performs that same kinds of tutoring as ProPl, but in a face-to-face, unconstrained context with a human tutor.  Lastly, the ProPl group uses the tutoring system to prepare for programming assignments. 

The study involves three projects from the introductory programming course along with an untutored post-test project.  In addition, all subjects will come in for a timed, untutored, and closed-lab post-test project that includes a follow-up interview.  During this project, a modified compiler is used that asks the student to explain the purpose behind each compile attempt.  The interview involves questions about the problem and how it was solved with the purpose being to discover how well the subject handled the difficulties they faced.  The goal is determine how subjects viewed their task and how they overcame any impasses.  In addition to this, online protocols and compiler usage data will be collected (included code submitted to the compiler) and analyzed in attempt to better understand the tendencies and strategies used during the implementation.

CURRENT STATUS:
Pilot testing of ProPl is currently underway, which mainly being done to debug and refine the knowledge sources.  The full evaluation is planned for the spring with subjects coming from three different introductory programming courses at the University of Pittsbugh.  ProPl runs as a Java applet and connects to a cenral Lisp server (that controls tutoring). 

INTERIM CONCLUSIONS:
In a controlled study of human-to-human tutoring of pseudocode design, it was found that tutored students, compared to those who were not tutored, were more prolific with comments in their programs, committed fewer structural mistakes, and exhibited less erratic programming behavior during their implementations [9].  This is suggestive that student programming behaviors can be improved by engaging them early.  The next step is to discover if a computer tutor can replicate these findings.

Students in the pilot study have responded well to the various tutoring strategies used by the system, and generally seem patient when the system does not understand them.  Without the presence of an experimenter, however, this is unlikely to hold true, and thus the engineering effort has been aimed at adding robustness and breadth to the knowledge sources.

OPEN ISSUES:
As I write this, I am in the process of refining my post-test measures to best reveal how subjects went about their implementation and to connect it to the tutoring provided in ProPl.  Some measures are straightforward (such as compile counts, task completion, satisfaction), while others are harder to quantify or will (hopefully) fall out of the qualitative analysis I have planned.  As of right now, the basic questions of how best to do this qualitative data analysis (and how to defend it to people who do not understand it) are very pressing.  A member of my committee asked "If you don't see any learning gains, do you think you should get your Ph.D.?"  This prompted the inclusion of the qualitative component of my evaluation plan. 

WHAT I HOPE TO GAIN:
At the time of the DC, I will be in the middle of my evaluation.  It will not be too late to adjust my post-test measures, and thus I hope to get some insights on that as well as advice on pulling everything together for my defense in the summer.  This will be my third and final trip to the DC.

[1]  Aleven, V.A., Popescu, O., and Koedinger, K.R. (2001). A Tutorial Dialogue System with Knowledge-Based Understanding and Classification of Student Explanations.  Working Notes of the 2nd IJCAI Workshop on Knowledge and Reasoning In Practical Dialogue Systems (August 2001). 

[2]  Bonar, J.G. and Cunningham, R.  Bridge:  Tutoring the Programming Process.  In Intelligent Tutoring Systems:  Lessons Learned, J. Psotka, L. D. Massey, & S. A. Mutter Eds. Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1988, pp. 409-434.

[3]  Chi, M., Feltovich, P. J., and Glaser, R.  Categorization and Representation of Physics Problems by Experts and Novices.  Cognitive Science 5 (1981), 121-152.

[4]  Deek, F. P.  A Framework for an Automated Problem Solving and Program Development Environment.  Journal of Integrated Design and Process Science 3, 3 (1999), 1-13.

[5]  DuBoulay, B.  Some Difficulties of Learning to Program.  Journal of Educational Computing Research 2, 1 (1986), 57-73.

[6]  Graesser, A., VanLehn, K., Rosé, C., Jordan, P., and Harter, D. Intelligent Tutoring Systems with Conversational Dialogue, AI Magazine (2001).

[7]  Guzdial, M., Hohmann, L., Konneman, M., Walton, C., and Soloway, E.  Supporting Programming and Learning-to-Program with an Integrated CAD and Scaffolding Workbench.  Interactive Learning Environments  6, 1&2 (1998), 143-179.

[8]  Guzdial, M.  Programming environments for novices. In Handbook of Computer Science Education Research, S. Fincher and M. Petre, Eds., Springer Verlag. (to appear)

[9]  Lane, H. C. and VanLehn, K.  Coached Program Planning:  Dialogue-Based Support for Novice Program Design.  In Proceedings of the Thirty-Fourth Technical Symposium on Computer Science Education (SIGCSE), Reno, NV, 2003, pp. 148-152.

[10]  Lane, H. C. and VanLehn, K.  A Dialogue-based Intelligent Tutoring System for Beginning Programming.  To appear in Proceedings of the 17th International Florida Artificial Intelligence Research Symposium (FLAIRS 2004), Miami Beach, FL, May 2004.

[11]  Pennington, N.  Comprehension Strategies in Programming.  In Empirical Studies of Programmers: Second Workshop, G. M. Olson, S. Sheppard, and E. Soloway, Eds.  Ablex Corp., Norwood, NJ, 1987.  pp. 100-113.

[12]  Pintrich, P. R., Berger, C. F., and Stemmer, P. M.  Students' Programming Behavior in a Pascal Course.  Journal of Research in Science Teaching 24, 5 (1987), 451-466.

[13]  Ramadhan, H. A., Deek F., and Shihab, K.  Incorporating Software Visualization in the Design of Intelligent Diagnosis Systems for User Programming.  Artificial Intelligence Review 16 (2001), 61-84.

[14]  Spohrer, J. C. and Soloway, E.  Putting It All Together is Hard For Novice Programmers.  In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (Tucson, Arizona, November 12-15 1985).

[15]  Soloway, E.  Learning to program = learning to construct mechanisms and explanations. Communications of the ACM, 29, 9, September 1989, pp.850-858.