Strategy-based feedback for imperative programming exercises

syntax. An abstract syntax has been designed to represent imperative programs, supporting the language constructs of Table 8. This abstract syntax is implemented using data types in Strategy-based feedback for imperative programming exercises 34 Haskell, in which a Program is the top-level type that consists of zero or more statements. The Statement data type represents the various program statements: data Statement = Block [Statement] | If Expression Statement | IfElse Expression Statement Statement | While Expression Statement | For ForInit [Expression] [Expression] Statement | Print Expression | VarDeclarations DataType [Expression] | ExprStat Expression | Empty | Break | Continue Many statements are composed of one or more expressions, represented by the Expression data type: data Expression = Infixed InfixOp Expression Expression | Assignment AssignOp Expression Expression | Prefixed UnaryOp Expression | Postfixed UnaryOp Expression | LiteralExpr Literal | IdExpr Identifier | Call Identifier [Expression] | Property Identifier Identifier | NewArray DataType Expression | ArrayAcc Identifier Expression The details of the other data types that are used, such as Identifier, DataType and Literal, are omitted. The abstract syntax is intentionally not very strict and specific, so various different languages can be represented by this data structure. An advantage of this general internal structure is that we can create programming exercises with solutions in one specific language that can also be solved using a different programming language. Parser. Program code from a specific imperative language should be transformed into this internal representation using a parser. Language constructs that are very specific for a certain language are converted into a more general structure in the parsing process. For instance, some languages only allow one condition in a for statement, whereas other languages allow multiple conditions separated by a comma. The presence of a compiler or interpreter is always assumed so the parser does not have to perform semantic checks, such as type checking and object binding. If a program cannot be parsed, the tutor is not able to deal with it and the student should repair the code first based on compiler messages. Currently there are parsers for two different programming languages: Java and PHP. Java is a wellknown and widely used object oriented programming language that is often taught in schools and universities. PHP is a server-side scripting language that is usually embedded into HTML. The parsers have been implemented using the Parsec library 11 that provides a large number of parser combinators 11 http://hackage.haskell.org/package/parsec, retrieved November 13, 2013 Strategy-based feedback for imperative programming exercises 35 to simplify the parsing process. The parsers are written in the applicative style. A lexer is automatically created using a language definition for Java or PHP defining their keywords and special characters. Using this lexer we can easily define parsers for the language constructs, such as the while statement in the following example. The functions exprP and statP are the parsers for an expression and a statement respectively: whileP :: Parser Statement whileP = While <$ reserved "while" <*> parens exprP <*> statP Pretty printer. In the output of the tutor we want to show programs in their original syntax. To enable this, a pretty printer has been implemented that converts abstract syntax into a textual representation that corresponds to the syntax of the programming language that is used. The printer is implemented using the PPrint library 12 that is based on the pretty printing combinators described by Philip Wadler (Wadler, 1998). The pretty printer is implemented for both the Java and PHP language. The next example shows the while statement converted into the Doc data type that represents a pretty document that can be shown as text. instance Pretty Statement where pretty (While e s) = text "while" <+> parens (pretty e) <$> nested s Support for incomplete programs. Students who have not finished their program yet should be able to receive feedback on their partial solution. Statements can be omitted in imperative programming, which should not create problems with parsing the program. To further support students in creating a program step by step the question mark (‘?’) character can be used inside a statement to represent an expression that is yet to be completed. A few examples are: int x = ?; sum = ? + ?; for (?; ?; ?); while (x < ?); The expression data type has been extended with a ‘hole’ constructor. An integer is used to uniquely identify a specific hole. data Expression = ... | HoleExpr LocationID The addition of a new symbol in the programing language will of course cause problems because the compiler is unfamiliar with this symbol. This implies that students cannot rely on compiler messages when using holes in their programs. Instead they are referred to the tutor that is able to recognise the holes and help the student complete the statement before continuing with the remaining program. Testing. The parsers have been tested by parsing a large number of source files that include the supported language constructs in various forms. We also use QuickCheck (Claessen & Hughes, 2000), a library for testing that automatically generates test cases attempting to falsify properties. The combination of a parser and pretty printer together should satisfy the following (simplified) QuickCheck property, stating that the pretty printed representation of a program should be parsed into a program that is equal to the original: 12 http://hackage.haskell.org/package/wl-pprint-1.1, retrieved November 13, 2013 Strategy-based feedback for imperative programming exercises 36 prop_parsePrettyPrintedProgram :: Program -> Bool prop_parsePrettyPrintedProgram program = program == (parse . pretty) program To generate random programs we provide instances of the Arbitrary class for statements, expressions and the other data types that are used. We also provide a separate generator for programs that do not contain any holes, such as model solutions. An example fragment that generates a statement by choosing one from a list of statements is shown next. These statements consist in their turn of other arbitrary components. To prevent the generation of structures that are nested too deeply, we use a ‘sized’ generator that recurses towards statements that do not include other statements. If the size integer n reaches zero, these nested statements are not included in the choice. instance Arbitrary Statement where arbitrary = sized $ sizedStatGen True sizedStatGen :: Bool -> Int -> Gen Statement sizedStatGen holes n = oneof $ notNested ++ if n > 0 then nested else [] where notNested = [ Print <$> arbEx, VarDeclarations IntType <$> sizedVector assignExprGen (1,3), return Break, ... ] nested = [ If <$> arbEx <*> smallerStat, IfElse <$> arbEx <*> sizedBlock <*> smallerStat, ... ] smallerStat = oneof [sizedStat, sizedBlock] counter = makeIdt "i" arbEx = exprGen holes sizedStat = sizedStatGen holes $ n `div` 10 sizedVector = (>=>) choose . flip vectorOf sizedBlock = makeBlock <$> sizedVector sizedStat (1, 5) To generate programs that do not deviate too much from real world programs, we have to further control the randomness. In the next example we show how a for statement is generated. To avoid the accidental creation of infinite loops, a counter variable is used that is initialised at a number between zero and ten, increments with one each iteration and ends at 99. For <$> ForInitExpr . (:[]) . Assignment Assign counter . makeInt <$> choose (0,10) <*> pure [Infixed Less counter $ makeInt 99] <*> pure [Postfixed Incr counter] <*> smallerBlock We control the generation of expressions by providing a distribution and using custom-made generators such as assignExprGen (for variable assignments) and arithExprGen (for basic arithmetic expressions) that generate common expressions. The frequency function randomly chooses one of the generators based on the distribution. Strategy-based feedback for imperative programming exercises 37 exprGen :: Bool -> Gen Expression exprGen holes = frequency $ [ (40, makeInt <$> choose (0, 999)), (40, IdExpr <$> arbitrary), (30, arithExprGen), (20, assignExprGen), (5, Call <$> arbitrary <*> vectorOf 2 (exprGen holes)), (5, ArrayAcc <$> arbitrary <*> exprGen holes) ] ++ [ (10, HoleExpr <$> arbitrary) | holes ] An example of a randomly generated program is shown below. Note that there are some language constructs that will not be accepted by a Java compiler. We do not check for correct declaration and initialisation of variables. There are many possible improvements; however, for testing purposes these programs are adequate. continue; int z = y != z; while (true) continue; for (i = 8; i < 99; i++) print (211); for (i = 2; i < 99; i++) { if (658) { continue; } if (161) y(599, 921 < 225); break; } print (x); 5.2 STRATEGIES FOR IMPERATIVE PROGRAMMING To use the Ideas framework for calculating feedback, we need to specify a strategy for each exercise. In an educational setting, the instructor serves as a guide to show students how to program. When an instructor is not present, we would like to stay close to what an instructor would have said when a student asks for help. Therefore model solutions from an instructor are used as a basis to provide feedback. This approach is also used in a number of other recent programming tutors (Dadic et al., 2008; Gerdes, Jeuring, et al., 2012; Hong, 2004; Singh et al., 2013) and provides a number of advantages:  An instructor can easily add new exercises.  Models programs can be annotated, providing extra opportunities for didactic guidance. Annotating model solutions is elaborated in Section 5.4.3. Potential difficulties that should be looked into are the large solution space, which is discussed in the following sections, and the lack of clarity on what exactly distinguishes one solution from the other. It is the instructor’s responsibility to provide model solutions that represent the solution space of an exercise. Every model solution should preferably represent a different algorithm that solves the problem. But what separates one algorithm from the other? This issue is addressed in Section 5.3.1. Strategy-based feedback for imperative programming exercises 38 The strategy to work towards a particular model solution should reflect how imperative programs are implemented. Imperative programs can be constructed in several ways:  Quickly constructing a coarse solution and then refactoring it until there are no errors left. This approach might reflect the trial and error style students often adopt.  Programming by contract: defining preand post-conditions prior to the actual implementation (Dijkstra, 1975).  The stepwise decomposition of a program using refinement steps (Wirth, 1971).  Building up a program line by line, manipulating the program state in the meantime. In recent tutors that support incomplete programs, we recognise the third option for an imperative language (Holland et al., 2009) and logic programming (Hong, 2004). In the Ask-Elle tutor for functional programming (Gerdes, Jeuring, et al., 2012) refinement steps are used to gradually make a program more complete by replacing unknown parts (holes) by actual code. The last option is used in two data-driven tutors for imperative programming (Jin et al., 2012; Rivers & Koedinger, 2013), although the steps are generally larger than just one line. An advantage of the last option is that the compiler can provide help in most situations, for example variables are always declared before they are used. We have selected this style for the tutor, because of this advantage together with the ability to help the student from start to finish and not only after creating a first solution entirely on their own. We also incorporate refinement for composed language constructs. To create a strategy using the Ideas framework we need two components that are elaborated in the next sections:  Rules that represent the steps a student can take to gradually build up a solution.  A strategy generator that generates a strategy from model solutions using these rules. 5.2.1 RULES A strategy to solve an exercise is made up of a number of steps, or ‘rules’. The Ideas framework enables the creation of rules based on a transformation function. Two types of rules are used in the tutor for imperative programming: append rules and refinement rules, which will be explained in more detail in this section. Append rules. An append rule appends a statement to the end of a block, which corresponds to updating the program state line by line. An example of three consecutive applications of an append rule is: x = 5;  x = 5; y = 7;  x = 5; y = 7; avg = (x+y)/2;  x = 5; y = 7; avg = (x+y)/2; print(avg); Suppose we have a program with multiple nested statements, such as: if (x > 10) { for (i = 0; i < x; i++) { print(i); } } Strategy-based feedback for imperative programming exercises 39 It is unclear where a new statement should be appended. There are three options: after the print statement inside the for statement, after the for statement inside the if statement or after the if statement. To identify the specific location of an append rule, we extend the construction of a Block with an integer that uniquely identifies this block. To enable adding statements to the highest level of the program, every program will be parsed into a program with a block at top level. data Statement = ... | Block LocationID [Statement] An append rule can be created using the appendStat function as shown below. The pref integer is used for rule ordering, which is explained at the end of this section. The rule can only be applied if the block with the specified identifier is present in the program exactly once. We use the transformBi function from the Uniplate library (Mitchell & Runciman, 2007) for generic traversals. The library provides functions to easily traverse and manipulate complex data structures to avoid writing a lot of repetitive, ‘boilerplate’, code. The Biplate variant (transformBi) is used because the Program data type combines multiple other data types. The transformation append' is applied to all statements in a program, including nested statements that can be found inside statements such as while and if. If the unique identifier of a block equals the location parameter, the new statement is appended to the block. appendStat :: Statement -> LocationID -> Int -> Rule Program appendStat newStat loc pref = makeRule ruleId $ append where append p | nrOfBlocksById loc p == 1 = Just $ transformBi append' p | otherwise = Nothing append' (Block i stats) = Block i $ stats ++ [newStat | i == loc] append' stat = stat ruleId = ... Refinement rules. A refinement rule replaces a hole by an expression. An example of applying a sequence of refinement rules is: avg = ?;  avg = ? / ?;  avg = sum / ?;  avg = sum / 2; The code for creating a refinement rule is similar to the code for the append rule, apart from the implementation of the transformation function refine' that replaces the hole with the new expression: refineExpr :: Expression -> LocationID -> Int -> Rule Program refineExpr newExpr loc pref = describe name . makeRule ruleId $ refine where refine p | nrOfHolesById loc p == 1 = Just $ transformBi refine' p | otherwise = Nothing refine' e@(HoleExpr i) | i == loc = newExpr | otherwise = e refine' e = e ruleId = ... name = ... Strategy-based feedback for imperative programming exercises 40 Other rules. We have defined a rule for inserting a statement at any location in a program; however this rule has not yet been used. Defining rewrite rules is also an option for future research, as described in Section 7.4. Rule ordering. Rule ordering is used to give preference to certain model solutions and language constructs. We define a rule ordering based on an integer that is the suffix of the rule identifier, for example rule ‘if-else-at-1.6’ will be ordered using the suffix ‘6’. We define preference during the strategy generation process, which is described in the next section. 5.2.2 GENERATING STRATEGIES Using the rules described in the previous section, we can now specify strategies for the stepwise development of a program. We have created a strategy generator that accepts a set of model programs as input and produces a strategy as output. A number of normalisations are performed on the program before the strategy generation. During the generation process we maintain a state that stores a counter and the feedback level (the usage of this level is elaborated in Section 5.4.3). The counter is used to uniquely number the blocks and holes during the generation of the strategy. type GenState a = State (LocationID, Int) a We define a GenStrategy class with a function that takes any value (and some additional parameters) and returns the current state and a corresponding strategy for a program. We provide implementations for the main data types Program, Statement and Expression. type StrategyGenerator a = LocationID -> Int -> a -> GenState (Strategy Program) class GenStrategy a where genStrat :: StrategyGenerator a We also define a class of types for which we can generate a strategy together with a specific location, which could either be a hole (for expressions) or a block (for statements). class GenStrategy a => GenStrategyWithLoc a where genStratWithLoc :: Int -> a -> GenState (a, Strategy Program) genStratsWithLocs :: Int -> [a] -> GenState ([a], [Strategy Program]) genStratsWithLocs = mapAndUnzipM . genStratWithLoc The overloaded function genStratWithLoc is defined for both statements and expressions and returns a tuple with either a new hole or a new block with a unique number together with the strategy of the input. The getNextNr function returns the next available number and updates the counter in the state. We only show the code for the expression variant. genStratWithLoc pref expr = do loc condition' <*> body' First, a hole is created for the condition in the if statement together with a corresponding strategy. Next, a block and the strategy for the body are generated, followed by an append rule for an empty if using the helper function appRule. The resulting strategy consists of a sequence of creating an empty if, building up the condition and finally creating the body using the sequence (<*>) combinator from the Ideas framework, as illustrated in the following example: if (?) {}  if (isOk) {}  if (isOk) { call(); } Infix expression strategy. The next fragment shows how a strategy is created for an expression, such as ‘(a + b) < 2’. The result is the introduction of an infix expression with holes on both sides of the operator, followed by the interleaving (<%>) of the sub strategies for the left and right operands of the expression. Refining the left hole first has a higher preference. genStrat loc pref (Infixed op e1 e2) = do (hole1, e1') (e1' <%> e2') This implies that we can arrive at an expression consisting of several sub expressions in multiple ways: