# Automatically check student answers in tests

I am a physics teacher and I am looking for a way to create homework assignments within Mathematica so that each problem has an input field that students can use to check their answers.

For example, if the problem reads "What is the integral of 2x^2" I want an input field immediately after the question that students can type their answer into. If they type in the correct answer, it will confirm that it is right where as if it is not right it will tell them it is incorrect.

Anyone have any ideas of how I could do this?

-
For having e-exams and e-homework this tool might be handy exelearning.org –  Ajasja May 18 '12 at 15:39
There is also the AcroTeX eDucation Bundle which I have used a few years ago, and just checked that it still is freely available. When I used it, I was impressed how it can take free-form input of functions and checks them for correctness - purely with JavaScript in a PDF document displayed with Adobe Reader. The trick they used to check functions is not to try and symbolically simplify them, but to numerically compare them to the correct solution at certain x-values. –  Jens May 18 '12 at 15:54
Hello, welcome to Mathematica! Please register your account so that you can continue to use this site from any computer and carry forward your points. It'll also enable you to participate in the site by voting good answers and questions :) –  rm -rf May 18 '12 at 17:27
I have a prototype for this. It uses a combination of mathematica and a web server. I'll try to find the time to post my detailed setup. –  Gustavo Delfino Jan 10 at 13:40

EDIT--- The code was updated to include anti-cheating filter to address important issues raised by @Jens in comments (thanks). More filters can be added to exclude other type of cheating. ---EDIT

I was teaching physics and math for many years and consider this to be a very important question. I would say Mathematica is very well equipped for this type of course-ware development. I cannot think of any other programming environment that can do this type of things better. Let me address here a couple of questions.

1) Symbolics, numerics & Course-ware. @R.M. numerically-based answer is very nice and is using approach that other systems (like WebWork Homework) use. This will work in many cases. Additionally Mathematica has symbolic engine, patterns, string patterns and regular expression processing which opens immensely wider field for automated assignment creation and testing. This is what I wanted to demonstrate.

2) CDF & Course-ware. This is all good for exploratory student work. What about strict testing? (see @Jens comment) Can't students just use Mathematica to compute the answer? CDF player can provide only app interface without any code around to read the hints form or ability to compute the answer outside the interface. Input filed is limited in free CDF, but there is Player Pro and (bvecause CDF platform was just launched) we may expect in future even more flexibility and functionality.

This app is just a start - a simple working prototype.

Manipulate[Grid[{{"What is the value of this integral?"},
{HoldForm[\[Integral]2 z^2 \[DifferentialD]z]}, {Framed@
If[FullSimplify[
If[StringCases[x, "Integrate"] === {},
ToExpression[x, StandardForm] // Quiet,
WindowSize -> {400, 60}];; Null,
WindowSize -> {400, 60}];; Null] == 2 z^3/3], "CORRECT",
"WRONG", "WRONG"]}]}}], {{x, "", ""},
InputField[#, String, FieldHint -> "enter answer here..."] &},


-
This cannot work because the student could simply enter the following input: Integrate[2 z^2,z] and get full credit. –  Jens May 18 '12 at 16:18
@Jens Well if the students have Mathematica then they can enter this anyway (and copy-paste the result into any e-exam). One could look for Integrate inside the entered expression I guess. –  Ajasja May 18 '12 at 16:39
@Andrew The question asks for something where the students enter their answers and get feedback. I'm simply pointing out that the main issue of the question isn't addressed in this answer - so I personally would not have posted that. –  Jens May 18 '12 at 17:16
What about the problem that FullSimplify is not guaranteed to find the same form for the solution, although it may be mathematically equivalent to the teacher's solution? –  Sjoerd C. de Vries May 18 '12 at 18:40
@Gabel CDF is a very new platform and we hope it's functionality will be expanded very soon. As a glimpse into future (not yet here ;-) ) I want to show you this: kaurov.com/wordpress/?page_id=845 For current practical solutions please contact Wolfram Research - it would be very good for them to have your feedback wolfram.com/cdf/contact-us –  Vitaliy Kaurov May 18 '12 at 22:41

As I mentioned in my comment to the question, I am quite fascinated by the idea of testing free-form input by numerical means similar to what the AcroTeX bundle has been able to do for a long time. This idea differs quite fundamentally from Vitaiy's suggestion, so I have tried to make it work in Mathematica.

I will suspend discussion of the crucial obstacle that the free CDF format, though being the obvious choice to deploy such tests, does not currently allow InputFields for strings. For the moment, I have to trust in the indirect promise made by Vitaliy that this will change in the future (I realize he doesn't speak for WRI).

Now it turns out that Mathematica has a function that is almost ideal for my suggested method: PossibleZeroQ. Implementing this as a test validator is the main idea of this answer.

The second idea is that with a tool as powerful as Mathematica, making a meaningful testing environment (even for harmless practice tests) requires us to shut off access to symbolic manipulations that can be used to cheat. This problem doesn't arise if we make tests where you only have to enter numbers, as in R.M.'s answer. The question goes much further than that, in that it wants us to validate student input in the form of functions (as in integrals and derivatives).

The main difference between my approach and Vitaliy's in this respect is that I implement a "white-list" instead of a "black-list" approach. That means I want allow only a certain class of functions to be entered. In particular, I will permit only those functions that have NumericFunction in their Attribute list.

The code below tries to implement these main ideas, putting them in the form of a basic dynamic grid where the questions appear next to answer fields. This requires some juggling with Hold, HoldForm, string conversion and replacements inside Dynamic. Strings appear because for security reasons I accept the answer as String input before converting it ton a held expression.

Here is the set of definitions:

Clear[check, quantQuestions, quantAnswers, prepareQuestions];

quantQuestions[
expressions__] := {ReleaseHold@Map[HoldForm, Hold[expressions]]};
SetAttributes[quantQuestions, HoldAll];

Map[ReleaseHold[(# /.
Integrate[f_, x_?AtomQ] :> integralWrapper[f, x]) /.
Integrate[f_, {x__}] :>
Integrate[f, {x}, GenerateConditions -> False]] &, questions];

check[ans_, reference_] := Module[{ans1, ref1},
If[
ans1 = D[ans, Last[reference]];
ref1 = First[reference],
ans1 = ans;
ref1 = reference
];
If[PossibleZeroQ[ans1 - ref1], 1, 2]]
check[ans_String, reference_] := 3;

stringCheck[
a_] := (Apply[And,
Map[MemberQ[Attributes[#], NumericFunction] &,
Cases[# /. Hold -> Unevaluated, x_[__] -> x, Infinity]]] /.
True :> ReleaseHold[#] /. False -> "no answer") &[
If[SyntaxQ[a], ToExpression[a, InputForm, Hold]]] /. Null -> ""

prepareQuestions[q_] :=
DynamicModule[{feedback = "Correct answers: ", maxAttempts = 3, ref,
ans, evaluation, attempts, active},
Print["Some questions cannot be answered"]; Abort[]];
ans = Map["" &, ref];
evaluation = Map[4 &, ref];
attempts = Map[0 &, ref];
active = Map[True &, ref];
Deploy@Labeled[
Framed[Grid[
Append[
Prepend[
Array[{
InputField[Dynamic[ans[[#]], Function[{a},
If[ans[[#]] != a, ans[[#]] = a;
If[! (active[[#]] = (attempts[[#]] += 1) <
maxAttempts),
FrontEndExecute[{FrontEndFrontEndToken[
FrontEndInputNotebook[], "MoveNextPlaceHolder"]}]];
];
evaluation[[#]] =
check[stringCheck[ans[[#]]], ref[[#]]]]],
String,
FieldSize -> 15,
Enabled -> Dynamic[active[[#]]]
],
""}[[evaluation[[#]]]]],
Row[{Dynamic[attempts[[#]]], " / ", maxAttempts}]} &,
Length[ref]
],
Map[Style[#, "Subsection"] &,
],
{InputField["", String,
FieldSize -> 1, ImageSize -> {0, 0},
Appearance -> "Frameless"]
}
]
],
FrameStyle -> Orange,
],
Row[{feedback, Dynamic[Count[evaluation, 1]]},
Background -> Orange, Frame -> True, FrameStyle -> None,
FrameMargins -> 5, RoundingRadius -> 5], Bottom,
BaseStyle -> "Subsubsection",
LabelStyle -> {"Subsection", LightOrange},
Background -> LightOrange]]


The starting point is quantQuestions which wraps the set of questions in HoldForm so they can be displayed as such. Internally, they are then unwrapped with quantAnswers to get the simplified results with which to compare the user's input later.

The only other user-facing function is prepareQuestions which is then applied like this:

prepareQuestions[
quantQuestions[
D[Sin[x + y], y],
D[Log[x], x],
Integrate[1/x^2, x],
Integrate[x, {x, 0, 1}]
]
]


This creates the dynamic input fields as shown here:

You can try to enter the answers in different ways. Also try to enter cheats - they are supposed to be caught by the function stringCheck. Its job is to comb the held form of the expression in the input string for non-numerical functions, of which D and Integrate are examples.

Edit

In the updated code above, I added a check for correct Mathematica syntax using SyntaxQ. Other improvements are:

• The number of attempts at an answer is counted, and the input field is disabled after a maximum number (3 here).
• The total number of correct answers is displayed at the bottom.
• If a question has no answer, the error is displayed and no table is created.
• The problems are displayed using TraditionalForm (whether this is better is a matter of taste).

Edit 2

As pointed out in comments by Andrew and Murray, indefinite integrals have to admit constant offset. This can be done by comparing the derivatives, and I've implemented that for a single variable by substituting a formal wrapper (integralWrapper) into the automatically computed answer (quantAnswers) that tells check to differentiate instead of integrate. It is only needed for indefinite integrals, and the rest of the validation (using PossibleZerQ) works as before.

I'm also suppressing the generation of conditions for definite integrals, so that questions such as Integrate[x,{x,y,z}] with unknown boundaries can be handled too.

-
The latter part of this answer of mine is complementary to your stringCheck function. While I like your idea, if I were to implement such an exam module, I'd most certainly use only a whitelisted set of functions (even if malice was not a concern). This will also take care of cheating with D or Integrate... –  rm -rf May 21 '12 at 5:14
@R.M. I agree - at the moment my white-list consists of symbols, and of functions that have attribute NumericFunction. If it turns out I should be more restrictive, I'll do that. I think that the current filter is enough to prevent ToExpression to be exploited for injecting code because NumericFunction functions aren't able to do that. If I'm wrong about that, I'd certainly be able to just admit just the "scientific calculator" range of functions. –  Jens May 21 '12 at 5:36
Very pretty and +1, but there are some problems: I evaluate your example, enter something incorrect, e.g. Cos[x], and press Enter (1). I press Enter again (2), without changing the answer, and the attempt counter goes to 2. It probably shouldn't increment if the input has not changed at all (it could be an accidental press of Enter). Then I press Enter again (3), and the input field gets disabled. Then I press Enter again (4), and the input field disappears (!) and the Correct Answers counter gets corrupted. I press Enter again (5) and my Front End crashes. Can you reproduce the crash? –  Szabolcs May 26 '12 at 19:12
@Szabolcs Yes I can indeed reproduce the crash. No big surprise - dynamic interactivity is just very unstable. As the other answers say: it's a "proof of concept"... just with slightly different concepts. I'll see if I can find any way around that crash later. –  Jens May 27 '12 at 0:47
@Szabolcs I've updated the code. De-activating an InputField while the cursor is in it seems to have caused the crash, so I now send a FrontEndToken to move to the next field when that happens. When all fields are inactive, there is a dummy field with ImageSize zero to jump to, so hopefully the cursor is now always in a legal place... –  Jens May 27 '12 at 6:06

Here is a proof-of-concept of something that you can build upon to create such homework assignments.

First, a helper function to check the correct answer and display the result. I'm only checking for accuracy to the third decimal, but you can tweak that as you wish.

ClearAll[checkAnswer]
Abs[N[ans] - correct] < 0.001,
Style[ "Correct!", Darker@Green],
Style[ "Wrong. Try again!", Red]
]


Next, the basic module that asks the question and gets the student to input the answer.

Module[{question = HoldForm[Integrate[x^2, {x, 0, 2}]], correct},
correct = ReleaseHold@question // N;

DynamicModule[{ans = Null},
Column[{
Row[{"What is the value of ", question, "?"}],