Evaluation types-Table for mathematical answers

In short, an evaluation type (when used in a solution) determines the way how the computer checks whether the student’s answer is correct or not.

Evaluation types are used in the “Solutions”, “Negative feedback” and “Positive feedback” tabs of an exercise. Below, we list a table of all evaluation types including their description. For some evaluation types, a separate article has been written including some examples of the use of this evaluation type. Links to these articles will be provided in the table. More information about automated feedback can be found here.

In case of relations like equations or inequalities, you can specify a rule to only apply to a side of the relations using rhs: for right hand side or lhs: for left hand side, before the definition. E.g. the evaluation type contains_var with Definition rhs:x checks for the variable x in only the right hand side.

A list of all evaluation types for mathematical answers

Name	Description	Automated feedback
contains_brackets	Checks whether the user input contains brackets. Note that brackets for function arguments like ln() also count as brackets. With no arguments at least one occurrence triggers the rule. With one argument, the argument defines the number of bracket pairs that occur, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =
contains_float	Checks whether the user input contains floats. With no arguments at least one occurrence triggers the rule. With one argument, the argument defines the number of occurrences, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =
contains_integer	Checks whether the user input contains integers. With no arguments at least one occurrence triggers the rule. With one argument, the argument defines the number of occurrences, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =
contains_number	Checks whether the number in the Definition field is present in the user input. Works for both positive and negative floats and integers. Without argument (Definition field is empty), at least one occurrence of a number in the answer triggers the rule. With two arguments (; separated) , the second argument defines the number of occurrences. e.g. 2;2 optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =. eg. 2; le 2
contains_one_base_power	Answer contains exactly one power at the base.
contains_symbol	Checks whether the user input contains the symbol from the Definition field. With one argument, at least one occurrence triggers the rule. e.g. minus With two arguments (; separated) , the second argument defines the number of occurrences. e.g. minus;2, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =. eg. minus; le 2. With a list in the Definition field like [arg1,arg2], each element will be treated as an argument like above, where the arguments will be treated as an OR. i.e. [cos;lt2,sin;lt2] is whether either sin or cos occur less than two times. Alternatively, the occurrence can be placed after the list, like so [arg1,arg2];lt2. e.g. [sin,cos];lt2 checks whether sin or cos together occur less than two times. The symbol is one of operators: abs, divide, minus, plus, power, root, times, unary_minus, factorial relations: approx, eq, geq, gt, leq, lt, neq trigonometry: arccos, arcsec, arcsin, arctan, cos, cot, sec, sin, tan logarithms: ln, log numbers: e, i, infinity, pi linear algebra: matrix,vector logic: and, equivalent, false, implies, not, or, true, xor sets: N, Q, R, Z, cartesian_product, in, intersect, notin, notsubset, set, setdiff, size, subset, union, emptyset, suchthat, list limits: limit, null, lambda permutations: permutation, cycle units: degree ​ You kind find the symbols in a student answer using the editor.
contains_symbol_other_than	The user answer does not contain any other symbol than the one in the list in the Definition field, separated by ;, e.g. plus;minus means that only the symbols plus or minus may be present in the user`s answer. The symbol is one the list as specified in the contains_symbol evaluation type.
contains_symbol_within	Checks if the second operator in the definition field is present in the first operator of the answer. i.e. times,plus checks whether there is a multiplication that has a plus as a descendant. The second argument can be a list, where multiple symbols can be searched with an OR. The third argument optionally specifies the position of the child. i.e. times,[plus,minus] checks whether there is a multiplication that has a plus or a minus as a descendant. divide,[plus,minus],2 checks whether the denominator of a fraction is a plus or minus or has one in it.
contains_variable	Checks whether the user input contains the variable from the Definition field. Without argument (Definition field is empty), at least one occurrence of a variable in the answer triggers the rule. With one argument, at least one occurrence triggers the rule. e.g. x With two arguments (; separated) , the second argument defines the number of occurrences. e.g. x;2, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =. eg. x;gt 2 ​ With a list in the Definition field like [arg1,arg2], each element will be treated as an argument like above, where the arguments will be treated as an OR. i.e. [x;lt2,y;lt2] is whether either x or y occur less than two times. Alternatively, the occurrence can be placed after the list, like so [arg1,arg2];lt2. e.g. [x,y];lt2 checks whether x or y together occur less than two times.
contains_variable_in_symbol	Checks if the variable in the definition field is present in the first operator/symbol of the answer. i.e times,x checks whether there is a multiplication that has a variable x as a descendant. The second argument can be a list, where multiple symbols can be searched with an OR. The third argument optionally specifies the position of the child. i.e. times,[x,y] checks whether there is a multiplication that has a variable x or a variable y as a descendant. divide,[x,y],2 checks whether the denominator of a fraction has a variable x or a variable y in it.
contains_variable_other_than	The user answer does not contain any other variable than the one in the list in the Definition field, separated by ;, e.g. x;y means that only the variables x or y may be present in the user answer.
default_incorrect	Always reacts on wrong answers and has a predefined priority of 99.
eq_one_side	One side of the equation must be equal to the variable in the Definition. e.g. x means the the answer must be either something like x=...x=... or ...=x...=x
eval_domain	Checks whether the user input is equal to the domain in Definition field where both should be in the form of a simplified equation in the disjunctive normal form with only one variable, e.g. x=3 or x=-3.
eval_eq_exact	Check whether solutions indicated in the Definition field are solutions to the equation in the user input, using substitution. Define a list of equations in the Definition field that should be substituted in the equation the user input.
eval_eq_numeric	Numerical check whether solutions indicated in the Definition field are solutions to the equation in the user input. The Precision field indicates the accepted margin of error. Define a list of variables in the Definition field that should be substituted in the equation the student answers. e.g. [x=3,y=4] when correct answer: x2+y=13x2+y=13.
eval_exact	Exact mathematical comparison of expression in the Definition field with user input. No simplification rules are applied to the student answer.
eval_graphs	Lines: Checks whether each pair of points from the user input is on a line defined in the Definition field and whether each line in the Definition fields has two points in the user input. e.g. x=y+1 ​Points: Checks whether each point from the user input is defined once in the Definition field. e.g. [2,1] ​Vector: Checks whether the start and end point of the vector from the user input is defined in the Definition field via a 2x2 matrix. Define the vector via a matrix e.g. matrix([0,0],[2,1])
eval_interval	Checks whether the interval provided by the student is equal to that in the Definition field, where the Definition is specified as a 1st or 2nd degree (in)equality. The variable you use is free, as long you do not use more than one. e.g. x^2+3*x >= 0 for [0,∞)∪(−∞,−3][0,∞)∪(−∞,−3] Note that automatic feedback is given for answers that have [−∞[−∞ or ∞]∞].
eval_lin_eqs	Evaluate a system of linear equations. As a Definition, write the maxima equations in disjunctive normal form, and after that write a ";" with a list of variables in "[]" and separated with a comma ",". e.g. (2*x=2) and (3*y=2) and (2*z=5) ; [x,y,z] Optionally, use a third argument and set it to 1 to escape the mandatory final form of the answer.
eval_list	Checks if the answer is a list and if each expression is equal to the expression at the same index in the list of the Definition field, using eval normal. The list in the Definition field should be comma "," separated. e.g. [2,3,1]. Note that this evaluation types automatically checks for the right input form (list or vector) and the length of the list and gives appropriate feedback accordingly.
eval_logic_equiv	Comparison of logical form in the Definition field with normal form of user input.
eval_matrix	Check if the matrix from the Definition field is equal to the answer. Can be used with fb_matrix for feedback. It accepts input of matrix or a list of lists. e.g. matrix([1,2,3],[4,5,6]) or [[1,2,3],[4,5,6]]
eval_matrix_element	Numerically compares the [row,column] entry of the matrix from the user input [row,col]=...with the number represented by the expression in the Definition field. E.g. [1,1]=2 means that the value 2 must be equal to the matrix element on dimension [1,1]
eval_normal	Exact mathematical comparison of normal form of expression in the Definition field with normal form of user input. e.g. x^2.
eval_numeric	Numeric comparison of expression in the Definition field with user input, e.g. 2*3. The Precision field defines the accepted absolute value of the margin of error for numerical answers, e.g. if the correct answer is 1, you can use 0.1 to accept all answers between 0.9 and 1.1. If any of the Range fields are used (not 0 or empty), the system checks whether the answer is within the range.
eval_openmath	Comparison between the OpenMath xml object in the Definition field and the result object of the student input in the formula editor. To create a solution, get the Transformed OpenMath from the editor here.
eval_param_rep	Evaluate a parameter representation of the solution set of a list of linear equations. In the Definition field, use 2 arguments (separated with a ;), where the first is the list of linear equations [eq1,eq2,...] and the second is a list of the coordinates [x,y,...]. Optionally, you can add a third argument to prescribe the parameters that may be used, like [lambda,mu]. e.g. [z-y+2*x-1=0,3*z+11*y-4*u+9=0];[x,y,z,u] or with a third argument [z-y+2*x-1=0,3*z+11*y-4*u+9=0];[x,y,z,u];[lambda,mu]
eval_polynomial	Automated polynomial checker that can be used together with fb_polynomial. In the Definition field a (multivariate) polynomial is entered to specify the correct answer. Then you can create just one positive and one negative feedback rule in with evaluation type fb_polynomial that will provide automated feedback. Feedback works for basic, free (any polynomial), expanded and factored form. See fb_polynomial for more information about these form types.
eval_rational_function	Automated checker for rational functions that can be used together with fb_rational_function. In the Definition field a rational function is entered to specify the correct answer, e.g. (x^2+1)/(x+6). Then you can create just one positive and one negative feedback rule in with evaluation type fb_rational_function that will provide automated feedback. Feedback works for basic, combined , split and quotient form. See fb_rational_function for more information about these form types.
eval_statement	Evaluate the maxima statement in the Definition field, where the user input can be referenced using answer for the answer that belongs to the rule, or answers for alist of the answers in all answer fields like [1,2,3]. In case an answer value is empty, the answer value to maxima gets replaced with %null in the answers list. A simple example of the usage of answer is e.g. elementp(answer,{sin(1),sin(2),sin(3)}). Or you can write your own maxma block function. For instance, you can differentiate the answer to check if it is a valid integral by automatically differentiation away the constants. e.g. for the integral of sin(x)sin⁡(x) you could use block(adiff:diff(answer,x),is(radcan(adiff-(sin(x)))=0))to allow for all answer that are correct after differentiating them, like −cos(x)+c−cos⁡(x)+c, but also −cos(x)+C−cos⁡(x)+C ​ Normally the answer field specifies the answer. For input graphs all answers are checked against the statement and the answer form is ​Lines: [[x_1,y_1],[x_2,y_2]] ​Points: [x,y] ​Vector: [[x_1,y_1],[x_2,y_2]]
eval_units	Converts the answer to base units and the compares the value and unit. For units support, maxima's ezunits package is used. In the Definition field you can specify the solution with a backtick ` between the value and the unit, like 1`g. The Precision field defines the accepted absolute value of the margin of error in terms of the base unit specified in the Definition field, e.g. ​Definition 10`g with Precision 0.1 accepts all answers between and including 9.9 and 10.1 grams. If any of the Range fields are used (not 0 or empty), the system checks whether the answer is within the range. e.g. ​Low 0.9`m and High 1.2`m accepts all answers between 0.9 and 1.2 meter. For supported units and their notation, have a look at the Units table.
eval_vector_rep	Checks whether the vector representation of the user input corresponds to the line defined by the linear equation in x and y in the Definition field. e.g. y+3*x=1
eval_vector_lin	Checks whether the two dimensional vector of the user input and the two dimensional vector in the Definition field are linearly dependant. e.g. [2,1]
fb_generic	Provides automatic negative feedback on the student answer. The student answer will be compared with the expression in the Definition field of this feedback rule. This feedback type is particularly good at pointing out minor mistakes in the student answer. Use the Precision field to indicate a minimum percentage of similarity between the Defintion and the answer at which the feedback kicks in (0 is the default).
fb_lin_eqs	Use only with a solution rule that uses evaluation type eval_lin_eqs to give feedback the following specified cases. Use one of the following in the Definition field. - 1 or nonlinear: the answer contains a nonlinear equation - 2 or toobig: each solution of the original system of equations occurs in the answer, but at least one solution of the answer is not a solution of the original system - 3 or toosmall: each solution of the answer is correct, but at least one solution of the original system does not appear in the answer - 4 or incomparable: the answer is incomparable - 5 or strangevar: the answer contains a variable that is not an unknown - 6 or noequation: at least one of the expressions separated by the logic symbols is not an equation - 7 or andornestingtoodeep: the nesting of the logic symbols is too complicated. We only accept answers in which the or symbol never appears in an expression that consists of two or more parts separated by an and symbol ​
fb_matrix	Use only with a solution rule that uses evaluation type eval_matrix to give feedback the following specified cases. Use one of the following in the Definition field. - 2 or wrong_length: answer does not have the same dimensions as the matrix in the Definition field of the solution - 3 or wrong_matrix: answer does have the right dimensions but is wrong - 4 or no_matrix: The answer does not evaluate to a matrix. ​
fb_param_rep	Use only with a solution rule that uses evaluation type eval_param_rep to give feedback the following specified cases. Use one of the following in the Definition field. - 2 or no_vector: answer is not a vector - 3 or wrong_length: answer does not have the right length - 4 or wrong_basevec: the base vector of the answer is not a solution of the equations - 5 or too_few_params: answer does not have enough parameters - 6 or too_many_params: answer has too many parameters - 7 or wrong_params: the parameters in the answer are not as required - 8 or non_linear: answer is non-linear on one of the parameters - 9 or wrong_vector: answer has at least one vector that is not a solution of the equations - 10 or lin_dep: the direction vectors in the answer are linearly dependent ​
fb_polynomial	Provides automated feedback coming from eval polynomial in the solution. The value in the Definition specifies what type of polynomial form is desired. The value in the Definition van be ​free: Polynomial in free form. Performs no form checks. ​basic: Polynomial in basic form. Positive/Negative feedback checks are the same: ​ Invalid variables Missing variables Invalid operations Non-constant exponents Dividing by non-constants Non-integer exponents Negative exponents expanded: Polynomial in expanded form. Positive feedback checks: ​ All basic checks Check for non-atoms under exponents, i.e. (x y)^3 Check for multiplication by non-constant non-atoms, i.e. (x + 1) 2 Check for variables appearing twice in multiplication, i.e. x^2 y x^3 Check for monomials appearing multiple times, i.e. x + x + x Negative feedback checks: ​ All basic checks Check if degrees match Check if there are superfluous monomials Check if there are monomials missing Check for wrong coefficients factored_basic: Polynomial in factored form. Postive feedback checks: ​ All basic checks Check if polynomial is factorized more then the authors answer. Performs positive feedback checks for expanded polynomial against each factor. Negative feedback checks: ​ All basic checks Check if all factors divide the authors answer. Check if there are factors missing. Check if the constant is right. factored_basic_count: Sames as factored_basic, but with an extra positive feedback check if the number of factors are correct. ​ ​factored_expanded: Polynomial in factored form. Postive feedback checks: ​ All basic checks Check if polynomial is factorized more then the authors answer. Negative feedback checks: ​ All basic checks Check if all factors divide the authors answer. Check if there are factors missing. Check if the constant is right. factored_expanded_count: Sames as factored_expanded, but with an extra positive feedback check if the number of factors are correct. ​
fb_rational_function	Use only with a solution rule that uses evaluation type eval rational function to give feedback on the following specified cases. Use one of the following in the Definition field: - basic: Answer should be constructed using standard operations [+, -, *, /] and raising to integral exponents - combined: Answer should be given as P / Q with P, Q coprime polynomials. - split: Answer should be given as a sum P + sum P_{ij} / Q_i^j, with the Q_i irreducible and deg(P_{ij}) <= deg(Q_i). - quotient: Answer is p/q with p and q expanded polynomials. Not necessarily coprime.
fb_solve_in(eq) previously called domain_dependence	Use only with a solution rule that uses evaluation type solve_inequality or solve_equality to give feedback on how the domain of the answer compares to the domain of the solution. Use one of the following in the Definition field. - 0 or none: the answer domain is neither in nor around the solution domain - 2 or inside: The answer is in the solution domain, but it does not contain it; it is too small. - 3 or contains: The answer contains the solution domain, but is not contained in it; it is too big.
fraction_denominator	Checks whether the user input contains a fraction with another denominator as defined in the Definition field. e.g. 4
fraction_diff_denominators	When the user input has more than one fraction, it checks whether all fractions have the same denominator.
fraction_is_improper	Checks whether the absolute value of all fractions in user input is greater than or equal to one
fraction_lcm_not_used	Checks for each fraction in user input if the denominators are not equal to or greater than the number in the Definition field.
fraction_not_simplified	Checks whether all fractions in the user input are simplified. In the Definition field you can use the same values as for is_not_simplified_fraction to control what type of fractions are considered simplified.
is_decimal_length	Checks whether the user input is a float with the same number of decimals as specified in the Definition field, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, = E.g. gt2 checks whether the answer contains a float with more than two decimals.
is_length	Checks whether the user input is a float or integer with the same length as specified in the Definition field, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =. eg. gt 2
is_not_factorized	Comparison of the factors in the Definition with the factors of the answer. e.g. Definition (x-3)*(x+2) can be used when it's asked to factorize x2−x−6x2−x−6
is_not_float	Check whether the user input is not a float.
is_not_fraction	Check whether the user input is not a fraction.
is_not_ifactorized	Checks if the answer is decomposed into prime factors. No Definition is needed.
is_not_integer	Checks whether the user input is not an integer.
is_not_number	Checks whether the user input is not an integer and not a float.
is_not_polynomial	Checks if the answer is a polynomial in the variables in the list from the Definition field. With two arguments, the second is a test for the grade. e.g. x;le6 checks if the answer is a polynomial in x of degree at most 6.
is_not_rational	Check whether the user input is either an integer or a fraction with the denominator unequal to zero.
is_not_simplified_fraction	Checks whether the user input is a simplified fraction. Also works for equations, where it checks whether at least one side is a simplified fraction. With Definition set to - 1 the final form must be a proper fraction or integer like 112112, 11 or 1212 but not 3232 or 4242. The minus must be in front of the fraction. - 2 allows for both proper and improper fractions and integers like 112112 and 3232 but not 4242. The minus is not allowed in the denominator. - 3 same as 1, only also float values will be accepted. - 4 same as 2, only also float values will be accepted. - otherwise it only allows improper fractions and integers like 1212 or 3232 but not 112112. The minus is not allowed in the denominator.
is_not_simplified_root	Checks whether the answer (which is assumed to be an expression involving integers, multiplications, divisions, and higher roots) is in the normal form (a/b) * (c-th root of d), where b,c,d are positive integers and a is an integer.
is_not_simplified_complex	Checks whether the answer is not in the form a+b*i, b*i, i or c where a,b and c are either simplified fractions, floats or integers and a and b are not 0 and, in the case of a+b*i, there's no minus in b*i. Examples of allowed forms are 3+2i,3+i,−3−2i,0,2,12+12i,2i,2.2i,1.5+0.5i,0.5,343+2i,3+i,-3-2i,0,2,12+12i,2i,2.2i,1.5+0.5i,0.5,34 Examples of disallowed forms are 0+i,1+0i,1+42i,22+2⋅i,1+1+i,1+i2,1+−2i,1−−2i,420+i,1+0i,1+42i,22+2⋅i,1+1+i,1+i2,1+-2i,1--2i,42
is_not_top_symbol	Checks whether the symbol with the highest precedence in the answer is not equal to one of the symbols in the Definition field. Define a list of one or more symbols in the Definition field. e.g. times;root means the highest symbol is either a times or a root symbol.
is_not_vector_length	Check whether the user input has all vectors of the same length. Optionally, use an integer in the Definition field to expect a vector with the prescribe the length, optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, = E.g. gt2 checks whether the answer is a vector with more than two elements.
is_not_unit	Check whether the student answer is a unit. With no Definition specified (empty), a check will be done whether the student answer is a unit. The Definition can be used to specify the unit(s) that must be used in the answer, like gram. When multiple units are needed, you can use a list (separated with a comma) of units that must be used in the answer, like gram,liter for g/l. By default, all prefixes are allowed to be used. However, per unit, you can specify which prefixes are allowed by specifying a preceding list of prefixes between square brackets, like gram[centi,milli]. If you want to allow for a unit to be used without a prefix, like m, then use the prefix one, like [one,kilo;centi]meter,[one,kilo;centi]gramme.
same_form	Checks the answer and the definition for having exactly the same form after maxima translation
solve_equation	Automatically solve and give feedback on 1st or 2nd degree equation. This type is limited to one variable equations. e.g. x^2=9
solve_inequality	Evaluate the inequality in the Definition field and evaluate the student answer accordingly. This type is limited to one variable inequalities. e.g. x^2<9
significant_digits	Checks if the number of significant digits present in the student answer is equal to the integer in the definition field. Optionally specified by one of le, lt, ge, gt, ne, eq, <=, >=, <>, ==, !=, <, >, =. e.g. >= 2 is triggered whenever the number of significant digits is greater than 2.
is_not_scientific_notation	Checks if the student answer is in scientific notation. I.e. a*10^n where n is an integer and 1 <= a < 10.