Define review criteria

Configuration -> Paper review -> Review criteria

The review criteria in Converia determine the standards by which the reviewers evaluate submissions. The program committee will also make the final evaluation based on many of these criteria. There is a wide range of evaluation options: between individual feedback and strict grading, there are many other gradations.

A classic paper review is shown in Fig. 1 in the frontend and Fig. 2 with the corresponding configuration in the backend.

Fig. 1: Review form for external reviewers in frontend

Fig. 2: Corresponding configuration of review criteria in Converia backend

Criteria groups

The review criteria can be arranged in different criteria groups. In the frontend, the names of the groups are displayed in the heading. These are clearly visible in Fig. 1 (Quality/Decision/Topics).

Fig. 3: Default setting for criteria groups

Creating a new review criterion

Fig. 4: Default setting for review criteria

Two review criteria are already set up by default in the copy template. If additional criteria are needed, they can be added using the Create new entry button.

Fig. 5 Creating a new review criterion

1. Determines the name of the criterion for differentiation in the backend.

2. The name that the criterion should have in the frontend (Fig. 1).

3. Additional information for the criterion can be added in the description.

4. Determines the reference of the criterion. If the same criteria are to be used in the abstract and manuscript review, the reference must be adjusted before the manuscript review begins. The following references are possible: a) Abstract b) Manuscript

5. Determines the type of criterion. The following types are available: a) Grade/Quality b) Classification c) Presentation d) Internal comment

6. Decide whether the criterion should be a mandatory field. The following options are available: a) Always mandatory b) Optional c) Mandatory if paper is not accepted d) Optional if paper is not accepted

7. Whether any points should be taken into account.

8. Assignment to a criteria group.

9. Short title for the criterion, used as a column title in the final review.

Grade/Quality

If a criterion of the type Grade/Quality has been created, the answers displayed in the frontend can be configured in the Answers tab.

Fig. 6: Sample answer options for a Grade/Quality criterion and internal values

A name, an effect and a value can be defined here. For the effect, a distinction is made between no effect and point score not applicable. The former is the default for a criterion of the type Grade/Quality. If, for example, there is an additional option to abstain from the review, the effect point score not applicable would be necessary so that the assigned value is not included in the review and distorts averages.

Classification

If a criterion of the type Classification has been created, the answers output in the frontend can be configured in the Answers tab.

Fig. 7: Sample answer options for a Classification criterion and their effects

The name and effect are assigned to the selection options here. The effects of the evaluation criterion are also required for step 3 in the workflows, since the classification of the abstracts is based on the review criterion.

It is essential to have created exactly one criterion of the type classification and assigned the appropriate effects. Otherwise paper review and workflows will not function correctly.

The effects of the review criterion of the type classification control the colored display of the reviewed papers in the frontend:

Fig. 8: Colored display of the reviewed papers in the frontend, controlled by effects of classification criterion

Presentation

In order for the program committee to be able to adjust the presentation form in the final review, a criterion of the type presentation must be created.

Fig. 9: Sample answer options of a presentation criterion

Here, appropriate effects can be assigned to the presentation forms. The effects refer to the presentation forms created in the configuration of the submission. In most cases, the effect of the same name is assigned to the presentation form.

Internal comment

An internal comment can be used to record messages or assessments from the reviewers to the program committee. It is essential for an internal comment criterion to configure the input field type to single-line or multi-line on the Answers tab.

Fig. 10: Sample answer configuration for a criterion of the type internal comment

The contents of the internal comment review criterion cannot be sent to the authors of the papers.

Power session reviewing criteria

Watch the Converia Power Session on review criteria (German).