System

Administration
Guide
Contents
Introduction ............................................................................................................................................ 4
Study Dashboard ..................................................................................................................................... 4
Active Users ........................................................................................................................................ 4
Removing Users .............................................................................................................................. 6
Missing Data........................................................................................................................................ 8
Stakeholder data breakdown............................................................................................................ 10
Additional Outcomes ........................................................................................................................ 10
Reasons for Change .......................................................................................................................... 11
Rounds .............................................................................................................................................. 11
Study Management............................................................................................................................... 12
Add Administrator............................................................................................................................. 13
Manage Users ................................................................................................................................... 13
Manage missing data ........................................................................................................................ 13
Manage Additional Outcomes .......................................................................................................... 13
Calculate feedback ............................................................................................................................ 16
Manage next round invites ............................................................................................................... 18
Send bespoke email .......................................................................................................................... 19
Send Consensus Meeting Invites ...................................................................................................... 23
Send Post Study Emails ..................................................................................................................... 24
Study Structure ..................................................................................................................................... 26
Manage Outcomes ............................................................................................................................ 26
Manage Stakeholders ....................................................................................................................... 28
Manage Registration Form ............................................................................................................... 30
Manage Logo..................................................................................................................................... 31
Manage Rounds ................................................................................................................................ 32
Manage Pages ................................................................................................................................... 32
Manage Email Templates .................................................................................................................. 34
Manage Footer Pages ....................................................................................................................... 37
Data Extracts ......................................................................................................................................... 39
Download Ratings ............................................................................................................................. 39
Download User Data ......................................................................................................................... 39

System Administration Guide v5.0 Page 2 of 41

 Download Additional Outcomes ....................................................................................................... 40
Download Round 1 Missing Data ...................................................................................................... 40
Download Acknowledgement ........................................................................................................... 40
Download OutcomeIDs ..................................................................................................................... 40
Consensus Reports ............................................................................................................................ 40

System Administration Guide v5.0 Page 3 of 41

Introduction
This document will describe how to use all of the administration functions within DelphiManager,
with specific reference as to how they will allow you to manage your Delphi study from start to
finish. However, it is assumed that your study is already set up and running. If not, you should refer
to the document “DelphiManager – Study Setup Guide”

Study Dashboard
The study dashboard provides a quick overview of the progress of your study. Once your study has
started, this is the page you will be taken to by default when you log into the administration system.

Active Users
The first section of the study dashboard is User Data. This shows how many people have registered
in total, and how many have completed/not completed for each of your study rounds. The image
below shows a study in round 1:

This shows that:

 4 users currently active for the study

 3 users completed round 1
 1 user has yet to complete round 1

To view all the users who have registered for your study, click the number of total active users, in
this case 4. You will see the following page:

System Administration Guide v5.0 Page 4 of 41

Here you can download all the user data by clicking the “Download this data” button. If you want to
edit the details of a registered user, if they have entered an invalid email address for example, you
can search for the user in question and edit their details by clicking the “[Edit]” button for that row.
You will see the following page:

System Administration Guide v5.0 Page 5 of 41

If you need to you can amend the Name, Email address or Stakeholder group (where applicable).
Click “Update” to save any changes you have made. This will automatically send the user an email
reaffirming their registration details. This will only work if any of the details have changed.

Removing Users
It may occasionally be necessary to remove users from the system. There are 2 categories of
removing users:

1. Delete User – to be used in the event of an erroneous or possibly spam registration.

2. Withdraw User – to be used when a participant has made it known that they no longer wish
to continue with the study.

To remove a user, click the Delete link next to their record in the Manage Users page, you should see
a page similar to the following:

System Administration Guide v5.0 Page 6 of 41

If you want to delete a user, just click the Delete button. This will remove the user from the system
entirely. They will no longer be counted as a registered user, nor will any of their data appear in a
data extract.

If you want to withdraw a user, change the “Type of deletion” drop down menu to Withdraw user;
the page should change slightly to be similar to that below:

System Administration Guide v5.0 Page 7 of 41

You are required to provide a reason why you wish to withdraw this participant e.g. they have said
they no longer want to continue. Click the Delete button when you have finished. This will also
remove this user from the Active User count in the Study Dashboard, however they will still appear
in User Data and Ratings data extracts but marked as “withdrawn”. A full list of deleted/withdrawn
users can be downloaded from the Data Extracts section.

Missing Data

To view a list of users who have not completed a round, click the number in brackets. In the example
below, we are looking missing data from round 1. You should see the following page:

System Administration Guide v5.0 Page 8 of 41

Again you can download this data by clicking the “Download this data” button. This page shows how
many of the outcomes have been rated by each user (outcomes answered) and when they last used
the system (date updated).

If you wish to send these users a reminder email, you can do so by clicking the “Send Email” button.
This email is based on the template “Missing Data”. If you have not edited the standard text for this
template, you can do so in Study Structure -> Manage Email Templates. This will be explained
further on page 31.

System Administration Guide v5.0 Page 9 of 41

Stakeholder data breakdown
If your study is using the Stakeholder model, you are shown the breakdown of completed/not
completed users per stakeholder group, as shown below:

Additional Outcomes
The total number of additional outcomes suggested by users and added to the study is displayed in
this section. The example below shows that 98 additional outcomes were suggested in round 1 but
only 4 were added to the study for round 2.

You can view all additional outcomes by clicking on the number, in this case 1. You should see a page
similar to below:

Including Additional Outcomes between study rounds is covered further on page 13.

System Administration Guide v5.0 Page 10 of 41

Reasons for Change
If selected during setup, Reasons for Change shows how many participants have changed their
ratings to cross the 3 Likert sections e.g. from 6 to 7 and given a reason for this change. The example
below shows that there are 11 reasons for change been given in round 2. This data can be
downloaded by clicking on the number.

Rounds
The start and close dates for each round of your study are displayed here, with the active round
being highlighted in blue. In the example below, round 1 is the active round:

You can make changes to the round dates at any point during your study. If you wish to extend the
time for a particular round, click “[Edit]” next to the round in question. You should see a page similar
to that below:

Use the calendar picker to change the date required and click “Update” when you have finished.

System Administration Guide v5.0 Page 11 of 41

Study Management

The Study Management section is located by clicking on the appropriate link in the main
administrator menu. The Study Management should be similar to the image below:

The key management functions for DelphiManager are found in this section. They are:

 Manage administrators
 Manage users
 Manage missing data
 Manage additional outcomes
 Calculate feedback
 Manage next round invite emails
 Send bespoke email
 Send Consensus Meeting Invites

System Administration Guide v5.0 Page 12 of 41

  Send post study emails
NOTE: The last 2 options are only available once the study has been closed i.e. after the close date
of the final round

Add Administrator
You can edit an existing administrator by clicking the “[Edit]” button next to the appropriate entry on
the Study Management page. If you wish to add a new administrator, click the “Add Administrator”
button. You should see the page below:

Fill in the appropriate details and click the “add” button to create a new administrator account.

Manage Users
This functionality has already been covered earlier in this document. See Active Users (page 4)

Manage missing data

This functionality has already been covered earlier in this document. See Missing Data (page 8)

Manage Additional Outcomes

This section will show how to add an additional outcome in between rounds. Click on the Manage
Additional Outcomes button, you should see a page similar to that below:

System Administration Guide v5.0 Page 13 of 41

If you do not see the “[Add]” option in the Add to Study column, this is because the current round is
still active. Additional Outcomes can only be added in between study rounds.

System Administration Guide v5.0 Page 14 of 41

To add an additional outcome, click the “[Add]” button; you should see a page similar to that below:

From the drop down list, choose the domain to which you would like to add this additional outcome.
The outcome text and help text can be edited at this point; the rating entered by the participant is
shown for reference only. When you have made any required changes, click the “Add outcome to
study” button. This will include the outcome in the upcoming round, in the domain selected, after all
other outcomes in that domain. You should then see a page denoting that the outcome has been
added to the next study round, similar to that below:

System Administration Guide v5.0 Page 15 of 41

Calculate feedback
In rounds 2 and 3, depending on the setup of your study, the participants are shown the percentage
distribution of scoring from 1-9 from the previous round, along with their own rating. This is
accomplished by the calculate feedback function. Click on “Calculate Round 1 Feedback”, you should
see the following screen:

System Administration Guide v5.0 Page 16 of 41

Clicking on the “Calculate” button will begin the process. This can take anything from a few seconds
to a few minutes, depending on the number of outcomes and participants in your study. During the
calculation of the feedback, the “Calculate” button will be disabled and you will be shown a
“processing” message, as below:

Once all the calculations have been made, you will be shown a message informing you of this.

System Administration Guide v5.0 Page 17 of 41

Manage next round invites
Depending upon which round has just closed, the Manage R2 Invites and/or Manage R3 Invites links
will be visible. The image below shows an example of Manage R2 Invites:

Users who have rated all the outcomes from the previous round will automatically be included in the
invites for the next round, as shown by the checked box in the “Include In R2” column. You can
choose to include any user in the next round by checking the box. Conversely, if you wish you
exclude any user from the next round, locate them using the search function and uncheck the box
accordingly.

The email that is sent for both round 2 and round 3 emails comes from 1 template called “Next
Round”. This template can be amended via Study Structure -> Manage Email Templates. More
information about this can be found on page 31.

When you are happy with the Next Round email template for the appropriate round, clicking the
“Send Email” button will send the email to all users for whom the Include in R2 box was checked.

System Administration Guide v5.0 Page 18 of 41

Send bespoke email
If you want to be able to send an email to an individual or group of stakeholders, outside the email
template functionality, click on “Send Bespoke Email”. You will see a page similar to the image
below:

If your study is using the Stakeholder model, you can choose to send an email to a particular
stakeholder group. To do this, choose “Stakeholder Group” from the “Send email to” drop down

System Administration Guide v5.0 Page 19 of 41

menu. The page should change to display a drop down list of your stakeholder groups, similar to that
below:

Select the stakeholder group you wish to email.

System Administration Guide v5.0 Page 20 of 41

If you are not using the Stakeholder model, or you only wish to contact individual participants, select
“Pick emails from list” from the “Send email to” drop down list. You should see a page similar to that
below:

System Administration Guide v5.0 Page 21 of 41

System Administration Guide v5.0 Page 22 of 41
Checking the box next to an email address will email that participant. You can select more than one
email address.

Enter a subject and whatever body text you wish to include. As with all DelphiManager emails, the
greeting of Dear [participant name] and footer of their registration details of Name, Email and
StudyID are included automatically when the email is sent.

You can send a preview email to yourself by clicking the “Send Preview Email” button.

When you are ready to send the bespoke email, click the “Send email to selected list” button. You
should see a page confirming who the email was sent to, similar to that below:

Send Consensus Meeting Invites

This functionality is similar to that of the Bespoke Email but it is only available once the study has
closed. The key difference here is that the Consensus Reports are automatically generated and
attached to the email that is sent. Once clicked, you should see a page similar to the following:

System Administration Guide v5.0 Page 23 of 41

The list of participants displayed is all those who have completed all rounds of your study. To include
them in the invite email, check the box next to the appropriate name. The subject and body text can
be whatever you want. When you click “Send email to selected list” the system creates and attaches
the consensus report for each user before sending the email.

Send Post Study Emails

This is again a variant of the Bespoke Email but allows for an attachment to be sent to a
predetermined groups of users. Similar to the Consensus Invite email, this option is only available
once the study has been completed i.e. after the final round has closed. When you click on “Send
Post Study Emails” you should see a page similar to the following:

System Administration Guide v5.0 Page 24 of 41

You can choose to email either all participants who completed the study or all those do did not.
There is also the option to attach a document to this email. When you are ready to send, click the
“Send email to selected list” button.

System Administration Guide v5.0 Page 25 of 41

Study Structure

The Study Structure section is located by clicking on the appropriate link in the main administrator
menu. The Study Structure page should be similar to the image below:

Manage Outcomes
The overall structure of a DelphiManager is determined during the Setup Wizard process by the
Choose Layout section and the uploading of your outcomes/domains via the template. If during
setup you have cause to amend your outcomes or domains, you will need to use the Reset Study
Setup function which is only available prior to launch. This will delete any existing setup data and
allow you to start the Setup Wizard process again, incorporating any outcome/domain changes via
the Excel template.

DelphiManager is designed so that it is not possible to edit outcomes or domains once they have
been uploaded. To maintain the integrity of the Delphi process, it is important that feedback on an

System Administration Guide v5.0 Page 26 of 41

outcome informs consideration of scoring of that outcome in the next round. We recommend the
outcome naming and descriptions are pilot tested before launch.

However, there may be occurrences where clarity of meaning of an outcome is needed in round two
(typically suggested by comments in round one), and amendments or additions to the outcome
description may be thought to be helpful. If you feel that changing the wording of an outcome
description is required, it can be managed via the DelphiManager Team. We recommend that you
alert participants to anything other than minor changes.

Outcomes can be deleted in between study rounds however and the link to this functionality will
only be available during this time on the Manage Study Structure page. Any outcomes deleted will
not appear in the subsequent study round.

System Administration Guide v5.0 Page 27 of 41

Manage Stakeholders
Click on the “Manage Stakeholders” link from the Study Structure menu and you will see a page
similar to the following:

System Administration Guide v5.0 Page 28 of 41

To edit an existing stakeholder group, click the edit button and you should see a page similar to the
following:

Change the stakeholder text as required then click the “Update” button.

NOTE: This option is only available prior to study launch.

System Administration Guide v5.0 Page 29 of 41

Manage Registration Form
Click on the “Mange Registration Form” button and you should see a page similar to below:

To change a registration form item, click the “[Edit]” button next to the appropriate item. You will
see a page similar to below:

System Administration Guide v5.0 Page 30 of 41

Amend the question text where appropriate then click the “Edit” button.

NOTE: This option is only available prior to study launch.

Manage Logo
To change the DelphiManager logo in the top left corner, click the “Manage Logo” button in the
Study Structure menu. You will see a page similar to below:

System Administration Guide v5.0 Page 31 of 41

The logo file you choose to upload must be either a PNG or JPEG format. As part of the upload
process, it will also be resized to be 294x120 pixels so the closer your logo is to this size, or at least
aspect ratio, the better it will look when uploaded.

Click the “Browse” button, find the image on your computer and then click the “Upload Logo”
button. The top left logo should change immediately.

Manage Rounds
See Study Dashboard – Rounds on page 10.

Manage Pages
To change any of the text on the main DelphiManager pages, click “Manage Pages” on the Study
Structure page. You should see a page similar to below:

System Administration Guide v5.0 Page 32 of 41

Click the “Edit” button next to the page you want to change, ensuring that it is the page for the
round required. You should see a page similar to below:

System Administration Guide v5.0 Page 33 of 41

Change the Page Content appropriate to your study and the page in question. Click the “Update”
button when finished.

Manage Email Templates

There are 7 email templates as part of DelphiManager which are sent at various stages of the survey
process. To amend any of these templates, click the “Manage Email Templates” button on the Study
Structure page. You should see a page similar to below:

System Administration Guide v5.0 Page 34 of 41

To amend the text for a particular template, click “Edit” next to that email type. You should see the
following page:

System Administration Guide v5.0 Page 35 of 41

Both the subject and the body text of the email can be amended. For all emails sent via
DelphiManager, the participant’s main registration details will be included in each email. There will
also be a greeting of “Dear [participant name]” so this does not need to be included in any body text
changes. To see an example of this, you can click the “Send Preview Email” button at any time –
please note this does not save any changes however. Click “Update” to save your changes.

System Administration Guide v5.0 Page 36 of 41

Manage Footer Pages
The footer pages refer to those links found in the blue bar at the bottom of each DelphiManager
page; specifically the About and Contact pages. To change the content for these pages click the
“Manage Footer Pages” button on the Study Structure page. You should see a page similar to below:

Click the “[Edit]” button next to the appropriate page to change the content. You should a page
similar to below:

System Administration Guide v5.0 Page 37 of 41

Amend the copy as required, the Page Headers cannot be changed. Click the “Update” button when
you have finished.

System Administration Guide v5.0 Page 38 of 41

Data Extracts

The Data Extracts section is located by clicking on the appropriate link in the main administrator
menu. The Data Extracts page should be similar to the image below:

All data entered into DelphiManager can be downloaded as a CSV file, via a number of different
options. To download any of the data extracts, just click on the appropriate button. Depending on
your browser, you will be offered the chance to open or download the file to your computer.

Download Ratings
This creates a CSV file of all user ratings from your DelphiManager survey. The ratings are
anonymised by StudyID. This file can be downloaded at any time during a study round. An example
of this file can be downloaded at: [Link]

Note: a rating value of -9 denotes that the outcome in question has not been rated. A rating value
of 10 denotes “Unable to rate”.

Download User Data

This creates a CSV file of all data from your registration form. Also included in this file are any
comments made by each participant at the end of a study round. This file can be downloaded at any

System Administration Guide v5.0 Page 39 of 41

time during a study round. An example of this file can be downloaded at:
[Link]

Download Additional Outcomes

This creates a CSV file of any additional outcomes suggested by participants. This file can be
downloaded at any time during a study round.

Download Round 1 Missing Data

This creates a CSV file of all participants who have not completed round 1 of your survey. It is the
same file as that available in the “Manage Missing Data” page. This file can be downloaded at any
time during a study round. An example of this file can be downloaded at:
[Link]

Not shown in the image above, the following data extracts are also available dependent upon study
structure or time point within a study:

Download Acknowledgement
If selected during setup, this creates a CSV file of the data entered into the Acknowledgement page.

Download OutcomeIDs
If graphs feedback is selected during setup, this creates a CSV file of your outcomes with their
associated OutcomeID. You will need to data so as to be able to name the graph images
appropriately before uploading them for rounds 2/3.

Consensus Reports
This section provides a Word document for selected participants who may be attending the
consensus meeting after your DelphiManager survey has concluded. This option is only available
once the study has closed. Click on the “Create Consensus Reports” button and you should see a
page similar to below:

System Administration Guide v5.0 Page 40 of 41

Only participants who have rated all outcomes for all rounds of your survey will be listed here. The
consensus report details all of the ratings over each round of the survey. Click the icon to
generate the consensus report in Word format for that participant. An example of a consensus
report can be downloaded at: [Link]

System Administration Guide v5.0 Page 41 of 41