Test with users

When to do it

Ideally, sites are tested in the draft stage before so much time is invested that you are reluctant to make major changes. You can test the original draft by showing it to students on a Word document without the graphics. The site should definitely be tested once the html pages have been completed. If there are major revisions, you may need one last round of testing (this is rare).

Whom to use

Choose about 7 users who represent your audience. The ideal is to get a spectrum of users from inexperienced to experienced. If you are testing with students, try to get different age levels. I also try for a mix of diversity.

This document assumes students as users but you could be using faculty, staff, parents, community members, etc.

Use different students for different tests.

It works best if there is a reward such a gift certificate. Remember to get ID numbers to give to accounting if you purchase certificates.

Students should not be your employees or anyone over whom you have hiring or salary influence. You want an honest opinion.

How to conduct testing

  • One person should communicate with the student
  • One person should take notes
  • There should not be more than three or four people in the room; it can make the student nervous
  • The screen should be where all can see it
  • If you videotape the session, make sure the student gives permission
  • The student should be made very comfortable
  • Make sure you have explained your mouse if it’s not the basic kind
  • Make clear that this is not a test of the student’s intelligence
  • Encourage honest feedback, even if it is negative. Especially if it is negative

Ask the student to think aloud as he or she is navigating. Often that is not easy for the student and you may have to ask questions about why they chose a certain link, etc.

Bring up the page on the screen only after everyone is settled and ready to review.

What to test

Some students want to focus on pictures. That’s okay to a certain extent but realize you will that imagery is very subjective and you have a wide range of opinions. Ask the overall impression of the page when it first comes up. Answers may include words such as cluttered, attractive, boring, friendly. This is usually a short conversation.

Navigation

Have a set of prepared questions asking the student to find certain critical items, such as major, health insurance waiver, financial aid FAFSA form, etc.  Watch as they try to navigate, you can ask questions about why they chose certain links, especially if they chose the wrong one. Ask them what they expected to find under a link if they have chosen the wrong one.

Usually you can ask a maximum of 10 find-the-information questions.

Then allow the student to find three or four pieces of information he or she would want to find. If they can only think of one, don’t force the issue. This is where you will find out if you have included the essential items. You can also ask the question directly – “What else should be included in this site?”

Terms

This is related to navigation but warrants a separate focus. Ask the student what certain terms mean to them; focus on the ones you suspect are more obscure.  Examples can include general information, student services, leadership, etc.

After these two testing topics, ask for additional comments, often the student will add more valuable information.

Ask the student to summarize his or her opinion of the site. Give them time to think about the answer, it’s at this point they will be feeling more comfortable with what the site is about.

The testing should not take much more than an hour. Occasionally you will get a student who is either reluctant to say anything but positive things, has no computer literacy or who wants to redesign the site for you. There is little information to be gained from those tests so make them short and move on to more productive users.

Useful tips

  • Do not explain why you used a certain word, design, etc, just listen to the feedback
  • Do not get defensive
  • Be patient and neutral in your approach
  • Do not help the student find information; just watch how they do it. Use words and concepts they would use such as “how would you reach your math teacher if you had a question” rather than “find faculty email”
  • If a student is totally stuck, bail them out and make sure they don’t feel stupid for not finding something

Thank them profusely.