Introduction

What is Concept Testing?

Concept testing is the process of sharing creative concepts with the intended audience to get their feedback and identify the best idea before designing materials (See the How to Develop a Creative Concept guide). Results from creative concept testing help the creative team to revise concepts, drop ones that do not resonate well with the audience and identify the ones the audience likes best. Sometimes none of the concepts is appropriate for the audience. In these cases, the creative team returns to brainstorming, keeping in mind lessons learned from the concept tests.

Keep in mind that for social and behavior change communication (SBCC) campaigns and materials to be most effective, they should be tested at several stages of development. In the SBCC process, four types of testing are typically conducted: concept testing, stakeholder reviews, pretesting and field testing. The graphic below demonstrates the relationship between the four types of testing. This guide covers concept testing.

The purpose of creative concept testing differs from that of materials pretesting. With creative concept testing, the intention is to seek audience input into an overall creative concept or idea. Materials pretesting, on the other hand, seeks to assess audiences’ reactions to creative materials before they are finalized. Since the purpose of concept testing is not to pretest the actual copy and visuals of a creative material, concept test materials need not include final copy or visuals. Concept tests only need to communicate the creative idea.

Why Test Creative Concepts?

Testing creative concepts helps identify the best way to meet campaign objectives. Understanding what works—or does not work—saves time and money. It helps ensure that communication materials are developed based on an effective creative concept that connects with the intended audience. Concept testing allows teams to evaluate big ideas side-by-side and select from among alternative versions. Testing also reveals:

• Confusing words, phrases or ideas that should be clarified
• Preferred language used by the intended audience
• Preferred visual styles
• Weak concepts that should be cut
• New ideas

Who Should Conduct Concept Testing?

Ideally, researchers trained in concept testing methodologies – either an external research firm or internal research staff – should conduct concept testing. Sometimes, the hired creative agency conducts the concept tests. It is important that whoever conducts the testing be objective and able to ask questions without biasing or leading participants to a certain decision. The campaign implementation team, including stakeholders, can observe the testing or read through transcripts of interviews and focus group discussions to understand audience reactions and suggestions.

When Should Concept Testing be Conducted?

Concept testing should be conducted after a few viable creative concepts have been developed and before materials development begins.

Learning Objectives

After completing the activities in the concept testing guide, the reader will:

• Select the most appropriate study participants and method for concept testing, based on available resources, timing and objectives
• Prepare tools for the concept test
• Refine a creative concept based on what has been learned during the testing process

Estimated Time Needed

Concept testing typically takes between one to two months depending on the testing method used and the number of revisions necessary. If creative concepts require a complete rework, it could take longer.

Prerequisites

• Creative Brief
• Creative Concept
• Audience Analysis

Steps

Step 1: Determine Which Elements to Test

After developing a few creative concepts that support the creative brief and the audience analysis, determine which of the following elements are most critical to test:

• Attention. Does the idea attract audience attention? This is often measured as a person’s ability to remember an idea, message or image.
• Comprehension. Is the idea clearly understood?
• Motivation. Does the idea inspire the audience to take a desired action?
• Personal relevance. Can the audience connect with the idea? Does it take their point of view into consideration?
• Cultural appropriateness. Is the idea consistent with the values, attitudes, beliefs, traditions and history shared by the intended audience?

Step 2: Identify Testing Audience

The best way to get a sense of how the audience will react to a concept is to test it with them. Review the audience profile created for the primary and influencing audiences. These descriptions can help determine where to find audience representatives, as well as the methods that will be used for concept testing. Some examples include recruiting pregnant women from antenatal care clinics, female sex workers from known hot spots, like night clubs, or mobile men from truck stops.

Based on the audience profiles, create a list of characteristics that concept test participants must have in order to participate. These will become part of the screening questionnaire in Step 5 that establishes criteria for participation. Consider demographic (age, sex, relationship status, number of children), psychographic (needs approval from friends, values intimacy), behavioral (never used family planning, do not want more children) and geographic (rural, urban, density) characteristics.

Also consider getting feedback from people who influence the intended audience, such as health care providers, teachers, religious leaders or spouses. Testing concepts with these influencers is a good way to learn about cultural appropriateness and to gauge their support for the campaign.

Step 3: Select the Best Testing Method

Decide which method(s) will be used to test the concepts: focus group discussions, in-depth interviews, intercept interviews or self-administered questionnaires. The method selected will depend on the depth of feedback required, characteristics of the audience representatives, time and budget.

Test Method	Sample Size	Appropriate For
Focus Group Discussions (FGDs)A discussion on the concept guided by an experienced moderator. Can be conducted online or face-to-face.	Six to ten people; Usually several FGDs/ concept.	Good for generating new ideas and gauging emotional impact. Work best when it is fairly easy to bring together a group of people.
In-Depth InterviewsOne-on-one discussion between an interviewer and a participant. Can be conducted in person or over the phone.	At least ten	Work best when testing concepts among people from several different geographic locations, or when the creative content is sensitive, such as reproductive and sexual health, making the audience feel uneasy discussing it in a group.
Intercept InterviewsTrained interviewer shows concept materials to the intended audiece in a place they frequently visit (markets, clinics) and conducts a quick survey with them.	60-300	Work best when many audience representatives frequent the intercept location, when testing for one element, such as attention or cultural relevance, across alternative concepts. They help reduce recruiting costs.
Self-Administered QuestionnairesQuestionnaires of mostly close-ended questions are completed by respondents. Can be done in-person, via email or mobile, or online.	20-200	Good for large numbers of respondents with a short timeframe. Work well when testing creative concepts for websites, mobile applications or communication campaigns intended for audiences who come from diverse geographic locations, such as from across countries or regions of one country.

Step 4: Develop Testing Methods

Once the testing method(s) have been chosen, the team will need to develop mockups or visual presentations of each alternative creative concept that can be shared during concept testing. How these materials are presented will depend on the method of testing, the nature of each concept and the characteristics of the people who will participate in the testing exercise. Often, the best way to present creative concepts is through a visual presentation that includes samples of various materials that could be used. This helps to minimize the effort and costs of preparing full sets of campaign materials for each creative concept. Presentations can be projected electronically, shared online and as poster boards, or printed as handouts. It is important that mockups of each creative concept are comparable in quality to minimize bias.

When using intercept interviews, it might be best to present each concept via a poster to minimize on time and eliminate the need for technology. When using in-depth interviews or focus group discussions, an electronic presentation, such as Powerpoint® or Prezi, might work well since they are more interactive and there is sufficient time to explore. When pretesting concepts for a video or film, storyboards work well, since they visually show a sequence of events. When testing concepts for a multi-channel campaign, an electronic presentation allows audience representatives to see mock-ups of a variety of media. On the other hand, when testing concepts for radio or TV spots among rural residents, where electric power is not readily available, dramatic readings work better than electronic presentations.

Several creative concepts were developed and tested for the Safe Male Circumcision Campaign Uganda. A Powerpoint® presentation was printed as poster boards and shared during focus group discussions with uncircumcised men to test the creative concept for a national campaign promoting voluntary medical male circumcision in Uganda. See the Samples section for the full presentation.

Step 5: Prepare Testing Tools

Develop the tools that will be used during concept testing. The tools needed depend on the testing methods to be used. Four types of tools may be required: recruitment screener, consent form, moderator’s guides and questionnaires.

• Recruitment Screener. A short questionnaire that is administered to potential participants—either in-person, online or by telephone—to ensure that they meet the criteria for participating in the concept testing, as described in Step Two. It should be used to guide the selection of testing participants, regardless of testing method used (see the Samples section for sample Screeners).

• Consent Form. A brief form that describes the research and what a participant should expect, including the risks involved, and allows a participants to voluntarily consent to participate. The form can be read to participants if they are not literate (see the Samples section for sample form).

• Moderator’s Guide. An important tool for focus groups and in-depth interviews, the moderator’s guide serves as an outline to help keep the moderator on track during discussions. The guide contains topics, questions and activities to spark discussion. Normally, when testing more than one creative concept, testing involves sharing each concept individually, followed by a few questions. After all alternative creative concepts have been shared and discussed, the moderator asks participants which of the concepts they prefer and why. In developing the guide, determine the key questions to ask and probes for obtaining answers in a conversational manner. A typical guide contains the following (see the Samples section for examples of moderator’s guides):
  • Objectives and elements to be tested
  • Brief respondent profile, including the number of interviews or discussions to complete
  • Where and when the interviews are to take place
  • An introduction script to be read to participants
  • Questions and activities
  • Closing script to be read to participants

• Questionnaire. A standard set of questions and response options that may be completed by the respondents themselves or by interviewers (see the Samples section for sample questionnaires). In general, questionnaire questions should:
  • Contain only one idea or question
  • Define the scope to consider, such as the time period or activities that are relevant to the question
  • Be written with neutral language to avoid leading the respondent to a specific answer
  • Use language that enables respondents to easily understand the question
  • Contain response options that are simple, clear, consistent and include the full range of responses that might occur
  • Provide mutually exclusive response options so that a respondent can pick only one option

Here are some sample questions that can be asked after showing each individual concept. These questions are designed to test concepts for each of the five elements listed in Step One.

Testing Element	Recommendation	Sample Question
Attention	Provide pre-coded questions	Have you seen similar concepts/advertisements? Compared to other advertising you have seen, would say this concept is very memorable, memorable or not so memorable?
Comprehension	These are open-ended questions that will be coded during analysis	What is the first thing that comes to your mind when you see this concept/poster/presentation? What is this concept/poster/presentation telling you? (Record the time required to understand the main message.)
Motivation	Provide pre-coded questions	After seeing this concept/poster/presentation, how likely are you to adopt the practice/use the service or product?
Personal Relevance	Provide pre-coded questions	Would you say that you completely believe/partially believe/do not believe what this presentation/poster is telling you? Is this concept made for people like you or someone else? If someone else, what sort of person is this concept for? (Provide alternative characteristics, such as older/younger, more/less wealthy, more/less educated.
Cultural Appropriateness	Use pre-coded questions to gauge degree of appropriateness; use open ended questions to probe how a concept is inappropriate	How well does this concept reflect the values and traditions of most people in your culture, such as very well, to some extent, not at all? Would you or others in your community think there is anything offensive about this concept? If so, what?
Comparing/rating concepts	Ask after all concepts have been shared and tested	Please rank these concepts in order of preference, from most preferred to least preferred.

Step 6: Conduct the Testing

Proceed with testing as detailed in the moderator’s guide or survey instrument. At the start of each focus group, interview or survey, be sure to welcome the participants and provide a brief overview on the purpose of the testing. Reinforce that there are no “right” or “wrong” answers, and encourage everyone to provide their honest views. Make sure to obtain permission from participants if planning to video or audio record the sessions. Take notes and/or record the feedback on an audio or video recorder, as appropriate.

Step 7: Analyze Testing Outcomes and Summarize Results

Review all notes and recordings from the testing exercise and write a report outlining the process and the findings. The report should have the following sections:

• Background. What was tested? What were the objectives of testing? Who participated in the testing? How was the testing done?
• Highlights. Summarize the main points that came up during the testing.
• Findings. Present a complete report on the findings. Where appropriate, describe the participants’ reactions, incorporate key quotes, describe which creative ideas and concepts worked the best versus those that were not appealing or effective, and share suggestions given by participants.
• Conclusions. Describe the key learnings that came up and/or the major differences that were observed across individuals and/or groups.
• Recommendations. Suggest and prioritize revisions for the tested creative concepts based on the findings and conclusions.

Step 8: Apply Testing Results

Share the testing report with the campaign implementation team and apply the recommendations to sharpen and hone in on one creative concept. Use the research to inform decisions and help develop improved concepts and creative materials. Sometimes the testing will identify one creative concept that is clearly preferable and the results can be used to improve that concept. In other cases, there will be no clearly preferred creative concept, and the results can be used to inform new concepts.

Be aware of the limitations of testing. Keep these things in mind to avoid using the results incorrectly.

A test is only as objective or fair as the person designing and conducting the research and interpreting the results. Testing cannot take the place of experienced judgment. Rather, it can provide additional information to help make good decisions. Testing results should not be overgeneralized. For example, the opinions of two or three people in a focus group of women ages 25-35 cannot be applied to the entire population of women in this age group. These are just a few perspectives to help see how concepts or materials might be received by some. Although testing is very important, it does not guarantee that a creative idea or product will succeed in reaching or appealing to an intended audience. Once developed, continually assess whether the campaign is meeting its objectives. If not, changes might be needed.

Templates

Informed Consent Form Template for Qualitative Studies

Samples

Sample Focus Group Screener Form

Sample Focus Group Moderator’s Guide

The CDC’s VERB[TM] Campaign

Concept Testing Discussion Guide

CP Phase 2 Concept Testing

Safe Male Circumcision Communication Campaign Logo and Concept Test Report

Concept-­Testing Research Findings for CP Tulizana Campaign Phase 2

Focus Group Participant Recruitment Screener

Intercept Questionnaire

External Resources

https://www.surveymonkey.com/mp/concept-testing/

Tips & Recommendations

• Be sure to test the concept test tools/resources developed for each method before using them with the audience
• Emphasize at the beginning of the concept test that these are only rough drafts/prototypes, and that they should evaluate based on the big idea, not the specifics. Colors, fonts, models and details will be further refined at a later point.
• Do not try to merge concepts following the concept test, picking the most well liked pieces from different concepts and combining them into one. This dilutes and confuses the entire campaign.

Lessons Learned

• When asked to compare creative concepts, respondents sometimes get distracted by the way the concepts are presented, and rate a concept based on the quality of artwork, color schemes or typeface used. A way around this is to test each creative concept on its own merits and not ask respondents to rate concepts against each other. Being consistent with the quality of visual materials also helps. Moderators also need to be well trained in how to steer the conversation away from this level of feedback and focus it on the bigger picture.
• Sometimes, the order in which creative concepts are introduced creates bias. This sort of bias can be reduced by varying the order in which the concepts are presented in each interview or focus group discussion.
• Concept testing is useful not only to test creative concepts for a campaign, but also to get audience feedback on design features of communication products. For example, when designing websites, it is useful to allow prospective users to interact with the site during the design stage to test the layout, features and look. Testing design features can save time and money over the long run by preventing costly redesign.
• Often, the differences between creative concepts is so slight that audience representatives cannot distinguish them. When testing more than one creative concept at a time, it is important that each concept is unique from one another and there is no ambiguity.
• The location where focus group discussions or in-depth interviews take place can influence results. Focus groups and interviews that take place in an office often yield different results from those that take place in communities where the respondents are from.
• Too many concepts produce concept fatigue in the respondent. The maximum number of concepts to test in a focus group or interview is three.

Glossary & Concepts

• Audience analysis: A description of the defining characteristics of the people who are targeted by a health communication campaign or product.
• Concept testing: seeks feedback about general ideas, concepts and creative concepts; typically done before materials are developed.
• Creative brief: A short document outlining the audience, objective, promise, key message content and the communication channels that will be used; a succinct account of a campaign strategy.
• Creative concept: The overarching artistic theme that ties together all elements of an SBCC campaign. It is an imaginative plan for capturing the audience’s heart and mind.
• Field testing: allows practitioners to observe whether the SBCC materials are used effectively in their intended settings and contexts, usually through observation and focus group discussions. It determines whether the material meets the intended purpose.
• Pretesting: is the process of bringing together members of the priority audience to react to the components of a communication campaign before they are produced in final form.
• Stakeholder reviews: are input from technical experts, partners and decision-makers prior to finalizing materials. These reviews do not replace pretesting with the priority audience and can be done before or after pretesting.
• Storyboard:
• Dramatic reading: is a public reading or recitation of a work of literature with an interpretative or dramatic use of the voice and often of gestures.
• Pre-coded questions: are a type of question that is asked by interviewers as though they are open-ended questions, but they have pre-coded responses that interviewers use to match (code) respondents’ answers, rather than copy down the verbatim response.

Resources and References

Resources

Making Health Communication Programs Work

A Step by Step Guide for Focus Group Research

Conducting Focus Group Interviews

Developing and Testing Creative Concepts

Six Concept Testing Strategies

References

• Making Health Communication Programs Work: A Planner’s Guide. December 2001. Washington, DC: U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health. Office of Cancer Communications, National Cancer Institute. NIH Publication No. 02-5145. Pp 53 – 89. www.cancer.gov/pinkbook
• Concept Testing in Qualitative Market Research, http://www.focusgrouptips.com/concept-testing.html

---

Banner photo: © 2005 Jennifer Orkis, Courtesy of Photoshare