[DDI-users] Need feedback from those doing questionnaire development work
Wendy Thomas
wlt at umn.edu
Thu Feb 7 13:08:16 EST 2019
All,
DDI 3.3 adds in the work from the Survey Design group which did most of
their work a number of years ago. In the review of DDI 3.3 it was
recommended that we revise the Questionnaire Development section and
integrate the use of Control Constructs to support a more detailed
description of the development steps. We are coming back to you to obtain
comments on this revision work.
We would specifically appreciate responses regarding whether this meets
your needs. It was based on the needs expresses by a working group composed
of members of GESIS, ISR (Umich), and others from the comparative research
community, but we would like to verify the relevance of the content for the
current DDI community. Please send your comments to me (wlt at umn.edu) and I
will post them to the JIRA issue so others can see them.
Thank you in advance for your feedback.
I have attached both a power point and an example document. Note the
following:
1) We have generalized it from Questionnaire Development to Data Capture
Development
2) The slide deck shows a simplified diagram of the originally proposed set
of classes, comments, an initial revision, more comments, and the current
revision which is the result of creating the example and a final discussion
of that work
3) "Simplified" means that I did not include the standard name, label,
description of all versionables and didn't list cardinality of
relationships.
4) The example provides a blank outline of content followed by a detailed
example, followed by a simplified example for those just wanting to provide
descriptive information on the DeveopmentImplementation using the Current
Population Survey TP66, chapter 6 Design of the Current Population Survey
Instrument
5) Documents and discussion on this section is located at DDILIFE-3640
https://ddi-alliance.atlassian.net/browse/DDILIFE-3640?filter=11208
--
Wendy L. Thomas Phone: +1 612.624.4389
Data Access Core Director Fax: +1 612.626.8375
Minnesota Population Center Email: wlt at umn.edu
University of Minnesota
50 Willey Hall
225 19th Avenue South
Minneapolis, MN 55455
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.icpsr.umich.edu/pipermail/ddi-users/attachments/20190207/1a22e6a0/attachment-0001.html
-------------- next part --------------
CURRENT STRUCTURE OF THE REVISED DATA CAPTURE DEVELOPMENT:
DataCaptureDevelopment:
DevelopmentPlan: (reference)
TypeOfDevelopmentPlan=
Name=
Label=
Description=
DevelopmentObjective=
CostStructure:
Budget:
FundingInformation:
DevelopmentObjectReference=
Contact=Agent
IncludesDevelopmentActivityReference=
DeveopmentActivty: (reference)
TypeOfDevelopmentActivity=
Description=
DesiredOutcome=
ProcessSummary=
RecommendedStaffRequiremnts:
StaffClass=
ParticipantRequirements=
RecruitmentProcess=
AdditionalRequiredResource=
DebriefingProcess:
DevelopmentResult: (inline or by reference)
TypeOfDevelopmentResult=
Name=
Label=
Description=
ResultDate=
ResultsDetail:
ResultsDate=
TypeOfResult=
Description=
RequirementsAssessment=
DevelopmentImplementationReference=
DevelopmentImplementation: (inline or by reference)
PerformedBy=Agent
ActivityDate=
ImplementsDevelopmentActivity=
ControlConstructReference=
DevelopmentStep[ControlConstruct]:
InParameter=
OutParameter=
Binding=
Description=
ExternalAid=
InterviewInstructionReference=
BasedOnDevelopmentResultReference=
Prerequisite=
ConditionForAcceptance=
ActivityDate=
PerformedBy=Agent
DevelopmentObjectReference=
DevelopmentActivityReference=
*************************************
SOURCE: Design and Methodology, Current Population Survey: Technical Paper 66, issued October 2006
Notes: Development content seemed to follow published information well. Few points of needing to determine where the content should go.
DataCaptureDevelopment:
DevelopmentPlan:
TypeOfDevelopmentPlan=Questionnaire
Name=
Label=
Description=
DevelopmentObjective=(1) to better operationalize existing definitions and reduce reliance on volunteered responses; (2) to reduce the potential for response error in the questionnaire-respondent-interviewer interaction and, hence, improve measurement of CPS concepts; (3) to implement minor definitional changes within the labor force classifications; (4) to expand the labor force data available and improve longitudinal measures; and (5) to exploit the capabilities of computer-assisted interviewing for improving data quality and reducing respondent burden (see Copeland and Rothget (1990) for a fuller discussion).
CostStructure:
Budget:
FundingInformation:
DevelopmentObjectReference=CPS Questionnaire
Contact=Agent
IncludesDevelopmentActivityReference=ItemResponseAnalysis
IncludesDevelopmentActivityReference=RespondentDebriefings
IncludesDevelopmentActivityReference=BehaviorCoding
IncludesDevelopmentActivityReference=InterviewerDebriefings
DeveopmentActivty: ItemResponseAnalysis
TypeOfDevelopmentActivity= ItemResponseAnalysis
Description=Statistical test are conducted to ascertain whether differences among the response patterns of different questionnaire versions were statistically significant.
DesiredOutcome=Consistency in item response which may affect the labor force estimates.
ProcessSummary=The statistical tests were adjusted to take into consideration the use of a nonrandom clustered sample, repeated measures over time, and multiple persons in a household. Response distributions were analyzed for all items on the questionnaires. The response distribution analysis indicated the degree to which new measurement process produced different patterns of responses. Data Gathered using the other methods outlined above also aided interpretation of the response differences observed. (Response distributions were calculated on the basis of people who responded to the item, excluding those whose response was "don't know" or "refused.")
RecommendedStaffRequiremnts:
StaffClass(CV)=
ParticipantRequirements=
RecruitmentProcess=
AdditionalRequiredResource= Unedited data from different questionnaires
DebriefingProcess:
DeveopmentActivty: RespondentDebriefings
TypeOfDevelopmentActivity= Respondent Debriefing
Description=At the end of the test interview, respondent debriefing questions were administered to a sample of respondents to measure respondent comprehension and response formulation. From these data, indicators of how respondents interpret and answer the questions and some measures of response accuracy were obtained.
DesiredOutcome=Measure of response accuracy based on variation of respondent comprehension and response formulation
ProcessSummary=The debriefing questions were tailored to the respondent debriefing questions were administered-- probing questions and vignette classification. Question-specific problems were used to ascertain whether certain works, phrases, or concepts were understood by respondents in the manner intended. For example, those who did not indicate in the main survey that they had done any work were asked the direct "LAST WEEK did you do any work at all, even for as little as 1 hour?" An example of the vignettes respondents received is "Last week, Amy spent 20 hours at home doing the accounting for her husband's business. She did not receive a paycheck." Individuals were asked to classify the person in the vignette as working or not working based on the wording of the question they received in the main survey (e.g. "Would you report her as working last week no counting work around the house?" if the respondent received the unrevised questionnaire, or "Would you report her as working for pay or profit last week?" if the respondent received the current, revised questionnaire.
RecommendedStaffRequiremnts: [information on debriefing staff skills]
AdditionalRequiredResource:
DebriefingProcess:
DeveopmentActivty: BehaviorCoding
TypeOfDevelopmentActivity= Behavior Coding
Description=Behavior coding entails monitoring or audiotaping interviews and recording significant interviewer and respondent behaviors (e.g., minor/major changes in question working, probing behavior, inadequate answers, requests for clarification).
DesiredOutcome= Identification for enhancements of interviewer training
DesiredOutcome= Test if revised wording of questionnaire improved the quality of interviewer/respondent interactions
ProcessSummary= During the early stages of testing, behavior coding data are useful in identifying problems with proposed questions. For example, if interviewers frequently reword a question, this may indicate that the question was too difficult to ask as worked; respondents' request for clarification may indicate that they were experiencing comprehension difficulties; and interruptions by respondents may indicate that a question was too lengthy. During later stages of testing, the objective of behavior coding is to determine whether the revised questionnaire improved the quality of interviewer/respondent interactions as measured by accurate reading of the questions and adequate responses by respondents. Additionally, results from behavior coding helped identify areas of the questionnaire that would benefit from enhancements to interviewer training.
RecommendedStaffRequiremnts:
AdditionalRequiredResource:
DebriefingProcess:
DeveopmentActivty: InterviewerDebriefings
TypeOfDevelopmentActivity= Interviewer Debriefing
Description=
DesiredOutcome= Identify areas of the revised questionnaire or interviewer procedures that were problematic for interviewers respondents
DesiredOutcome= Identify questions needing revision
DesiredOutcome= Improve/modify initial interviewer training and interviewer manual
ProcessSummary=Two different techniques were used to debrief interviewers. The first was the use of focus groups at the centralized telephone interviewing facilities and in geographically dispersed regional offices. The focus groups were conducted after interviewers had at least 3 to 4 months experience using the revised CPS instrument. Approximately 8 to 10 interviewers were selected to represent different levels of experience and ability. The second was the use of a self-administered standardized interviewer debriefing questionnaire. Once problematic areas of the revised questionnaire were identified through the focus groups, a standardized debriefing questionnaire was developed and administered to all interviewers.
RecommendedStaffRequiremnts: [information on debriefing staff skills]
AdditionalRequiredResource:
DebriefingProcess:
DevelopmentResult:
TypeOfDevelopmentResult= WordingRefinement
Name= Definition of reference week
Label=
Description= In the revised questionnaire, an introductory statement was added with the reference period clearly stated. The new introductory statement reads, "I am going to ask a few questions about work-related activities LAST WEEK. By last week I mean the week beginning on Sunday, August 9 and ending Saturday, August 15." This statement makes the reference period more explicit to respondents. Additionally, the former Q19 has been deleted from the questionnaire. In the past, Q19 has served as a preamble to the labor force questions, but in the revised questionnaire the survey content is defined in the introductory statement, which also defines the reference week.
ResultDate=
ResultDetail:
ResultsDate=1988
TypeOfResult=ProblemIdentification
Description=In the interviewer debriefings that were conducted in 14 different geographic areas during 1988, interviewers reported that the current quesiton 19 (Q19, major activity question) "What were you doing most of LAST WEEK, working or something else?" was unwieldy and sometimes misunderstood by respondents.
RequirementsAssessment=Identifed general location of problem area
ResultDetail:
ResultsDate=1988
TypeOfResult=ProblemIdentification
Description=Respondents were unsure of what was meant by the time period "last week". In early testing Behavior Coding was useful in identifying problems with proposed questions. Interviewers frequently reworded Q19 or were asked to provide clarification of the term "last week".
RequirementsAssessment=Identified definition of term "last week" to be problematic.
ResultDetail:
ResultDate=1988
TypeOfResult=ProblemIdentification
Description=A respondent debriefing conducted in 1988 found the only 17 percent of respondents had definitions of "last week" that matched the CPS definition of Sunday through Saturday of the reference week. The majority (54 percent) of respondents defined "last week" as Monday through Friday.
RequirementsAssessment=Identified definition of term "last week" to be problematic.
ResultDetail:
ResultDate=1988
TypeOfResult=Verification
Description=Behavior coding used to determine whether the revised questionnaire improved the quality of interviewer/respondent interactions as measured by accureate reading of the questions and adequate responses by respondents.
RequirementsAssessment=Verified that wording change by addition of introductory statement and dropping of Q19 resolved the problem as identified.
DevelopmentImplementation=CPS1988QuestionnaireReview
DevelopmentResult:
TypeOfDevelopmentResult= NewQuestion
Name= PresenceOfFamilyBusiness
Label=
Description= In the former questiontionnaire, there was no direct question on the presence of a business in the household. Such a question is included in the revised questionnaire. This question is asked only once for the entire household prior to the labor force questions. The question reads, "Does anyone in this household have a business or a farm?" This question determins whether a business exists and who in the household owns the business. The primary purpose of this question is to screen for housholds that may have unpaid family workers, not to obtain an stimate of houshold businesses. The term "at work" was revised from "LAST WEEK, did you do any work at all, not counting work around the house?" to read "LAST WEEK did you do ANY work for (either) pay (or profit)?" (The parentheticals in the question are read only when a business or farm is in the household).
ResultDate=1988
ResultDetail:
ResultsDate=1988
TypeOfResult=ProblemIdentification
Description=The definition of employed person includes those who work without pay for at least 15 hours per week in a family business.
RequirementsAssessment=Clarified that not all respondents were counting work without pay in a family business in determining the number of hours worked AND in answering "LAST WEEK, did you do any work at all, not counting work around the house?"
RequirementAssessement=Identified the problem of accurately identifying unpaid family workers
ResultDetail:
ResultsDate=1988
TypeOfResult=ProblemIdentification
Description=Respondents were unsure of what was meant by the time period "last week". In early testing Behavior Coding was useful in identifying problems with proposed questions. Interviewers frequently reworded Q19 or were asked to provide clarification of the term "last week".
RequirementsAssessment=Identified the problem of the definition of "at work".
ResultDetail:
ResultDate=1988
TypeOfResult=Verification
Description=Behavior coding reulted in affirming that the revised wording "work for pay (or profit)" better captures the concept of work that BLS is attempting to measure
DevelopmentImplementation=CPS1988QuestionnaireReview
DevelopmentImplementation:
Name=CPS1988QuestionnaireReview
PerformedBy=Agent
ActivityDate=1988
ImplementsDevelopmentActivity=RespondentDebriefings
ImplementsDevelopmentActivity=InterviewerDebriefings
ImplementsDevelopmentActivity=BehaviorCoding
ImplementsDevelopmentActivity=ItemResponseAnalysis
ControlConstructReference=SequenceMain
Sequence:
ID: SequenceMain
DevelopmentStep=ProblemIdentification
Loop=QuestionRevision
DevelopmentStep
ID: ProblemIdentification
InParameter=
OutParameter=QuestionArray
Binding=
Description=Identify Questions that affect response patterns which may affect the labor force esitmates
ExternalAid=Unedited data
InterviewInstructionReference=
Prerequisite=
ConditionForAcceptance=
ActivityDate=
PerformedBy=Agent
ImplementsDevelopmentActivity=ItemResponseAnalysis
DevelopmentObjectReference=InstrumentCPS1987
Loop:
ID:QuestionRevision
InParameter=ObjectArray
Binding:
Source=QuestionArray
Target=ObjectArray
alias=QUESTION
initialValue=0
stepValue=1
LoopWhile= for i=n; QUESTION[n] > " ";
ControlConstruct=EvaluationSequence
Sequence:
ID: EvaluationSequence
SplitJoin=AnalysisRevision
DevelopmentStep=WordingRevision
DevelopmentStep=RevisionRetest_BehaviorCoding
SplitJoin:
ID:AnalysisRevision
ControlConstruct= ProblemSpec_RespondentDebrief
ControlConstruct= ProblemSpec_InterviewerDebrief
ControlConstruct= ProblemSpec_BehaviorCoding
DevelopmentStep
ID: ProblemSpec_RespondentDebrief
InParameter=
OutParameter=
Binding=
Description=
ExternalAid=
InterviewInstructionReference=
Prerequisite=
ConditionForAcceptance=
ActivityDate=
PerformedBy=Agent
ImplementsDevelopmentActivity=RespondentDebriefings
DevelopmentObjectReference=CPS1987_Q19
DevelopmentStep
ID: ProblemSpec_InterviewerDebrief
InParameter=
OutParameter=
Binding=
Description=
ExternalAid=
InterviewInstructionReference=
Prerequisite=
ConditionForAcceptance=
ActivityDate=
PerformedBy=Agent
ImplementsDevelopmentActivity=InterviewerDebriefings
DevelopmentObjectReference=CPS1987_Q19
DevelopmentStep
ID: ProblemSpec_BehaviorCoding
InParameter=
OutParameter=
Binding=
Description=
ExternalAid=
InterviewInstructionReference=
Prerequisite=
ConditionForAcceptance=
ActivityDate=
PerformedBy=Agent
ImplementsDevelopmentActivity=BehaviorCoding
DevelopmentObjectReference=CPS1987_Q19
DevelopmentStep
ID: WordingRevision
InParameter=
OutParameter=RevisedWordingOUT
Binding=
Description=Develop revised wording based on the results of the problem specification work
ExternalAid=
InterviewInstructionReference=
Prerequisite=Results from problem specification work
ConditionForAcceptance=
ActivityDate=
PerformedBy=Agent
DevelopmentObjectReference=RevisedCPS1988_LastWeek
DevelopmentStep
ID: RevisionTest_BehaviorCoding
InParameter=RevisedWordingIN
OutParameter=
Binding:
Source=RevisedWordingOUT
Target=RevisedWordingIN
Description=
ExternalAid=
InterviewInstructionReference=
Prerequisite=
ConditionForAcceptance=Correction of identified behavior
ActivityDate=
PerformedBy=Agent
ImplementsDevelopmentActivity=BehaviorCoding
DevelopmentObjectReference=RevisedCPS1988_LastWeek
[Note that this main sequence might then go into a set of activities implemented to test a recompiled questionnaire using the revised/new questions and statements.]
*************************************
Simple descriptive version of the above detailed content. Note this retains the ability to describe the activites completed, object(s) of development, results, and the ability to link a data capture object to the development results upon which it is based.
SOURCE: Design and Methodology, Current Population Survey: Technical Paper 66, issued October 2006
Notes: Development content seemed to follow published information well. Few points of needing to determine where the content should go.
DataCaptureDevelopment:
DevelopmentPlan:
TypeOfDevelopmentPlan=Questionnaire
Name=
Label=
Description=
DevelopmentObjective=(1) to better operationalize existing definitions and reduce reliance on volunteered responses; (2) to reduce the potential for response error in the questionnaire-respondent-interviewer interaction and, hence, improve measurement of CPS concepts; (3) to implement minor definitional changes within the labor force classifications; (4) to expand the labor force data available and improve longitudinal measures; and (5) to exploit the capabilities of computer-assisted interviewing for improving data quality and reducing respondent burden (see Copeland and Rothget (1990) for a fuller discussion).
CostStructure:
Budget:
FundingInformation:
DevelopmentObjectReference=CPS Questionnaire
Contact=Agent
IncludesDevelopmentActivityReference=ItemResponseAnalysis
IncludesDevelopmentActivityReference=RespondentDebriefings
IncludesDevelopmentActivityReference=BehaviorCoding
IncludesDevelopmentActivityReference=InterviewerDebriefings
DeveopmentActivty: ItemResponseAnalysis
TypeOfDevelopmentActivity= ItemResponseAnalysis
Description=Statistical test are conducted to ascertain whether differences among the response patterns of different questionnaire versions were statistically significant.
DesiredOutcome=Consistency in item response which may affect the labor force estimates.
ProcessSummary=The statistical tests were adjusted to take into consideration the use of a nonrandom clustered sample, repeated measures over time, and multiple persons in a household. Response distributions were analyzed for all items on the questionnaires. The response distribution analysis indicated the degree to which new measurement process produced different patterns of responses. Data Gathered using the other methods outlined above also aided interpretation of the response differences observed. (Response distributions were calculated on the basis of people who responded to the item, excluding those whose response was "don't know" or "refused.")
RecommendedStaffRequiremnts:
StaffClass(CV)=
ParticipantRequirements=
RecruitmentProcess=
AdditionalRequiredResource= Unedited data from different questionnaires
DebriefingProcess:
DeveopmentActivty: RespondentDebriefings
TypeOfDevelopmentActivity= Respondent Debriefing
Description=At the end of the test interview, respondent debriefing questions were administered to a sample of respondents to measure respondent comprehension and response formulation. From these data, indicators of how respondents interpret and answer the questions and some measures of response accuracy were obtained.
DesiredOutcome=Measure of response accuracy based on variation of respondent comprehension and response formulation
ProcessSummary=The debriefing questions were tailored to the respondent debriefing questions were administered-- probing questions and vignette classification. Question-specific problems were used to ascertain whether certain works, phrases, or concepts were understood by respondents in the manner intended. For example, those who did not indicate in the main survey that they had done any work were asked the direct "LAST WEEK did you do any work at all, even for as little as 1 hour?" An example of the vignettes respondents received is "Last week, Amy spent 20 hours at home doing the accounting for her husband's business. She did not receive a paycheck." Individuals were asked to classify the person in the vignette as working or not working based on the wording of the question they received in the main survey (e.g. "Would you report her as working last week no counting work around the house?" if the respondent received the unrevised questionnaire, or "Would you report her as working for pay or profit last week?" if the respondent received the current, revised questionnaire.
RecommendedStaffRequiremnts: [information on debriefing staff skills]
AdditionalRequiredResource:
DebriefingProcess:
DeveopmentActivty: BehaviorCoding
TypeOfDevelopmentActivity= Behavior Coding
Description=Behavior coding entails monitoring or audiotaping interviews and recording significant interviewer and respondent behaviors (e.g., minor/major changes in question working, probing behavior, inadequate answers, requests for clarification).
DesiredOutcome= Identification for enhancements of interviewer training
DesiredOutcome= Test if revised wording of questionnaire improved the quality of interviewer/respondent interactions
ProcessSummary= During the early stages of testing, behavior coding data are useful in identifying problems with proposed questions. For example, if interviewers frequently reword a question, this may indicate that the question was too difficult to ask as worked; respondents' request for clarification may indicate that they were experiencing comprehension difficulties; and interruptions by respondents may indicate that a question was too lengthy. During later stages of testing, the objective of behavior coding is to determine whether the revised questionnaire improved the quality of interviewer/respondent interactions as measured by accurate reading of the questions and adequate responses by respondents. Additionally, results from behavior coding helped identify areas of the questionnaire that would benefit from enhancements to interviewer training.
RecommendedStaffRequiremnts:
AdditionalRequiredResource:
DebriefingProcess:
DeveopmentActivty: InterviewerDebriefings
TypeOfDevelopmentActivity= Interviewer Debriefing
Description=
DesiredOutcome= Identify areas of the revised questionnaire or interviewer procedures that were problematic for interviewers respondents
DesiredOutcome= Identify questions needing revision
DesiredOutcome= Improve/modify initial interviewer training and interviewer manual
ProcessSummary=Two different techniques were used to debrief interviewers. The first was the use of focus groups at the centralized telephone interviewing facilities and in geographically dispersed regional offices. The focus groups were conducted after interviewers had at least 3 to 4 months experience using the revised CPS instrument. Approximately 8 to 10 interviewers were selected to represent different levels of experience and ability. The second was the use of a self-administered standardized interviewer debriefing questionnaire. Once problematic areas of the revised questionnaire were identified through the focus groups, a standardized debriefing questionnaire was developed and administered to all interviewers.
RecommendedStaffRequiremnts: [information on debriefing staff skills]
AdditionalRequiredResource:
DebriefingProcess:
DevelopmentResult:
TypeOfDevelopmentResult= WordingRefinement
Name= Definition of reference week
Label=
Description= In the revised questionnaire, an introductory statement was added with the reference period clearly stated. The new introductory statement reads, "I am going to ask a few questions about work-related activities LAST WEEK. By last week I mean the week beginning on Sunday, August 9 and ending Saturday, August 15." This statement makes the reference period more explicit to respondents. Additionally, the former Q19 has been deleted from the questionnaire. In the past, Q19 has served as a preamble to the labor force questions, but in the revised questionnaire the survey content is defined in the introductory statement, which also defines the reference week. In the interviewer debriefings that were conducted in 14 different geographic areas during 1988, interviewers reported that the current quesiton 19 (Q19, major activity question) "What were you doing most of LAST WEEK, working or something else?" was unwieldy and sometimes misunderstood by respondents. Respondents were unsure of what was meant by the time period "last week". In early testing Behavior Coding was useful in identifying problems with proposed questions. Interviewers frequently reworded Q19 or were asked to provide clarification of the term "last week". A respondent debriefing conducted in 1988 found the only 17 percent of respondents had definitions of "last week" that matched the CPS definition of Sunday through Saturday of the reference week. The majority (54 percent) of respondents defined "last week" as Monday through Friday. Behavior coding used to determine whether the revised questionnaire improved the quality of interviewer/respondent interactions as measured by accureate reading of the questions and adequate responses by respondents.
ResultDate=
DevelopmentImplementation=CPS1988QuestionnaireReview
DevelopmentResult:
TypeOfDevelopmentResult= NewQuestion
Name= PresenceOfFamilyBusiness
Label=
Description= In the former questiontionnaire, there was no direct question on the presence of a business in the household. Such a question is included in the revised questionnaire. This question is asked only once for the entire household prior to the labor force questions. The question reads, "Does anyone in this household have a business or a farm?" This question determins whether a business exists and who in the household owns the business. The primary purpose of this question is to screen for housholds that may have unpaid family workers, not to obtain an stimate of houshold businesses. The term "at work" was revised from "LAST WEEK, did you do any work at all, not counting work around the house?" to read "LAST WEEK did you do ANY work for (either) pay (or profit)?" (The parentheticals in the question are read only when a business or farm is in the household). The definition of employed person includes those who work without pay for at least 15 hours per week in a family business. Respondents were unsure of what was meant by the time period "last week". In early testing Behavior Coding was useful in identifying problems with proposed questions. Interviewers frequently reworded Q19 or were asked to provide clarification of the term "last week". Behavior coding reulted in affirming that the revised wording "work for pay (or profit)" better captures the concept of work that BLS is attempting to measure
ResultDate=1988
DevelopmentImplementation=CPS1988QuestionnaireReview
DevelopmentImplementation:
Name=CPS1988QuestionnaireReview
Description=Studies employing methods from the cognitive sciences were conducted to test possible solutions to the problems identified. These studies included inteviewer focus groups, respondent focus groups, respondent debriefings, a test of interviewers' knowledge of concepts, in-depthm cognitive laboratory interviews, response categoriation research, and a study of respondents' comprehension of alternative versions of labor force questions. In addition to qualitative research, the revised questionnaire,...used information collected in a large two-phase test of question wording. During Phase I, two alterative quetionnaires were tested using the official questionnaire as the control. During Phase II, one alterative questionnaire was tested with the control.The questionnaires were teted using computer-assisted telephone intervierwing and a random digit dialing sample. During these test, interviews were conducted from the centalized telphone interviewing facilities of the tlephone interviewing facilities of the Census Bureau. Both qualitative and qualitative information was usind in the two phases to select questions, identify problems, and suggest solutions. Anlyses were based on information fromitem respone distributions, respondent and interviewer debriefing data, and behavior coding of interviewer/respondent interactions.
PerformedBy=Agent
ActivityDate=1988
ImplementsDevelopmentActivity=RespondentDebriefings
ImplementsDevelopmentActivity=InterviewerDebriefings
ImplementsDevelopmentActivity=BehaviorCoding
ImplementsDevelopmentActivity=ItemResponseAnalysis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Updated_DataCaptureDevelopment.pptx
Type: application/vnd.openxmlformats-officedocument.presentationml.presentation
Size: 48666 bytes
Desc: not available
Url : http://lists.icpsr.umich.edu/pipermail/ddi-users/attachments/20190207/1a22e6a0/attachment-0001.bin
More information about the DDI-users
mailing list