<div dir="ltr">That could also be a good solution, Wendy.<div>I'm working on some R functions to produce specific setup files based on the DDI XML metadata, and should the information you mention be present in the DDI files I could then instruct the particular software to replace the valid missing values with the appropriate special values <span style="font-family:arial,sans-serif;font-size:13px">for the particular statistical package.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Since no common standard exists for the missing values between various statistical packages, using negative values as a rule of thumb might achieve portability.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Just my 2 cents,</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Adrian</span></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 24, 2014 at 5:50 PM, Wendy Thomas <span dir="ltr"><<a href="mailto:wlt@umn.edu" target="_blank">wlt@umn.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>Given that a) different software has different conventions, b) training people to do the "right thing" is difficult and the definition of "right" varies, and c) we have a lot of legacy data, is the following a fair statement.</div>
<div><br></div><div>In DDI where you declare the file type (PhysicalStructure) there should be an option for mapping previously declared missing (invalid) values to the appropriate value for the particular statistical package (and version if applicable). This seems to be necessary even if someone creates the "right" values within a specific statistical package as many archives/libraries/etc. provide data or set up files in multiple formats. </div>
<div><br></div><div>If so, then I'll enter this in the issue tracker along with the email texts so it doesn't get lost. I will then talk with Therese and figure out the best way to make sure that this issue along with other "future" issues in Mantis get fed into the Moving Forward development process.</div>
<div><br></div><div>Wendy</div></div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Tue, Jun 24, 2014 at 8:56 AM, Hoyle, Larry <span dir="ltr"><<a href="mailto:larryhoyle@ku.edu" target="_blank">larryhoyle@ku.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" vlink="purple" link="blue">
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">In SAS and Stata the values ._ .a - .z are special numeric values, treated as missing, which compare less than the smallest valid value.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">If you use -1, for example, to represent “refused” and compute a mean on the variable the -1 will be included in the computation – not ignored.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Using a scheme like<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">value timetopg <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">1 = '1-2 mos'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">2 = '3-5 mos'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">3 = '6-12 mos'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">4 = ' > 1 yr'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">.r = 'Refused'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">.d = "don't remember"<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">.s = 'set to missing by rule'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">.o = 'other missing'<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Would allow you to compute statistics ignoring the missing values as well as tabulations using the missing values (e.g. computing the % refused).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">In packages like SPSS one can specify that otherwise valid values (like -1 in your example) can be treated as missing. The advantage of using “out of band”
values is that they cannot accidentally be treated as valid values.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">R, I believe, only has two missing values: NA and NaN. In order to prevent treating -1 - -4 as valid values in your example in R you would need to transform
the variable to convert all of these values to NA. If you are moving data from any software that allows multiple missing values SPSS, SAS or Stata to R you may need to use NA as the missing value for all of the categories and perhaps create a secondary variable
preserving the different values of missing. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">--- Larry Hoyle<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-family:"Tahoma","sans-serif";font-size:10pt">From:</span></b><span style="font-family:"Tahoma","sans-serif";font-size:10pt"> <a href="mailto:ddi-users-bounces@icpsr.umich.edu" target="_blank">ddi-users-bounces@icpsr.umich.edu</a> [mailto:<a href="mailto:ddi-users-bounces@icpsr.umich.edu" target="_blank">ddi-users-bounces@icpsr.umich.edu</a>]
<b>On Behalf Of </b>Adrian Du?a<br>
<b>Sent:</b> Tuesday, June 24, 2014 4:03 AM<br>
<b>To:</b> Data Documentation Initiative Users Group<br>
<b>Subject:</b> Re: [DDI-users] DDI-users Digest, Vol 105, Issue 6 (SAS/Stata extended missings)<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Bob,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've never used SAS, but have to ask something regarding these different types of missings.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Is there any particular advantage of .r, .d and .m over something like:<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">value timetopg <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">1 = '1-2 mos'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">2 = '3-5 mos'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">3 = '6-12 mos'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">4 = ' > 1 yr'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">-1 = 'Refused'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">-2 = "don't remember"<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">-3 = 'set to missing by rule'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">-4 = 'other missing'<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif";font-size:10pt">;<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm thinking about cross portability of these codes, and the above suggestion would work (I think) in every statistical package while .d and .r etc are specific for SAS only.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Adrian<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Jun 23, 2014 at 7:49 PM, Bob McConnaughey <<a href="mailto:bobmcconn@gmail.com" target="_blank">bobmcconn@gmail.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">i suspect i'm belaboring the obvious here, but here's how SAS treats numeric missings<u></u><u></u></p>
</div>
<p class="MsoNormal">SAS numeric missings appear to be "character strings" - but they are treated, within SAS (and Stata i believe) as "invented" numbers, smaller than the "smallest" negative number. eg -1*10**10000 > .z > .a > . > ._ ; (though i don't think
i've ever seen "._" used). However their great virtues are: 1. As "known" missings they automatically get excluded from computations involving the variable they represent. And, like any other value (character or numeric) the can be described using formats..
That is when you do, say, a frequency proc and assign formats to the missing you'd see something like:<u></u><u></u></p>
<div>
<p class="MsoNormal">time_to_pregnancy1<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">value timetopg <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 1-2 = '1-2 mos'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 3-5 = '3-5 mos'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 6-12 = '6-12 mos'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> 13-high = ' > 1 yr'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> .r = 'Refused'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> .d = "don't remember"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> .m = 'set to missing by rule'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">. = 'other missing'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Value labels are the equivalent SPSS feature (i think..i haven't used SPSS in 35 yrs) and even now most of our original questionnaires use "out of range" numbers for special missing values. But the number of times post-docs and researchers
have come up with funky basic descriptive statistics because, oh, "99" was used for a missing value for "height_inches" is well nigh uncountable. And matters are getting worse because there's a general tendency to not use codebooks any more; instead projects
rely on "annotated questionnaires" and SAS "proc contents" I am very much hoping to get people here to go back to using codebooks and the various DDI products SHOULD be convincing. (well, convincing for people other than the small group of reproductive epidemiology
researchers I work with most closely).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">thanks for the responses!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="background:white;color:rgb(34,34,34);font-family:"Arial","sans-serif";font-size:9.5pt">Bob McC....</span><span style="color:rgb(34,34,34);font-family:"Arial","sans-serif";font-size:9.5pt"><br>
<br>
<span style="background:white">"At times like this, an adult needs a drink."</span><br>
<span style="background:white">Dance, Dance, Dance. H. Murakami</span></span> <u></u>
<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
_______________________________________________<br>
DDI-users mailing list<br>
<a href="mailto:DDI-users@icpsr.umich.edu" target="_blank">DDI-users@icpsr.umich.edu</a><br>
<a href="http://lists.icpsr.umich.edu/mailman/listinfo/ddi-users" target="_blank">http://lists.icpsr.umich.edu/mailman/listinfo/ddi-users</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">Adrian Dusa<br>
University of Bucharest<br>
Romanian Social Data Archive<br>
1, Schitu Magureanu Bd.<br>
050025 Bucharest sector 5<br>
Romania<br>
Tel.:<a href="tel:%2B40%2021%203126618" value="+40213126618" target="_blank">+40 21 3126618</a> \<br>
<a href="tel:%2B40%2021%203120210" value="+40213120210" target="_blank">+40 21 3120210</a> / int.101<br>
Fax: <a href="tel:%2B40%2021%203158391" value="+40213158391" target="_blank">+40 21 3158391</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
<br>_______________________________________________<br>
DDI-users mailing list<br>
<a href="mailto:DDI-users@icpsr.umich.edu" target="_blank">DDI-users@icpsr.umich.edu</a><br>
<a href="http://lists.icpsr.umich.edu/mailman/listinfo/ddi-users" target="_blank">http://lists.icpsr.umich.edu/mailman/listinfo/ddi-users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br></div></div><div>Wendy L. Thomas Phone: <a href="tel:%2B1%20612.624.4389" value="+16126244389" target="_blank">+1 612.624.4389</a></div><div>
Data Access Core Director Fax: <a href="tel:%2B1%20612.626.8375" value="+16126268375" target="_blank">+1 612.626.8375</a></div><div>Minnesota Population Center Email: <a href="mailto:wlt@umn.edu" target="_blank">wlt@umn.edu</a></div>
<div>University of Minnesota</div><div>50 Willey Hall</div><div>225 19th Avenue South</div><div>Minneapolis, MN 55455</div>
</div>
<br>_______________________________________________<br>
DDI-users mailing list<br>
<a href="mailto:DDI-users@icpsr.umich.edu">DDI-users@icpsr.umich.edu</a><br>
<a href="http://lists.icpsr.umich.edu/mailman/listinfo/ddi-users" target="_blank">http://lists.icpsr.umich.edu/mailman/listinfo/ddi-users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Adrian Dusa<br>University of Bucharest<br>Romanian Social Data Archive<br>1, Schitu Magureanu Bd.<br>050025 Bucharest sector 5<br>Romania<br>
Tel.:+40 21 3126618 \<br> +40 21 3120210 / int.101<br>Fax: +40 21 3158391</div>
</div>