[DDI-users] A "home" for the DDI

[Mark R. Diggory]

I-Lin Kuo wrote:
> Comments inline
> At 02:21 PM 8/13/03 -0400, you wrote:
>> Its only read access to the content itself that is exposed to the
>> public. This transparency has its value, the users of the DDI may not
>> necessarily be its developers or even on its committee. Exposing this
>> process to the public provides room for comment from such users "prior"
>> to a versioning release (not after). Relating positively to user needs
>> is a critical aspect of producing a spec that the community will
>> continue to want to use.
> 
> I'm not of the same opinion that transparency is good in and of itself. 
> Transparency will open up the DDI to more suggestions, and it seems to 
> me the council already has its hands full with all the suggestions it 
> currently receives. It's a lot of work to process these suggestions, 
> separating the wheat from the chaff. I also have problems with releasing 
> immature work-in-progress for public comment, which is what this kind of 
> transparency would seem to do.
> 

One point I can make is that an issue tracking system would reduce the 
amount of "replicate" requests the council may possibly have to deal 
with. Retaining historical state of issues addressed in the past 
provides a searchable source for those who may be tempted to post a new 
issue. In fact, your process outlined below would benefit significantly 
if managed in a issue tracking system.

I wish I could locate a private CVS service that was managed by someone 
else. I currently maintain my own secure cvs service on my workstation 
for several of my own projects, but I don't feel such a location could 
be considered "stable". A good discussion might be "where to setup a 
private CVS server?". I think the HMDC group might have an interest in 
providing a private CVS here at HMDC unless this is something that ICPSR 
would like to provide locally.

> The goal of "relating positively to user needs" is one I agree with, but 
> I just don't think that the DDI governance structure is set up to be 
> able to do this, with a committee that meets only 2 or 3 times a year 
> making all the major and minor decisions.
> 
> However, the suggestion for "comment prior to a versioning release" is 
> one I heartily agree with as part of an overhaul in the 
> versioning/change process. I'd prefer a more structured process than the 
> informal one currently in place, something like
> 
>         --------
>         PUBLIC COMMENT PHASE: VERSION 2.1
>                 Announce to DDI-Users list that suggestions are 
> currently being accepted for 2.1
>                 ACTIVELY solicit feature requests and suggestions for 
> change
>                 Close: Announce to DDI-Users that no more feature 
> requests accepted for 2.1
>         --------
>         SUGGESTION EVALUATION PHASE
>                 Begin an official thread for each request in the 
> previous phase.
>                 Contact original authors of suggestions and invite them 
> to participate.
>                 Discuss each thread using the forum software rather than 
> private emails so the entire discussion is viewable and archived
>                 At some point, decide to create a formal written 
> proposal out of each thread. These threads will become the basis of the 
> next council meeting
>         --------
>         COUNCIL MEETING:
>                 Discuss each thread
>                 Decide a course of action for each thread. Similar to 
> the process of peer-reviewed journals, possible actions are: Approve, 
> Reject, or Revise & Resubmit
>                 All approved threads are sent to implementers
>         ---------
>         IMPLEMENTATION PHASE:
>                 Core group of developers decide on best way to implement 
> council directives
>         ---------
>         PUBLIC COMMENT PHASE: VERSION 2.1 BETA RELEASE
>                 Freeze changes on DTD/Schema
>                 Announce to DDI-Users availability of beta version of 
> 2.1 for comment
>                 Test and revise based on comments. Small 
> revisions/corrections made by implementers. Major revisions must go back 
> to council (and will push back date of official release)
>         ---------
>         VERSION 2.1 OFFICIAL RELEASE
> 
> I envision that this whole process takes nearly 2 council meeting 
> intervals, so probably about 10-11 months for suggested changes to get 
> into the next version. Because of this, I would also have overlapping 
> processes, with the PUBLIC COMMENT PHASE for VERSION 2.2 to begin at 
> roughly the same time as the IMPLEMENTATION PHASE for VERSION 2.1
> 
I like this idea too.

In open source software development, much of this process would be 
managed in a bugzilla or issue tracking system. It would appear there 
are different timelines in the project for dealing with "bugs" vs 
dealing with "feature requests". Obvious bugs require fixing almost 
immediately in our case without much council involvement. The recent 
codeBook "version" attribute problems that arose in the dtd and schemas 
can attest to this necessity.

> I'm afraid I've gone a bit afield in my reply, but the essence of it is 
> that I think that there is indeed a process problem within the DDI, but 
> it's not one that I think can be solved by exposing the whole process 
> via SourceForge.
> 

I do understand your position on this, I would say as well that exposing 
the whole process on sourceforge would not be a "solution" in and of 
itself.  Certainly how the projects issue management and feature request 
process is managed is also critical to it's organization and progress. 
Such systems can help to separate the "wheat from the chaff" long before 
you reach a council meeting.

>>>>
>>
>> An interesting idea. I expect that the any development in the area of
>> XSD, that the XSD technical implementation would currently be required
>> to maintain a certain amount of "backward compatibility" in reference to
>> validation against the DTD implementation for that Version of the DDI.
>>
>> But in reality, there are technical limitations to the above idea:
>>
>> 1.) More "specific" control of content where it is appropriate. This can
>> maintain "backward compatibility". A XML document validated against an
>> XSD would also be valid against a DTD. But not necessarily the other
>> way around.
> 
> 
> I would make a stronger statement. Any document validated against a 
> previous DTD should have no expectations of being valid against an XSD. 
> When and if the DDI moves to an XSD schema, there must be a resolve to 
> make a clean break with DTDs. In my opinion, faced with the choice 
> having an XSD with the albatross of "backwards compatibility", or having 
> no XSD at all, I would prefer to remain with DTDs. To  me, there is no 
> point in having an XSD that simply duplicates the functionality of the 
> DTD. There are things which can be done to ease the migration, such as 
> providing an XSLT stylesheet to check for potential problems, but I 
> think "backwards compatibility" has to be sacrificed because this is a 
> significant change.
> 

I agree, my sentimate are that dropping the DTD technical implementation 
would be required to taking advantage of many of the capabilities of 
XSD. Its just a question of how dramatic a "transition" would be 
required to achieve that goal.

Currently, the provided XSD's validate DDI documents that are not valid 
against the DTD. Simply because of the technical requirements in XSD of 
the existence of the schema/namespace attributes in the root element of 
the DDI. The DTD could be adjusted to support such attributes. Then any 
DDI produced against the XSD would also be valid against the DTD. I 
addressed this in the past with Matt and Mary.

>> 2.) Embedding the DDI into other xml content and embedding other content
>> into the DDI (such as OAI). This would not be able to maintain backward
>> compatibility because DTD just doesn't support any thing near this
>> capability.
> 
> 
> This is something that requires major structural changes which would 
> again necessitate the abandonment of backwards compatibility. Despite 
> the X in XML, the DDI currently is not designed for extensibility. I'm 
> not familiar with OAI schema, so I wouldn't know if they're designed for 
> it. But if this were a council goal, then I'd suggest that XSD change 
> and OAI embedding change be implemented simultaneously because these 
> changes cannot be implemented without breaking backward compatibility.
> 
Again, this embedding is just a characteristic capability of XSD, its 
just the capability of an XSD "Container" element, that you can place 
any other content into that container element ( in OAI's approach, they 
require it to carry a schema that can be validated against). From our 
experience, there is no major modification to the contents XSD (the DDI 
in this case) that needs occur to make it "embeddable" in OAI.

I think the issue of actually introducing this same container-like 
functionality into the DDI, for example, with the DDI containing MARCXML 
markup or Dublin Core XML markup, that would totally break any 
compatibility. It would be impossible to maintain a comparable DTD if 
such features were introduced. I'm sure there are other feature that I'm 
not strongly familiar with that would have such requirements as well.

thanks,
Mark