[DDI-users] A "home" for the DDI

[I-Lin Kuo]

Comments inline

At 02:21 PM 8/13/03 -0400, you wrote:

>>This would, I think, be an appropriate use of CVS, because you have 
>>multiple people working on the same set of documents. All right, so 
>>you've convinced me.
>>Now, if you were just to set up a private CVS, I don't think there would 
>>be any objection to it. But as you say, it would be far easier to set 
>>this up via SourceForge. However, whether this process should be placed 
>>on SourceForge with an open license is something that would require the 
>>blessing of the DDI Council. I'd certainly like to see this made an item 
>>on the next Council agenda.
>
>Does it necessarily need to be a private facility? In terms of
>sourceforge, the ability to "modify" contents of the cvs are controlled
>by the admin/development team and this control is very private and
>secure. One would setup a core group of administrators to manage the
>project and control the access rights for the developers who are doing
>the modifications.

>Its only read access to the content itself that is exposed to the
>public. This transparency has its value, the users of the DDI may not
>necessarily be its developers or even on its committee. Exposing this
>process to the public provides room for comment from such users "prior"
>to a versioning release (not after). Relating positively to user needs
>is a critical aspect of producing a spec that the community will
>continue to want to use.

I'm not of the same opinion that transparency is good in and of itself. 
Transparency will open up the DDI to more suggestions, and it seems to me 
the council already has its hands full with all the suggestions it 
currently receives. It's a lot of work to process these suggestions, 
separating the wheat from the chaff. I also have problems with releasing 
immature work-in-progress for public comment, which is what this kind of 
transparency would seem to do.

The goal of "relating positively to user needs" is one I agree with, but I 
just don't think that the DDI governance structure is set up to be able to 
do this, with a committee that meets only 2 or 3 times a year making all 
the major and minor decisions.

However, the suggestion for "comment prior to a versioning release" is one 
I heartily agree with as part of an overhaul in the versioning/change 
process. I'd prefer a more structured process than the informal one 
currently in place, something like

         --------
         PUBLIC COMMENT PHASE: VERSION 2.1
                 Announce to DDI-Users list that suggestions are currently 
being accepted for 2.1
                 ACTIVELY solicit feature requests and suggestions for change
                 Close: Announce to DDI-Users that no more feature requests 
accepted for 2.1
         --------
         SUGGESTION EVALUATION PHASE
                 Begin an official thread for each request in the previous 
phase.
                 Contact original authors of suggestions and invite them to 
participate.
                 Discuss each thread using the forum software rather than 
private emails so the entire discussion is viewable and archived
                 At some point, decide to create a formal written proposal 
out of each thread. These threads will become the basis of the next council 
meeting
         --------
         COUNCIL MEETING:
                 Discuss each thread
                 Decide a course of action for each thread. Similar to the 
process of peer-reviewed journals, possible actions are: Approve, Reject, 
or Revise & Resubmit
                 All approved threads are sent to implementers
         ---------
         IMPLEMENTATION PHASE:
                 Core group of developers decide on best way to implement 
council directives
         ---------
         PUBLIC COMMENT PHASE: VERSION 2.1 BETA RELEASE
                 Freeze changes on DTD/Schema
                 Announce to DDI-Users availability of beta version of 2.1 
for comment
                 Test and revise based on comments. Small 
revisions/corrections made by implementers. Major revisions must go back to 
council (and will push back date of official release)
         ---------
         VERSION 2.1 OFFICIAL RELEASE

I envision that this whole process takes nearly 2 council meeting 
intervals, so probably about 10-11 months for suggested changes to get into 
the next version. Because of this, I would also have overlapping processes, 
with the PUBLIC COMMENT PHASE for VERSION 2.2 to begin at roughly the same 
time as the IMPLEMENTATION PHASE for VERSION 2.1

I'm afraid I've gone a bit afield in my reply, but the essence of it is 
that I think that there is indeed a process problem within the DDI, but 
it's not one that I think can be solved by exposing the whole process via 
SourceForge.

>>>4.) I required the development of a w3c Schema for many specific reasons 
>>>to deal with the limitations in the integration of DTD based XML 
>>>content. OAI's Harvesting Protocol being the largest requirement for xsd 
>>>based validation and a central location for DDI w3c schema's. It would 
>>>be a false statement to suggest that these w3c Schema implementations 
>>>had any Council involvement beyond my direct interaction with the DDI 
>>>group. Unfortunately, I was not able to attend the Conference last 
>>>month, so I do not know if any discussions occurred around this subject 
>>>at the meeting.
>>
>>I'm not familiar with OAI Harvesting Protocol so I'm not sure how it 
>>mandates the use of a schema, but your problem seems to be a general 
>>problem of ensuring that the markup meets certain standards. You've 
>>chosen (or OAI mandates that you choose) to use XSD to do this.
>>I have a similar problem in that my application accepts DDI documents to 
>>be input the database so that a search can be performed on the variable 
>>level. However, I've found that documents that validate to the DTD do not 
>>necessarily provide high-enough quality markup for a search to be 
>>effective. My solution to the problem is that I've started working on is 
>>an XSLT quality-checker stylesheet to supplement DTD validation.
>>The advantages of this XSLT approach are:
>>   - I can restrict attribute type even if the specifications do not.
>>   - I can check validity, type, and number of ID references. Even XSD 
>> cannot do this, and there's a lot more I can do via XSLT that XSD will 
>> never be able to do.
>>   - While this "XSLT validation" overlaps with DDI specification 
>> development, it does not actually conflict with it. I will have no 
>> problems if and when the DDI becomes an XSD. Using your approach, 
>> because a single XML document cannot validate against two XSDs (unless 
>> namespaces are used), you would have to abandon your XSD when an 
>> official DDI XSD came out which did not adopt all your suggested changes.
>>   - Because I'm not writing a DTD or XSD, I don't need pre-approval from 
>> the Council, I can go ahead and start working.
>>   - I'm effectively separating validation into mostly "routine 
>> validation" to be handled by DTD/XSD, and a little bit of "custom 
>> validation" to be handled by XSLT.
>
>An interesting idea. I expect that the any development in the area of
>XSD, that the XSD technical implementation would currently be required
>to maintain a certain amount of "backward compatibility" in reference to
>validation against the DTD implementation for that Version of the DDI.
>
>But in reality, there are technical limitations to the above idea:
>
>1.) More "specific" control of content where it is appropriate. This can
>maintain "backward compatibility". A XML document validated against an
>XSD would also be valid against a DTD. But not necessarily the other
>way around.

I would make a stronger statement. Any document validated against a 
previous DTD should have no expectations of being valid against an XSD. 
When and if the DDI moves to an XSD schema, there must be a resolve to make 
a clean break with DTDs. In my opinion, faced with the choice having an XSD 
with the albatross of "backwards compatibility", or having no XSD at all, I 
would prefer to remain with DTDs. To  me, there is no point in having an 
XSD that simply duplicates the functionality of the DTD. There are things 
which can be done to ease the migration, such as providing an XSLT 
stylesheet to check for potential problems, but I think "backwards 
compatibility" has to be sacrificed because this is a significant change.

>2.) Embedding the DDI into other xml content and embedding other content
>into the DDI (such as OAI). This would not be able to maintain backward
>compatibility because DTD just doesn't support any thing near this
>capability.

This is something that requires major structural changes which would again 
necessitate the abandonment of backwards compatibility. Despite the X in 
XML, the DDI currently is not designed for extensibility. I'm not familiar 
with OAI schema, so I wouldn't know if they're designed for it. But if this 
were a council goal, then I'd suggest that XSD change and OAI embedding 
change be implemented simultaneously because these changes cannot be 
implemented without breaking backward compatibility.