Skip to main content
x

INCF Standards Review Criteria v.2.0

INCF Standards Review Criteria v.2.0

Version 2.0 (DOI: 10.7490/f1000research.1118367.1)
Approved, 9th November, 2020
Authors:  INCF SBP Committee
Link to earlier versions

Open

It is essential that a FAIR supporting standard is open and allows free use by the community. Open development practices are also strongly encouraged to facilitate transparency and adoption. If questions do not apply, leave them blank or mark N/A.

1. Is the SBP covered under an open license so that it is free to implement and reuse by all interested parties (including commercial)? (List of open source licenses)
2. What license is used?
3. Does the SBP follow open development practices?
4. Where and how are the code/documents managed?
5. Any additional comments on the openness of the SBP?

FAIR

Considers the SBP from the point of view of some (not all) of the FAIR criteria (Wilkinson et al. 2016). Is the SBP itself FAIR? Does it result in the production of FAIR research objects?  Note that many of these may not apply. If so, leave blank or mark N/A.

1. SBP uses/permits persistent identifiers where appropriate (F1)
2. SBP allows addition of rich metadata to research objects (F2)
3. SBP uses/permits addition of appropriate PIDs to metadata (F3)
4. The protocol allows for an authentication and authorization when required (A1-2)
5. SBP uses or allows the use of vocabularies that follow the FAIR principles (I2)
6. SBP includes/allows qualified links to other identifiers (I3)
7. Does the standard interoperate with other relevant standards in the same domain? (I1)
8. Does the SBP provide citation metadata so its use can be documented and tracked? (R1.2)
9. Does the SBP have a clear versioning scheme and appropriate
documentation?
10. Any additional comments on aspects of FAIR?

Design, Testing and implementation

These may not all apply, if so, leave blank or mark N/A. Proper design, testing, and implementation, in addition to supporting tools greatly aid in adoption of a standard.

1. What is the technical expertise level required to implement this? Even if it is quite difficult, should it be implemented anyway?
2. Does the SBP provide an architectural concept to understand its implementation and relationships to external entities?
3. Does the SBP have a reference implementation?
4. What software resources (files/scripts/libraries/tools) are available to support the SBP?
5. Are the supporting software resources tools and implementations covered under an open source license?
      a.    Are the supporting software resources well documented (documentation of I/O operations, programming interfaces, user interfaces, installation)?
      b.    Were the supporting software resources validated?
      c.    What is your assessment of the quality of the code/document?
      d.    Have the supporting software resources been deployed, is there any experience or references to their use by the community?
6. Any additional comments on design, testing, and implementation?

Governance

Ongoing governance is key to ensuring the transparency about how a standard was created, and ensuring the stewards are responsive to the needs of the community. Standards require transparent governance practices; however it is possible some of the following questions do not apply; if so, leave blank or mark N/A.

1. Does the SBP have a clear description on who is maintaining the SBP and how decisions regarding its development are made?
2. Is the governing model document for maintenance and updates compatible with the INCF project governing model document and the open standards principles?
3. Is the SBP actively supported by the community? If so, what is the evidence?
4. Does the SBP provide tools for community feedback and support?
5. Any additional comments on governance?

Adoption and use

The standard must have substantive evidence of use outside of the group or individual that develops and maintains it. However, different levels of adoption and use will be taken into consideration depending on the purpose of the standard and the size/type of audience that might implement the standard. Because INCF represents organizations world-wide, evidence of international use is highly desirable.

1. Is there evidence of community use beyond the group that developed the SBP?
2. Please provide some concrete examples of use, e.g., publications where the use of the SBP is cited; databases or other projects that have adopted the SBP
3. Is there evidence of international use?
4. Any additional comments on use?

Stability and support

Standards need some sort of ongoing stability and support to ensure it will be useful in the future. However, given the nature of research projects, the level of acceptable stability and support is somewhat at the discretion of the SBP Committee and reviewers.

1. Who is responsible for maintaining the SBP? 
2. How is it currently supported?
3. What is the plan for long term support?
4. Are training and other supporting materials available? 
5. Any additional comments on sustainability and support

Extensibility

If it is possible to update or potentially apply the standard to other areas, that should be indicated. The ability for a standard to be extensible is highly desirable, but not required. This is an area where having this knowledge is valuable to INCF and the community in general.

1. Can the SBP be extended to cover additional domains/use cases? 
2. If so, how is the process documented and managed?
3. Any additional comments on extensibility?

Comparison 1. Are there other similar SBP's available?
2. If yes, how do they compare on key INCF criteria?
3. What are the key advantages of the SBP when compared to other SBPs?
4. Any additional comments on comparison with other SBPs?

Link to version 1.0 of the review criteria

Endorsement process

Part 1:  Expert review

  1. Nominations are received from the community via an on-line form.
  2. Nominations are reviewed on a monthly basis by the committee and those that are deemed appropriate are selected for further vetting.
  3. If a large number of SBP’s are submitted, the committee will establish a schedule for vetting and will send a communication to the submitter regarding the time line
  4. A subcommittee of 2-3 individuals is selected to further review the SBP:
    1. The submitter will be contacted to provide more detail about the SBP
    2. Committee members will use an on-line form for their review
    3. Reviewers may interact with the contact individual or other relevant parties to obtain clarifications or additional information
    4. Reviewers may solicit a proxy from their node to conduct the review
  1. The findings of the subcommittee are discussed at the next monthly meeting and a recommendation put forward for vote:
    1. Possible outcomes:
      1. Put forward the SBP for community feedback
      2. Reject the SBP
      3. Request further details
    1. Voting procedure: Ideally, the SBP committee will work through concerns to arrive at consensus on the required action. In the case of a disagreement, the action will be put to a vote to be decided by a simple majority of SBP members.
  1. Outcomes:
    1. Put forward the SBP for community feedback: A description of the SBP will be prepared and put forward to the INCF community for comment.   
    2. Reject the SBP: SBP chair will put together an email to be sent to the submitter and/or steward as appropriate detailing the reasons why the SBP will not be considered further.
    3. Request further details: Questions and comments will be sent to the submitter/steward so they can respond. Action will be deferred until SBP committee hears back.

Part 2:  Community comment

  1. SBP’s under consideration will be posted for community comment.  The current plan is to use the F1000 platform: The public review may extend a minimum of sixty (60) days. The CTSI reserves the right to extend the public review period if it deems necessary.
  2. Once an SBP is posted, it will be broadly advertised through INCF and other networks
  3. Members of the community will be given the opportunity to respond via email or using the comment feature of the platform.
  4. After the commenting period, the subcommittee in charge of the SBP will summarize the feedback.

Part 3: Committee review

  1. The SBP committee will vote on the action:
    1. Endorse the SBP as an INCF SBP:  Any suggestions/criticisms by the community are easy addressable, e.g., typos, wording
    2. Reject the SBP:  The community provides information which disqualifies the SBP from endorsement
    3. Request further details:  Significant concerns have been raised that will have to be addressed before further consideration
  1. Endorsement: The INCF secretariat notifies the submitter/steward and that they are now allowed to display the “Endorsed by INCF” badge. The final SPB review is posted online. The SBP is included on the INCF Portal list of endorsed SBPs.
  2. Reject: The secretariat sends a note to the submitter/steward detailing reasons why the SBP was rejected.
  3. Request further details: Questions/comments are sent to the submitter/steward. The SBP is then considered at the next SBP meeting to determine whether the SBP needs to be posted for further community review.
  4. Voting Procedure: Ideally, the SBP committee will work through concerns to arrive at consensus on the required action. In the case of a disagreement, the action will be put to a vote to be decided by a ⅔ vote of SBP members.

 

Conflicts of interest

  1. Members who have a conflict of interest should declare those conflicts and recuse themselves from the review and voting process
  2. Examples of conflicts:  
    1. Member is an author or steward of an SBP
    2. Member is in a line management or other supervisory chain (above or below) with the author of an SBP
    3. Member has a financial interest in the SBP or tools related to the SBP
    4. Member serves on the advisory board for an SBP
    5. Member is developing or has developed a competing standard
    6. Member does not feel that s/he can be an impartial reviewer 

 

Grievance procedures

Competing standards

The INCF SBP procedure should operate in a transparent manner and seek to avoid at all times any type of bias or appearance of bias.  The process should be fair to those who are developing SBP’s, but also in the best interests of the broader neuroscience community we seek to serve.  

Transparency and lack of bias are particularly critical when competing SBP’s exist. Competing SBP’s should ideally be identified by the submitter, the review committee, or during the period of community comment. If both SBP’s are found to be substantially compliant with INCF criteria, then INCF may opt to not issue an endorsement and list both SBP’s on their website along with the results of the review to provide information to the community about SBP’s relevant to their work.  INCF should encourage and support the community to come together via working groups to address interoperability and possibility of reconciliation of competings SBP’s can be achieved.

If a competing SBP comes to light after an endorsement is issued, the committee should determine whether the review process was fair and comprehensive, i.e., should the competing SBP have been identified?  If the process was fair and comprehensive, INCF will not withdraw the endorsement, but will develop a mitigation plan. For example, it may turn out that a competing SBP was developed in another domain, or that it is possible to develop approaches to make the two interoperable.  Again, INCF should encourage and support the community to come together via working groups to address interoperability and possibility of reconciliation of competings SBP’s can be achieved.

Grievance with the review process

Situations may arise where an author/steward or community member feel that the review process itself was not performed thoroughly or fairly or where they feel an incorrect decision was reached.  For example, an individual(s) may disagree with the endorsement of a standard or a decision not to endorse.

Complaints about the review process and any decisions will be reviewed by the committee.  The SBP should set up an issue tracker where such concerns can be expressed. Anonymous complaints will not be accepted.  

Ideally, issues should be able to be resolved through communication and discussion between the committee and the aggrieved individual. The issue should be posted to a public forum for community input if appropriate, although public posting will be at the discretion of the SBP Committee.  Appropriate actions should be taken to mitigate the concern if possible. If not possible, then procedures should be reviewed and revised to prevent similar issues from occurring in the future. The committee should ensure that the complaint and its resolution are documented and all parties indicate their agreement with the decision.  If appropriate, the complaint and its resolution should be posted on the INCF’s website. 

If the complaint cannot be resolved amicably through this mechanism, the complaint will be referred to the INCF CTSI. The CTSI can either review the complaint and recommend an action or appoint a committee with the appropriate expertise to provide advice.  The committee should ensure that the complaint and its resolution are posted in a public place and that all parties indicate their agreement with the decision.