Link to version 1.0 of the review criteria

Endorsement process

Part 1:  Expert review

1. Nominations are received from the community via an on-line form.
2. Nominations are reviewed on a monthly basis by the committee and those that are deemed appropriate are selected for further vetting.
3. If a large number of SBP’s are submitted, the committee will establish a schedule for vetting and will send a communication to the submitter regarding the time line
4. A subcommittee of 2-3 individuals is selected to further review the SBP:

1. The submitter will be contacted to provide more detail about the SBP
2. Committee members will use an on-line form for their review
3. Reviewers may interact with the contact individual or other relevant parties to obtain clarifications or additional information
4. Reviewers may solicit a proxy from their node to conduct the review

5. The findings of the subcommittee are discussed at the next monthly meeting and a recommendation put forward for vote:

1. Possible outcomes:

1. Put forward the SBP for community feedback
2. Reject the SBP
3. Request further details

2. Voting procedure: Ideally, the SBP committee will work through concerns to arrive at consensus on the required action. In the case of a disagreement, the action will be put to a vote to be decided by a simple majority of SBP members.

6. Outcomes:

1. Put forward the SBP for community feedback: A description of the SBP will be prepared and put forward to the INCF community for comment.   
2. Reject the SBP: SBP chair will put together an email to be sent to the submitter and/or steward as appropriate detailing the reasons why the SBP will not be considered further.
3. Request further details: Questions and comments will be sent to the submitter/steward so they can respond. Action will be deferred until SBP committee hears back.

Part 2:  Community comment

1. SBP’s under consideration will be posted for community comment.  The current plan is to use the F1000 platform: The public review may extend a minimum of sixty (60) days. The CTSI reserves the right to extend the public review period if it deems necessary.
2. Once an SBP is posted, it will be broadly advertised through INCF and other networks
3. Members of the community will be given the opportunity to respond via email or using the comment feature of the platform.
4. After the commenting period, the subcommittee in charge of the SBP will summarize the feedback.

Part 3: Committee review

1. The SBP committee will vote on the action:

1. Endorse the SBP as an INCF SBP:  Any suggestions/criticisms by the community are easy addressable, e.g., typos, wording
2. Reject the SBP:  The community provides information which disqualifies the SBP from endorsement
3. Request further details:  Significant concerns have been raised that will have to be addressed before further consideration

2. Endorsement: The INCF secretariat notifies the submitter/steward and that they are now allowed to display the “Endorsed by INCF” badge. The final SPB review is posted online. The SBP is included on the INCF Portal list of endorsed SBPs.
3. Reject: The secretariat sends a note to the submitter/steward detailing reasons why the SBP was rejected.
4. Request further details: Questions/comments are sent to the submitter/steward. The SBP is then considered at the next SBP meeting to determine whether the SBP needs to be posted for further community review.
5. Voting Procedure: Ideally, the SBP committee will work through concerns to arrive at consensus on the required action. In the case of a disagreement, the action will be put to a vote to be decided by a ⅔ vote of SBP members.

Conflicts of interest

1. Members who have a conflict of interest should declare those conflicts and recuse themselves from the review and voting process
2. Examples of conflicts:  

1. Member is an author or steward of an SBP
2. Member is in a line management or other supervisory chain (above or below) with the author of an SBP
3. Member has a financial interest in the SBP or tools related to the SBP
4. Member serves on the advisory board for an SBP
5. Member is developing or has developed a competing standard
6. Member does not feel that s/he can be an impartial reviewer 

Grievance procedures

Competing standards

The INCF SBP procedure should operate in a transparent manner and seek to avoid at all times any type of bias or appearance of bias.  The process should be fair to those who are developing SBP’s, but also in the best interests of the broader neuroscience community we seek to serve.  

Transparency and lack of bias are particularly critical when competing SBP’s exist. Competing SBP’s should ideally be identified by the submitter, the review committee, or during the period of community comment. If both SBP’s are found to be substantially compliant with INCF criteria, then INCF may opt to not issue an endorsement and list both SBP’s on their website along with the results of the review to provide information to the community about SBP’s relevant to their work.  INCF should encourage and support the community to come together via working groups to address interoperability and possibility of reconciliation of competings SBP’s can be achieved.

If a competing SBP comes to light after an endorsement is issued, the committee should determine whether the review process was fair and comprehensive, i.e., should the competing SBP have been identified?  If the process was fair and comprehensive, INCF will not withdraw the endorsement, but will develop a mitigation plan. For example, it may turn out that a competing SBP was developed in another domain, or that it is possible to develop approaches to make the two interoperable.  Again, INCF should encourage and support the community to come together via working groups to address interoperability and possibility of reconciliation of competings SBP’s can be achieved.

Grievance with the review process

Situations may arise where an author/steward or community member feel that the review process itself was not performed thoroughly or fairly or where they feel an incorrect decision was reached.  For example, an individual(s) may disagree with the endorsement of a standard or a decision not to endorse.

Complaints about the review process and any decisions will be reviewed by the committee.  The SBP should set up an issue tracker where such concerns can be expressed. Anonymous complaints will not be accepted.  

Ideally, issues should be able to be resolved through communication and discussion between the committee and the aggrieved individual. The issue should be posted to a public forum for community input if appropriate, although public posting will be at the discretion of the SBP Committee.  Appropriate actions should be taken to mitigate the concern if possible. If not possible, then procedures should be reviewed and revised to prevent similar issues from occurring in the future. The committee should ensure that the complaint and its resolution are documented and all parties indicate their agreement with the decision.  If appropriate, the complaint and its resolution should be posted on the INCF’s website. 

If the complaint cannot be resolved amicably through this mechanism, the complaint will be referred to the INCF CTSI. The CTSI can either review the complaint and recommend an action or appoint a committee with the appropriate expertise to provide advice.  The committee should ensure that the complaint and its resolution are posted in a public place and that all parties indicate their agreement with the decision.