INCF Standards Review Criteria v.2.0
It is essential that a FAIR supporting standard is open and allows free use by the community. Open development practices are also strongly encouraged to facilitate transparency and adoption. If questions do not apply, leave them blank or mark N/A.
1. Is the SBP covered under an open license so that it is free to implement and reuse by all interested parties (including commercial)? (List of open source licenses)
Considers the SBP from the point of view of some (not all) of the FAIR criteria (Wilkinson et al. 2016). Is the SBP itself FAIR? Does it result in the production of FAIR research objects? Note that many of these may not apply. If so, leave blank or mark N/A.
1. SBP uses/permits persistent identifiers where appropriate (F1)
|Design, Testing and implementation||
These may not all apply, if so, leave blank or mark N/A. Proper design, testing, and implementation, in addition to supporting tools greatly aid in adoption of a standard.
1. What is the technical expertise level required to implement this? Even if it is quite difficult, should it be implemented anyway?
Ongoing governance is key to ensuring the transparency about how a standard was created, and ensuring the stewards are responsive to the needs of the community. Standards require transparent governance practices; however it is possible some of the following questions do not apply; if so, leave blank or mark N/A.
1. Does the SBP have a clear description on who is maintaining the SBP and how decisions regarding its development are made?
|Adoption and use||
The standard must have substantive evidence of use outside of the group or individual that develops and maintains it. However, different levels of adoption and use will be taken into consideration depending on the purpose of the standard and the size/type of audience that might implement the standard. Because INCF represents organizations world-wide, evidence of international use is highly desirable.
1. Is there evidence of community use beyond the group that developed the SBP?
|Stability and support||
Standards need some sort of ongoing stability and support to ensure it will be useful in the future. However, given the nature of research projects, the level of acceptable stability and support is somewhat at the discretion of the SBP Committee and reviewers.
1. Who is responsible for maintaining the SBP?
If it is possible to update or potentially apply the standard to other areas, that should be indicated. The ability for a standard to be extensible is highly desirable, but not required. This is an area where having this knowledge is valuable to INCF and the community in general.
1. Can the SBP be extended to cover additional domains/use cases?
|Comparison||1. Are there other similar SBP's available?
2. If yes, how do they compare on key INCF criteria?
3. What are the key advantages of the SBP when compared to other SBPs?
4. Any additional comments on comparison with other SBPs?
Part 1: Expert review
- Nominations are received from the community via an on-line form.
- Nominations are reviewed on a monthly basis by the committee and those that are deemed appropriate are selected for further vetting.
- If a large number of SBP’s are submitted, the committee will establish a schedule for vetting and will send a communication to the submitter regarding the time line
- A subcommittee of 2-3 individuals is selected to further review the SBP:
- The submitter will be contacted to provide more detail about the SBP
- Committee members will use an on-line form for their review
- Reviewers may interact with the contact individual or other relevant parties to obtain clarifications or additional information
- Reviewers may solicit a proxy from their node to conduct the review
- The findings of the subcommittee are discussed at the next monthly meeting and a recommendation put forward for vote:
- Possible outcomes:
- Put forward the SBP for community feedback
- Reject the SBP
- Request further details
- Voting procedure: Ideally, the SBP committee will work through concerns to arrive at consensus on the required action. In the case of a disagreement, the action will be put to a vote to be decided by a simple majority of SBP members.
- Put forward the SBP for community feedback: A description of the SBP will be prepared and put forward to the INCF community for comment.
- Reject the SBP: SBP chair will put together an email to be sent to the submitter and/or steward as appropriate detailing the reasons why the SBP will not be considered further.
- Request further details: Questions and comments will be sent to the submitter/steward so they can respond. Action will be deferred until SBP committee hears back.
Part 2: Community comment
- SBP’s under consideration will be posted for community comment. The current plan is to use the F1000 platform: The public review may extend a minimum of sixty (60) days. The CTSI reserves the right to extend the public review period if it deems necessary.
- Once an SBP is posted, it will be broadly advertised through INCF and other networks
- Members of the community will be given the opportunity to respond via email or using the comment feature of the platform.
- After the commenting period, the subcommittee in charge of the SBP will summarize the feedback.
Part 3: Committee review
- The SBP committee will vote on the action:
- Endorse the SBP as an INCF SBP: Any suggestions/criticisms by the community are easy addressable, e.g., typos, wording
- Reject the SBP: The community provides information which disqualifies the SBP from endorsement
- Request further details: Significant concerns have been raised that will have to be addressed before further consideration
- Endorsement: The INCF secretariat notifies the submitter/steward and that they are now allowed to display the “Endorsed by INCF” badge. The final SPB review is posted online. The SBP is included on the INCF Portal list of endorsed SBPs.
- Reject: The secretariat sends a note to the submitter/steward detailing reasons why the SBP was rejected.
- Request further details: Questions/comments are sent to the submitter/steward. The SBP is then considered at the next SBP meeting to determine whether the SBP needs to be posted for further community review.
- Voting Procedure: Ideally, the SBP committee will work through concerns to arrive at consensus on the required action. In the case of a disagreement, the action will be put to a vote to be decided by a ⅔ vote of SBP members.
Conflicts of interest
- Members who have a conflict of interest should declare those conflicts and recuse themselves from the review and voting process
- Examples of conflicts:
- Member is an author or steward of an SBP
- Member is in a line management or other supervisory chain (above or below) with the author of an SBP
- Member has a financial interest in the SBP or tools related to the SBP
- Member serves on the advisory board for an SBP
- Member is developing or has developed a competing standard
- Member does not feel that s/he can be an impartial reviewer
The INCF SBP procedure should operate in a transparent manner and seek to avoid at all times any type of bias or appearance of bias. The process should be fair to those who are developing SBP’s, but also in the best interests of the broader neuroscience community we seek to serve.
Transparency and lack of bias are particularly critical when competing SBP’s exist. Competing SBP’s should ideally be identified by the submitter, the review committee, or during the period of community comment. If both SBP’s are found to be substantially compliant with INCF criteria, then INCF may opt to not issue an endorsement and list both SBP’s on their website along with the results of the review to provide information to the community about SBP’s relevant to their work. INCF should encourage and support the community to come together via working groups to address interoperability and possibility of reconciliation of competings SBP’s can be achieved.
If a competing SBP comes to light after an endorsement is issued, the committee should determine whether the review process was fair and comprehensive, i.e., should the competing SBP have been identified? If the process was fair and comprehensive, INCF will not withdraw the endorsement, but will develop a mitigation plan. For example, it may turn out that a competing SBP was developed in another domain, or that it is possible to develop approaches to make the two interoperable. Again, INCF should encourage and support the community to come together via working groups to address interoperability and possibility of reconciliation of competings SBP’s can be achieved.
Grievance with the review process
Situations may arise where an author/steward or community member feel that the review process itself was not performed thoroughly or fairly or where they feel an incorrect decision was reached. For example, an individual(s) may disagree with the endorsement of a standard or a decision not to endorse.
Complaints about the review process and any decisions will be reviewed by the committee. The SBP should set up an issue tracker where such concerns can be expressed. Anonymous complaints will not be accepted.
Ideally, issues should be able to be resolved through communication and discussion between the committee and the aggrieved individual. The issue should be posted to a public forum for community input if appropriate, although public posting will be at the discretion of the SBP Committee. Appropriate actions should be taken to mitigate the concern if possible. If not possible, then procedures should be reviewed and revised to prevent similar issues from occurring in the future. The committee should ensure that the complaint and its resolution are documented and all parties indicate their agreement with the decision. If appropriate, the complaint and its resolution should be posted on the INCF’s website.
If the complaint cannot be resolved amicably through this mechanism, the complaint will be referred to the INCF CTSI. The CTSI can either review the complaint and recommend an action or appoint a committee with the appropriate expertise to provide advice. The committee should ensure that the complaint and its resolution are posted in a public place and that all parties indicate their agreement with the decision.