Frequently asked questions (FAQ)
Why a new challenge, are there not too many already?
Machine learning models are now widely applied to predict biologically meaningful relations. Examples include the prediction of protein functions, gene disease associations, association of proteins and pathways, drug targets, drug indications, pharmacological effects, and similar. Often, the evaluation of these methods is based on synthetic datasets or by withholding parts of a data set and reporting predictive performance measures on this set. However, such evaluation methods could be prone to biases; for example, testing on a randomly selected subset of data may retain some information that makes predictions overly simple and consequently the performance measures may not accurately reflect how well relations can be predicted in the
One effort to resolve this issue is to perform a time-based evaluation where predictions are submitted at a time and, after a period of waiting, these predictions are evaluated on new knowledge that became available. Large-scale evaluation efforts such as CASP (for protein structures) or CAFA (for protein functions) follow such an approach. However, these are efforts that are run at specific time points; they do not, for example, allow evaluating how model performance is affected when the models are used as part of the decision making process.
Running such an evaluation usually takes a significant amount of time and resources. As biological database are now increasingly being made available through Semantic Web technologies, in particular SPARQL endpoints, we have an opportunity to define such challenges for any type of association that can be discovered through (federated) SPARQL queries. We provide a method to define and run challenges, to submit predictions for them and evaluate. SPARQL queries we use for the challenges can be found and tested at the BioHackathon SPARQlist.
What kind of predictions do you evaluate?
We evaluate the prediction for relations between (biological) entities. Entities and relations are identified by an IRI, and predictions may be associated with a confidence score. The predictions can be made using any model, including link prediction models, text mining, or others. In the future, we may introduce specific sub-challenges that focus on using only particular types of data (such as text-only, or graph-only, challenges).
Will you evaluate negative predictions?
Not yet. We consider a
negative prediction a
prediction that an existing relation is false. We plan to
include these predictions in our evaluation in the future.
Is there a meeting/workshop/conference associated with CERLIB to discuss results and new ideas?
Yes, we plan to meet annually at the Bio-Ontologies SIG at ISMB. Additionally, we intend to publish the methods used and developed as part of CERLIB in a journal special issue.