Answers to the most common questions you may have about Aftershock.
Are there any rules or restrictions on submitted systems?
The only firm rule is that you may not train or tune your systems on any data related to the available tasks other than the provided development dataset.
Is there a deadline?
No. Aftershock is an open-ended benchmark with no deadline or set end date. However, Aftershock may be updated with new tasks with or without notice.
Where can I retrieve the data for the tasks?
You can download per-task development datasets from the "Tasks" section on this website. To ensure the integrity of the benchmark, the testing datasets shall remain hidden.
What license is the Aftershock data distributed under?
The tasks herein are built on and derived from existing datasets. We refer users to the original licenses accompanying each task under the "Tasks" section on this website.
Can I make a submission anonymously?
You are welcome to submit to the leaderboard without using your real name as long as you include a link to a paper describing your submission. This will often involve anonymous paper archives like OpenReview Anonymous Preprint.
How does the global leaderboard work?
We calculate scores for each of the tasks based on their individual metrics. The geometric mean of these scores scaled by 100 (i.e. as a percentage) becomes the final score. If your system does not participate in a given task, it is assigned a score of 0.
Do you have a discussion group?
TBD
How should I cite Aftershock?
TBD