---
# System prepended metadata

title: Competition specs 1.0  summary
tags: [Research, Comeptition]

---

# Competition specs 1.0  summary

###### tags: `Research` `Comeptition`

## Main Website and details
Main website: https://sites.google.com/view/pgdl2020
Specs 1.0:  https://drive.google.com/file/d/1maCI6XNxlAGjtqgzo7WKlOdI4_WBmX1P/view
Goal

## Task description
Our code is expected to take a training dataset of image label pairs as well as a model fully trained on it as input, and generate a real number as output. The value of the output should ideally be larger for models that have larger generalization gaps.

## Data

collection of trained neural networks
Keras API (integrated with TensorFlow 2.x)
all our models will be sequential
To remove a potential confounding factor, all models are trained until they reach the interpolation regime (close to 100% accuracy on the training set1 or to a fixed cross entropy value). 
Each model is represented by a JSON file describing its architecture and a set of trained weights (HDF5 format).
Public dataset

2 groups of model:
1. Models trained on CIFAR-10
2. Models trained on SVHN

With the different combinations of hyperparams such as batch sizes and dropout rates, there are 150 unique models in the entire dataset.

## Points about creating model

The largest challenge of designing such complexity measure is making it robust to changes in the model architectures and the dataset used for training the model. Many complexity measures only work on a single dataset or only correlates with one particular type of hyper-parameter change (e.g. depth). While such a generic complexity measure may seem to be difficult to obtain, recent work [https://arxiv.org/abs/1912.02178] has proposed rigorous protocols for identifying promising complexity measure, and shows that it is possible to find complexity measures that fulfill the above criteria.
complexity measures can also be instrumental in Neural Architecture Search (NAS)
if the complexity measure is fully differentiable, it may act as a regularizer to improve generalization

## Ranking/ Evaluation:

1. Conditional Mutual Information[Section 1.5.2 Metric 2] is the only metric used for ranking submissions If needed, human evaluation

## Timeline

- Jul 15: Phase 1 starts. Participants submit their solution to be evaluated on the
Phase 1 private dataset.
If the solution finishes on Phase 1 within time (tentative- 5 minutes) then their code will be run on the Phase 2 private dataset without any time limit.
- Oct 08: Phase 2 begins. Participants’ code are evaluated on the Phase 2 private
dataset.
- Oct 24: Phase 2 ends. All computation finalized.
- Oct 31: Results are announced.
- Dec 11: Winning teams are invited to present at the conference

A maximum submissions of 5 is allowed per day for each team.

