---
# System prepended metadata

title: Using CNN to Predict Bone age
tags: [tensorflow, NN, python, Keras]

---

# Using CNN to Predict Bone age
###### tags: `python` `tensorflow` `NN` `Keras`

## VGG16

### Import library
```python
# Import data & preprocessing
import tensorflow as tf
import keras
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # showing and rendering figures
# io related
from skimage.io import imread
import os
from glob import glob
# not needed in Kaggle, but required in Jupyter
%matplotlib inline 
```
### Import dataset

```python
# Import data
base_bone_dir = os.path.join('.')
print (base_bone_dir)
age_df = pd.read_csv(os.path.join(base_bone_dir, 'boneage-training-dataset.csv'))

ion for now
boneage_mean = 0
boneage_div = 1.0# 製作需要的欄位
age_df['path'] = age_df['id'].map(lambda x: os.path.join(base_bone_dir,
                                                         'resize',                                                          
                                                         '{}.png'.format(x)))
age_df['exists'] = age_df['path'].map(os.path.exists)
print(age_df['exists'].sum(), 'images found of', age_df.shape[0], 'total')
age_df['gender'] = age_df['male'].map(lambda x: 'male' if x else 'female')
boneage_mean = age_df['boneage'].mean()
boneage_div = 2*age_df['boneage'].std()
# we don't want normalizat
age_df['boneage_zscore'] = age_df['boneage'].map(lambda x: (x-boneage_mean)/boneage_div)
age_df.dropna(inplace = True)

age_df['male']= age_df['male'].map(lambda x: 1 if x else 0)
age_df[['boneage', 'male', 'boneage_zscore']].hist(figsize = (10, 5))
age_df['boneage_category'] = pd.cut(age_df['boneage'], 10)
```

![](https://i.imgur.com/RFGcEPX.png)


```python
age_df['boneage_category'] = pd.cut(age_df['boneage'], 10)
new_age_df = age_df.groupby(['boneage_category', 'male']).apply(lambda x: x.sample(400, replace = True)
                                                      ).reset_index(drop = True)
print('New Data Size:', new_age_df.shape[0], 'Old Size:', age_df.shape[0])
new_age_df[['boneage', 'male']].hist(figsize = (10, 5))
```

    New Data Size: 8000 Old Size: 12611
    




    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x00000134C390E948>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x00000134C3C0BB88>]],
          dtype=object)



![](https://i.imgur.com/lZuLTyX.png)



```python
# 將Training & validation 以3:1比例分開
from sklearn.model_selection import train_test_split
raw_train_df, valid_df = train_test_split(age_df, 
                                   test_size = 0.25, 
                                   random_state = 2018,
                                   stratify = age_df['boneage_category'])
print('train', raw_train_df.shape[0], 'validation', valid_df.shape[0])
```

    train 9458 validation 3153
    


```python
# Balance the distribution in the training set
train_df = raw_train_df.groupby(['boneage_category', 'male']).apply(lambda x: x.sample(500, replace = True)
                                                      ).reset_index(drop = True)
print('New Data Size:', train_df.shape[0], 'Old Size:', raw_train_df.shape[0])
train_df[['boneage', 'male']].hist(figsize = (10, 5))

# 圖檔預處理
from keras.preprocessing.image import ImageDataGenerator
from keras.applications.vgg16 import preprocess_input
IMG_SIZE = (384, 384) # slightly smaller than vgg16 normally expects
core_idg = ImageDataGenerator(samplewise_center=False, 
                              samplewise_std_normalization=False, 
                              horizontal_flip = True, 
                              vertical_flip = False, 
                              height_shift_range = 0.15, 
                              width_shift_range = 0.15, 
                              rotation_range = 5, 
                              shear_range = 0.01,
                              fill_mode = 'nearest',
                              zoom_range=0.25,
                             preprocessing_function = preprocess_input)
```

    New Data Size: 10000 Old Size: 9458




```python
def flow_from_dataframe(img_data_gen, in_df, path_col, y_col, **dflow_args):
    base_dir = os.path.dirname(in_df[path_col].values[0])
    print('## Ignore next message from keras, values are replaced anyways')
    df_gen = img_data_gen.flow_from_directory(base_dir, 
                                     class_mode = 'sparse',
                                    **dflow_args)
    df_gen.filenames = in_df[path_col].values
    df_gen.filepaths.extend(df_gen.filenames)
    df_gen.classes = np.stack(in_df[y_col].values)
    df_gen.samples = in_df.shape[0]
    df_gen.n = in_df.shape[0]
    df_gen._set_index_array()
    df_gen.directory = '' # since we have the full path
    print('Reinserting dataframe: {} images'.format(in_df.shape[0]))
    return df_gen
	
train_gen = flow_from_dataframe(core_idg, train_df, 
                             path_col = 'path',
                            y_col = 'boneage_zscore', 
                            target_size = IMG_SIZE,
                             color_mode = 'rgb',
                            batch_size = 32)

valid_gen = flow_from_dataframe(core_idg, valid_df, 
                             path_col = 'path',
                            y_col = 'boneage_zscore', 
                            target_size = IMG_SIZE,
                             color_mode = 'rgb',
                            batch_size = 256) # we can use much larger batches for evaluation
# used a fixed dataset for evaluating the algorithm
test_X, test_Y = next(flow_from_dataframe(core_idg, 
                               valid_df, 
                             path_col = 'path',
                            y_col = 'boneage_zscore', 
                            target_size = IMG_SIZE,
                             color_mode = 'rgb',
                            batch_size = 1024)) # one big batch
```

    ## Ignore next message from keras, values are replaced anyways
    Found 0 images belonging to 0 classes.
    Reinserting dataframe: 10000 images
    ## Ignore next message from keras, values are replaced anyways
    Found 0 images belonging to 0 classes.
    Reinserting dataframe: 3153 images
    ## Ignore next message from keras, values are replaced anyways
    Found 0 images belonging to 0 classes.
    Reinserting dataframe: 3153 images
    


```python
# 確認載入的圖檔
t_x, t_y = next(train_gen)
fig, m_axs = plt.subplots(2, 4, figsize = (16, 8))
for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):
    c_ax.imshow(c_x[:,:,0], cmap = 'bone', vmin = -127, vmax = 127)
    c_ax.set_title('%2.0f months' % (c_y*boneage_div+boneage_mean))
    c_ax.axis('off')
```


![](https://i.imgur.com/GKFwzC6.png)


```python
# VGG16參數設定
from keras.applications.vgg16 import VGG16
from keras.layers import GlobalAveragePooling2D, Dense, Dropout, Flatten, Input, Conv2D, multiply, LocallyConnected2D, Lambda
from keras.models import Model
in_lay = Input(t_x.shape[1:])
base_pretrained_model = VGG16(input_shape =  t_x.shape[1:], include_top = False, weights = 'imagenet')
base_pretrained_model.trainable = False
pt_depth = base_pretrained_model.get_output_shape_at(0)[-1]
pt_features = base_pretrained_model(in_lay)
from keras.layers import BatchNormalization
bn_features = BatchNormalization()(pt_features)

# here we do an attention mechanism to turn pixels in the GAP on an off

attn_layer = Conv2D(64, kernel_size = (1,1), padding = 'same', activation = 'relu')(bn_features)
attn_layer = Conv2D(16, kernel_size = (1,1), padding = 'same', activation = 'relu')(attn_layer)
attn_layer = LocallyConnected2D(1, 
                                kernel_size = (1,1), 
                                padding = 'valid', 
                                activation = 'sigmoid')(attn_layer)
# fan it out to all of the channels
up_c2_w = np.ones((1, 1, 1, pt_depth))
up_c2 = Conv2D(pt_depth, kernel_size = (1,1), padding = 'same', 
               activation = 'linear', use_bias = False, weights = [up_c2_w])
up_c2.trainable = False
attn_layer = up_c2(attn_layer)

mask_features = multiply([attn_layer, bn_features])
gap_features = GlobalAveragePooling2D()(mask_features)
gap_mask = GlobalAveragePooling2D()(attn_layer)
# to account for missing values from the attention model
gap = Lambda(lambda x: x[0]/x[1], name = 'RescaleGAP')([gap_features, gap_mask])
gap_dr = Dropout(0.5)(gap)
dr_steps = Dropout(0.25)(Dense(1024, activation = 'elu')(gap_dr))
out_layer = Dense(1, activation = 'linear')(dr_steps) # linear is what 16bit did
bone_age_model = Model(inputs = [in_lay], outputs = [out_layer])

# MAE
from keras.metrics import mean_absolute_error
def mae_months(in_gt, in_pred):
    return mean_absolute_error(boneage_div*in_gt, boneage_div*in_pred)

bone_age_model.compile(optimizer = 'adam', loss = 'mse',
                           metrics = [mae_months])

bone_age_model.summary()
```

    Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
    58892288/58889256 [==============================] - 32s 1us/step
    Model: "model_1"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to                     
    ==================================================================================================
    input_1 (InputLayer)            (None, 384, 384, 3)  0                                            
    __________________________________________________________________________________________________
    vgg16 (Model)                   (None, 12, 12, 512)  14714688    input_1[0][0]                    
    __________________________________________________________________________________________________
    batch_normalization_1 (BatchNor (None, 12, 12, 512)  2048        vgg16[1][0]                      
    __________________________________________________________________________________________________
    conv2d_1 (Conv2D)               (None, 12, 12, 64)   32832       batch_normalization_1[0][0]      
    __________________________________________________________________________________________________
    conv2d_2 (Conv2D)               (None, 12, 12, 16)   1040        conv2d_1[0][0]                   
    __________________________________________________________________________________________________
    locally_connected2d_1 (LocallyC (None, 12, 12, 1)    2448        conv2d_2[0][0]                   
    __________________________________________________________________________________________________
    conv2d_3 (Conv2D)               (None, 12, 12, 512)  512         locally_connected2d_1[0][0]      
    __________________________________________________________________________________________________
    multiply_1 (Multiply)           (None, 12, 12, 512)  0           conv2d_3[0][0]                   
                                                                     batch_normalization_1[0][0]      
    __________________________________________________________________________________________________
    global_average_pooling2d_1 (Glo (None, 512)          0           multiply_1[0][0]                 
    __________________________________________________________________________________________________
    global_average_pooling2d_2 (Glo (None, 512)          0           conv2d_3[0][0]                   
    __________________________________________________________________________________________________
    RescaleGAP (Lambda)             (None, 512)          0           global_average_pooling2d_1[0][0] 
                                                                     global_average_pooling2d_2[0][0] 
    __________________________________________________________________________________________________
    dropout_1 (Dropout)             (None, 512)          0           RescaleGAP[0][0]                 
    __________________________________________________________________________________________________
    dense_1 (Dense)                 (None, 1024)         525312      dropout_1[0][0]                  
    __________________________________________________________________________________________________
    dropout_2 (Dropout)             (None, 1024)         0           dense_1[0][0]                    
    __________________________________________________________________________________________________
    dense_2 (Dense)                 (None, 1)            1025        dropout_2[0][0]                  
    ==================================================================================================
    Total params: 15,279,905
    Trainable params: 563,681
    Non-trainable params: 14,716,224
    __________________________________________________________________________________________________
    


```python
# VGG16參數設定(?)
from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau
weight_path="{}_weights.best.hdf5".format('bone_age')

checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, 
                             save_best_only=True, mode='min', save_weights_only = True)


reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)
early = EarlyStopping(monitor="val_loss", 
                      mode="min", 
                      patience=5) # probably needs to be more patient, but kaggle time is limited
callbacks_list = [checkpoint, early, reduceLROnPlat]
```

    C:\Users\Camille_Seknova\anaconda3\envs\tensorflow1\lib\site-packages\keras\callbacks\callbacks.py:998: UserWarning: `epsilon` argument is deprecated and will be removed, use `min_delta` instead.
      warnings.warn('`epsilon` argument is deprecated and '
    


```python
# 開始training
# train_gen.batch_size = 16
bone_age_model.fit_generator(train_gen, 
                                  validation_data = (test_X, test_Y), 
                                  epochs = 15, 
                                  callbacks = callbacks_list)
```

    Epoch 1/15
    313/313 [==============================] - 6021s 19s/step - loss: 2887.5958 - mae_months: 37.1822 - val_loss: 601.9203 - val_mae_months: 19.8439
    
    Epoch 00001: val_loss improved from inf to 601.92025, saving model to bone_age_weights.best.hdf5
    Epoch 2/15
    313/313 [==============================] - 5879s 19s/step - loss: 834.0107 - mae_months: 22.8183 - val_loss: 517.4436 - val_mae_months: 18.1152
    
    Epoch 00002: val_loss improved from 601.92025 to 517.44357, saving model to bone_age_weights.best.hdf5
    Epoch 3/15
    313/313 [==============================] - 5853s 19s/step - loss: 674.4779 - mae_months: 20.3838 - val_loss: 476.4907 - val_mae_months: 17.3899
    
    Epoch 00003: val_loss improved from 517.44357 to 476.49066, saving model to bone_age_weights.best.hdf5
    Epoch 4/15
    313/313 [==============================] - 5839s 19s/step - loss: 591.0403 - mae_months: 18.9640 - val_loss: 452.4837 - val_mae_months: 16.9138
    
    Epoch 00004: val_loss improved from 476.49066 to 452.48373, saving model to bone_age_weights.best.hdf5
    Epoch 5/15
    313/313 [==============================] - 5902s 19s/step - loss: 503.1557 - mae_months: 17.5464 - val_loss: 398.2757 - val_mae_months: 15.7191
    
    Epoch 00005: val_loss improved from 452.48373 to 398.27571, saving model to bone_age_weights.best.hdf5
    Epoch 6/15
    313/313 [==============================] - 5870s 19s/step - loss: 467.0240 - mae_months: 16.8923 - val_loss: 409.5181 - val_mae_months: 15.8301
    
    Epoch 00006: val_loss did not improve from 398.27571
    Epoch 7/15
    313/313 [==============================] - 5883s 19s/step - loss: 439.9315 - mae_months: 16.3247 - val_loss: 392.8785 - val_mae_months: 15.4780
    
    Epoch 00007: val_loss improved from 398.27571 to 392.87852, saving model to bone_age_weights.best.hdf5
    Epoch 8/15
    313/313 [==============================] - 5855s 19s/step - loss: 419.6139 - mae_months: 15.9334 - val_loss: 394.2897 - val_mae_months: 15.6384
    
    Epoch 00008: val_loss did not improve from 392.87852
    Epoch 9/15
    313/313 [==============================] - 5861s 19s/step - loss: 410.1131 - mae_months: 15.7633 - val_loss: 402.8027 - val_mae_months: 15.7428
    
    Epoch 00009: val_loss did not improve from 392.87852
    Epoch 10/15
    313/313 [==============================] - 5876s 19s/step - loss: 396.9571 - mae_months: 15.4562 - val_loss: 378.6116 - val_mae_months: 15.3340
    
    Epoch 00010: val_loss improved from 392.87852 to 378.61159, saving model to bone_age_weights.best.hdf5
    Epoch 11/15
    313/313 [==============================] - 6019s 19s/step - loss: 372.0665 - mae_months: 14.9971 - val_loss: 352.4316 - val_mae_months: 14.8172
    
    Epoch 00011: val_loss improved from 378.61159 to 352.43158, saving model to bone_age_weights.best.hdf5
    Epoch 12/15
    313/313 [==============================] - 6130s 20s/step - loss: 359.2982 - mae_months: 14.8550 - val_loss: 387.3231 - val_mae_months: 15.6298
    
    Epoch 00012: val_loss did not improve from 352.43158
    Epoch 13/15
    313/313 [==============================] - 6070s 19s/step - loss: 359.9599 - mae_months: 14.8379 - val_loss: 360.8462 - val_mae_months: 15.0052
    
    Epoch 00013: val_loss did not improve from 352.43158
    Epoch 14/15
    313/313 [==============================] - 6096s 19s/step - loss: 343.5727 - mae_months: 14.5815 - val_loss: 351.4307 - val_mae_months: 14.8820
    
    Epoch 00014: val_loss improved from 352.43158 to 351.43066, saving model to bone_age_weights.best.hdf5
    Epoch 15/15
    313/313 [==============================] - 6245s 20s/step - loss: 347.8223 - mae_months: 14.5407 - val_loss: 345.8929 - val_mae_months: 14.8902
    
    Epoch 00015: val_loss improved from 351.43066 to 345.89290, saving model to bone_age_weights.best.hdf5
    




    <keras.callbacks.callbacks.History at 0x134cff46c48>




```python
# load the best version of the model
bone_age_model.load_weights(weight_path)
```


```python
# 開始預測
pred_Y = boneage_div*bone_age_model.predict(test_X, batch_size = 32, verbose = True)+boneage_mean
test_Y_months = boneage_div*test_Y+boneage_mean
```

    1024/1024 [==============================] - 578s 564ms/step
    


```python
# 結果放置於 pred_Y 和實際月份 test_Y_months 進行參數比對
fig, ax1 = plt.subplots(1,1, figsize = (6,6))
ax1.plot(test_Y_months, pred_Y, 'r.', label = 'predictions')
ax1.plot(test_Y_months, test_Y_months, 'b-', label = 'actual')
ax1.legend()
ax1.set_xlabel('Actual Age (Months)')
ax1.set_ylabel('Predicted Age (Months)')
```




    Text(0, 0.5, 'Predicted Age (Months)')



![](https://i.imgur.com/1a31fcd.png)



```python
from sklearn.metrics import mean_absolute_error, median_absolute_error
print(mean_absolute_error(pred_Y, test_Y_months))
print(median_absolute_error(pred_Y, test_Y_months))
```

    14.890207
    12.889206
## MobieNet

* 啟動Anaconda powershell 切到工作目錄下，將影像dataset跟 boneage-training-dataset.csv 都放在同一目錄下

![](https://i.imgur.com/VLvoIwP.png)

* Import data & preprocessing
* 
```python
import tensorflow as tf
import keras
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # showing and rendering figures
# io related
from skimage.io import imread
import os
from glob import glob
# not needed in Kaggle, but required in Jupyter
%matplotlib inline
```

* Import data
```python
#base_bone_dir = os.path.join('c:', os.sep, 'Users', 'Elfay', 'ML')
base_bone_dir = os.path.join('.')
print (base_bone_dir)
age_df = pd.read_csv(os.path.join(base_bone_dir,'boneage-training-dataset.csv'))
age_df.describe
```

    .
    <index        id  boneage  male
        0       1377      180     0
        1       1378       12     0
        2       1379       94     0
        3       1380      120     1
        4       1381       82     0
        ...      ...      ...   ...
        [12611 rows x 3 columns]>


* 製作需要的欄位

```python
age_df['path'] = age_df['id'].map(lambda x: os.path.join(base_bone_dir,
                                                         'resize',  
                                                         '{}.png'.format(x)))
age_df['exists'] = age_df['path'].map(os.path.exists)
print(age_df['exists'].sum(), 'images found of', age_df.shape[0], 'total')
age_df['gender'] = age_df['male'].map(lambda x: 'male' if x else 'female')
boneage_mean = age_df['boneage'].mean()
boneage_div = 2*age_df['boneage'].std()
age_df['boneage_zscore'] = age_df['boneage'].map(lambda x: (x-boneage_mean)/boneage_div)
age_df.dropna(inplace = True)
age_df.sample(3)
```

    12611 images found of 12611 total
    



* 完成之CSV檔案預覽
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>boneage</th>
      <th>male</th>
      <th>path</th>
      <th>exists</th>
      <th>gender</th>
      <th>boneage_zscore</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2770</th>
      <td>4650</td>
      <td>132</td>
      <td>0</td>
      <td>.\resize\4650.png</td>
      <td>True</td>
      <td>female</td>
      <td>0.056812</td>
    </tr>
    <tr>
      <th>10121</th>
      <td>12828</td>
      <td>120</td>
      <td>0</td>
      <td>.\resize\12828.png</td>
      <td>True</td>
      <td>female</td>
      <td>-0.088883</td>
    </tr>
    <tr>
      <th>4176</th>
      <td>6195</td>
      <td>162</td>
      <td>0</td>
      <td>.\resize\6195.png</td>
      <td>True</td>
      <td>female</td>
      <td>0.421048</td>
    </tr>
  </tbody>
</table>
</div>



* 展示資料分佈狀況
```python
age_df[['boneage','male', 'boneage_zscore']].hist(figsize = (10, 5))
```




    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x000000000FE5DE88>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x00000000100AEA88>],
           [<matplotlib.axes._subplots.AxesSubplot object at 0x00000000100E9448>,
            <matplotlib.axes._subplots.AxesSubplot object at 0x000000001011FE08>]],
          dtype=object)
          
         
          
![](https://i.imgur.com/igFzMRG.png)



* 按照boneage & 性別分布取8000筆資料
```python
age_df['boneage_category'] = pd.cut(age_df['boneage'], 10)
new_age_df = age_df.groupby(['boneage_category', 'male']).apply(lambda x: x.sample(400, replace = True)
                                                      ).reset_index(drop = True)
print('New Data Size:', new_age_df.shape[0], 'Old Size:', age_df.shape[0])
new_age_df[['boneage', 'male']].hist(figsize = (10, 5))
```

    New Data Size: 8000 Old Size: 12611
    




![](https://i.imgur.com/7vO8mhE.png)


* 將Training & validation 以3:1比例分開

```python
from sklearn.model_selection import train_test_split
train_df, valid_df = train_test_split(new_age_df, 
                                   test_size = 0.25, 
                                   random_state = 2018,
                                   stratify = new_age_df['boneage_category'])
print('train', train_df.shape[0], 'validation', valid_df.shape[0])
```

    train 6000 validation 2000
    
* 圖檔預處理

```python
from keras.preprocessing.image import ImageDataGenerator
IMG_SIZE = (256, 256)
core_idg = ImageDataGenerator(samplewise_center=True, 
                              samplewise_std_normalization=True, 
                              horizontal_flip = True, 
                              vertical_flip = False, 
                              height_shift_range = 0.1, 
                              width_shift_range = 0.1, 
                              rotation_range = 10, 
                              shear_range = 0.05,
                              fill_mode = 'nearest',
                              zoom_range=0.15)
```
* 載入圖檔

```python
def flow_from_dataframe(img_data_gen, in_df, path_col, y_col, **dflow_args):
    base_dir = os.path.dirname(in_df[path_col].values[0])
    print('## Ignore next message from keras, values are replaced anyways')
    df_gen = img_data_gen.flow_from_directory(base_dir, 
                                     class_mode = 'sparse',
                                    **dflow_args)
    df_gen.filenames = in_df[path_col].values
    df_gen.filepaths.extend(df_gen.filenames)
    df_gen.classes = np.stack(in_df[y_col].values)
    df_gen.samples = in_df.shape[0]
    df_gen.n = in_df.shape[0]
    df_gen._set_index_array()
    df_gen.directory = '' # since we have the full path
    print('Reinserting dataframe: {} images'.format(in_df.shape[0]))
    return df_gen
```


```python
train_gen = flow_from_dataframe(core_idg, train_df, 
                             path_col = 'path',
                            y_col = 'boneage_zscore', 
                            target_size = IMG_SIZE,
                             color_mode = 'grayscale',
                            batch_size = 64)

valid_gen = flow_from_dataframe(core_idg, valid_df, 
                             path_col = 'path',
                            y_col = 'boneage_zscore', 
                            target_size = IMG_SIZE,
                             color_mode = 'grayscale',
                            batch_size = 64) # we can use much larger batches for evaluation
# used a fixed dataset for evaluating the algorithm
test_X, test_Y = next(flow_from_dataframe(core_idg, 
                               valid_df, 
                             path_col = 'path',
                            y_col = 'boneage_zscore', 
                            target_size = IMG_SIZE,
                             color_mode = 'grayscale',
                            batch_size = 500)) # one big batch
```

    ## Ignore next message from keras, values are replaced anyways
    Found 0 images belonging to 0 classes.
    Reinserting dataframe: 6000 images
    ## Ignore next message from keras, values are replaced anyways
    Found 0 images belonging to 0 classes.
    Reinserting dataframe: 2000 images
    ## Ignore next message from keras, values are replaced anyways
    Found 0 images belonging to 0 classes.
    Reinserting dataframe: 2000 images
    
* 確認載入的圖檔

```python
t_x, t_y = next(train_gen)
fig, m_axs = plt.subplots(4, 4, figsize = (16, 16))
for (c_x, c_y, c_ax) in zip(t_x, t_y, m_axs.flatten()):
    c_ax.imshow(c_x[:,:,0], cmap = 'bone', vmin = -3, vmax = 3)
    c_ax.set_title('%2.0f months' % (c_y*boneage_div+boneage_mean))
    c_ax.axis('off')
```

![](https://i.imgur.com/hWUsMpO.png)

* MobileNet 參數設定

```python
from keras.applications.mobilenet import MobileNet
from keras.layers import GlobalAveragePooling2D, Dense, Dropout, Flatten, BatchNormalization
from keras.models import Sequential
base_mobilenet_model = MobileNet(input_shape =  t_x.shape[1:], 
                                 include_top = False, 
                                 weights = None)
bone_age_model = Sequential()
bone_age_model.add(BatchNormalization(input_shape = t_x.shape[1:]))
bone_age_model.add(base_mobilenet_model)
bone_age_model.add(BatchNormalization())
bone_age_model.add(GlobalAveragePooling2D())
bone_age_model.add(Dropout(0.5))
bone_age_model.add(Dense(1, activation = 'linear' )) # linear is what 16bit did
from keras.metrics import mean_absolute_error
def mae_months(in_gt, in_pred):
    return mean_absolute_error(boneage_div*in_gt, boneage_div*in_pred)

bone_age_model.compile(optimizer = 'adam', loss = 'mse',
                           metrics = [mae_months])

bone_age_model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    batch_normalization_1 (Batch (None, 256, 256, 1)       4         
    _________________________________________________________________
    mobilenet_1.00_256 (Model)   (None, 8, 8, 1024)        3228288   
    _________________________________________________________________
    batch_normalization_2 (Batch (None, 8, 8, 1024)        4096      
    _________________________________________________________________
    global_average_pooling2d_1 ( (None, 1024)              0         
    _________________________________________________________________
    dropout_1 (Dropout)          (None, 1024)              0         
    _________________________________________________________________
    dense_1 (Dense)              (None, 1)                 1025      
    =================================================================
    Total params: 3,233,413
    Trainable params: 3,209,475
    Non-trainable params: 23,938
    _________________________________________________________________
    
* MobileNet 參數設定

```python
from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau
weight_path="{}_weights.best.hdf5".format('bone_age')

checkpoint = ModelCheckpoint(weight_path, monitor='val_loss', verbose=1, 
                             save_best_only=True, mode='min', save_weights_only = True)


reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.8, patience=10, verbose=1, mode='auto', epsilon=0.0001, cooldown=5, min_lr=0.0001)
early = EarlyStopping(monitor="val_loss", 
                      mode="min", 
                      patience=5) # probably needs to be more patient, but kaggle time is limited
callbacks_list = [checkpoint, early, reduceLROnPlat]
```

* 開始training，訓練參數置入 bone_age_weights.best.hdf5

```python
train_gen.batch_size = 16
bone_age_model.fit_generator(train_gen, 
                                  validation_data = (test_X, test_Y), 
                                  epochs = 10, 
                                  callbacks = callbacks_list)
```

    Epoch 1/10
    375/375 [==============================] - 1261s 3s/step - loss: 0.6289 - mae_months: 52.4197 - val_loss: 0.6183 - val_mae_months: 56.1310
    
    Epoch 00001: val_loss improved from inf to 0.61833, saving model to bone_age_weights.best.hdf5
    Epoch 2/10
    375/375 [==============================] - 1219s 3s/step - loss: 0.3611 - mae_months: 39.8186 - val_loss: 0.4336 - val_mae_months: 46.1007
    
    Epoch 00002: val_loss improved from 0.61833 to 0.43358, saving model to bone_age_weights.best.hdf5
    Epoch 3/10
    375/375 [==============================] - 1275s 3s/step - loss: 0.2738 - mae_months: 34.3837 - val_loss: 0.2607 - val_mae_months: 32.8493
    
    Epoch 00003: val_loss improved from 0.43358 to 0.26069, saving model to bone_age_weights.best.hdf5
    Epoch 4/10
    375/375 [==============================] - 1275s 3s/step - loss: 0.2304 - mae_months: 31.2175 - val_loss: 0.4354 - val_mae_months: 45.7760
    
    Epoch 00004: val_loss did not improve from 0.26069
    Epoch 5/10
    375/375 [==============================] - 1276s 3s/step - loss: 0.1979 - mae_months: 28.9312 - val_loss: 0.1548 - val_mae_months: 24.7830
    
    Epoch 00005: val_loss improved from 0.26069 to 0.15478, saving model to bone_age_weights.best.hdf5
    Epoch 6/10
    375/375 [==============================] - 1279s 3s/step - loss: 0.1733 - mae_months: 27.0213 - val_loss: 0.1598 - val_mae_months: 25.1762
    
    Epoch 00006: val_loss did not improve from 0.15478
    Epoch 7/10
    375/375 [==============================] - 1289s 3s/step - loss: 0.1580 - mae_months: 25.6752 - val_loss: 0.1208 - val_mae_months: 22.1466
    
    Epoch 00007: val_loss improved from 0.15478 to 0.12078, saving model to bone_age_weights.best.hdf5
    Epoch 8/10
    375/375 [==============================] - 1290s 3s/step - loss: 0.1391 - mae_months: 24.2044 - val_loss: 0.1435 - val_mae_months: 24.5172
    
    Epoch 00008: val_loss did not improve from 0.12078
    Epoch 9/10
    375/375 [==============================] - 1303s 3s/step - loss: 0.1306 - mae_months: 23.4019 - val_loss: 0.0976 - val_mae_months: 19.3900
    
    Epoch 00009: val_loss improved from 0.12078 to 0.09761, saving model to bone_age_weights.best.hdf5
    Epoch 10/10
    375/375 [==============================] - 1278s 3s/step - loss: 0.1241 - mae_months: 22.9744 - val_loss: 0.0910 - val_mae_months: 19.1049
    
    Epoch 00010: val_loss improved from 0.09761 to 0.09099, saving model to bone_age_weights.best.hdf5
    
### 預測資料

* 載入訓練參數 bone_age_weights.best.hdf5

```python
weight_path= 
bone_age_model.load_weights(weight_path)
```
* 開始預測

```python
pred_Y = boneage_div*bone_age_model.predict(test_X, batch_size = 16, verbose = True)+boneage_mean
test_Y_months = boneage_div*test_Y+boneage_mean
```

    500/500 [==============================] - 33s 67ms/step
    
* 結果放置於 pred_Y 和實際月份 test_Y_months 進行參數比對

```python
fig, ax1 = plt.subplots(1,1, figsize = (6,6))
ax1.plot(test_Y_months, pred_Y, 'r.', label = 'predictions')
ax1.plot(test_Y_months, test_Y_months, 'b-', label = 'actual')
ax1.legend()
ax1.set_xlabel('Actual Age (Months)')
ax1.set_ylabel('Predicted Age (Months)')
```
* 結果展示

![](https://i.imgur.com/efkQRmO.png)



```python
from sklearn.metrics import mean_absolute_error, median_absolute_error
print(mean_absolute_error(pred_Y, test_Y_months))
print(median_absolute_error(pred_Y, test_Y_months))
```

### 改影像前處理


## Reference
 * [VGG16](https://www.kaggle.com/kmader/attention-on-pretrained-vgg16-for-bone-age)
 * [InceptionV3](https://www.kaggle.com/kmader/pretrained-inceptionv3-for-bone-age)
 * [MobileNet](https://www.kaggle.com/kmader/mobilenet-for-bone-age)
 * [MobileNet Introduction](https://medium.com/@chih.sheng.huang821/%E6%B7%B1%E5%BA%A6%E5%AD%B8%E7%BF%92-mobilenet-depthwise-separable-convolution-f1ed016b3467)
 * [RestNet50](https://www.kaggle.com/suniliitb96/tutorial-keras-transfer-learning-with-resnet50)