rc2021-ICJAISV74 Note 2

# rc2021-ICJAISV74 Note 2 ###### tags: `mlrc2021` ## Roadmap 1. Run RS & US results and compare with [Zhan2021](). 2. Contact with *Zhan* again. Prepare the original process and `sklearn.StandardScaler` (Suggested by *Zhan*). - Ask for RS & US query history. Check the AUBC is different or not. - How do you run `alipy.BMDR` and `alipy.LAL` for multi-class classification problems. (Comment the `if len(ul) != 2: ...` condition?) - Ask for several query stratigies AUBC(acc) results. - Ask for the code, if possible. - Which `method` of `libact.US-1`? `{'lc', 'sm', 'entropy'}, (default='lc')`? Or other detail in parameters? - Detail of using multiple base model for BSO (Suggested by *Zhan*). - What is *saturated performance* of datasets in *page 4682*. - What is *BSO results on some datasets are not as large as one might expect* in *page 4682*? How can we evaluate it? - More detail on base model in QBC. 3. Share my experience. - Run multicore (Pool) of QBC and QUIRE. - ## Supplementary 1. Ask for RS & US query history ```python= (Pdb) p results["indices"][:90] # iris array([106, 91, 28, 117, 13, 121, 128, 35, 7, 45, 123, 102, 137, 142, 27, 22, 125, 92, 129, 34, 23, 29, 15, 6, 33, 140, 52, 5, 58, 98, 41, 99, 109, 149, 65, 17, 96, 84, 130, 9, 57, 1, 42, 61, 26, 66, 95, 87, 70, 82, 116, 93, 32, 56, 143, 2, 10, 131, 31, 20, 21, 135, 12, 108, 0, 64, 48, 39, 132, 145, 85, 18, 89, 50, 16, 51, 80, 111, 107, 38, 81, 53, 79, 24, 126, 73, 100, 4, 97, 115]) ``` 2. How do you run `alipy.BMDR` ```python= # alipy/query_strategy/query_labels.py#L1089 1089 if len(ul) != 2: 1090 raise ValueError(f"This query strategy is implemented for binary classification only, " 1091 f"but {len(ul)} classes are detected.") ``` 3. Ask for several query stratigies AUBC(acc) results. See EXCEL---TABLE3. 4. Ask for the code, if possible. Would you mind to send the partial of example code of RS or US? I may try to find some difference between us. 5. Detail of using multiple base model for BSO (Suggested by *Zhan*). - Where should I train with `SVC(), LinearSVC(), LogisticRegression(), GaussianProcessClassifier()`? - What is your `sklearn.__version__`? ## Research Log I list the results pass test or not of random sampling (RS) in *TABLE 3*. | dataset | [Zhan2021] | No preprocessing | sklearn.StandardScaler | sklearn.Normalizer | | ------------ | ---------- | ---------------- | ---------------------- | ------------------------- | | appendicitis | 0.836 | No 0.79 | Yes 0.838 | No 0.786 | | sonar | 0.617 | No 0.67 | No 0.755 | No 0.51 | | iris | 0.835 | No 0.93 | No 0.924 | Yes(?) 0.845 | | wine | 0.858 | No 0.95 | No 0.960 | Yes(?) 0.870 | | parkinsons | 0.840 | Yes 0.84 | Yes 0.837 | No 0.75 | | ex8b | 0.866 | No 0.835 | No 0.885 | No 0.845 | | seeds | 0.862 | No 0.92 | No 0.907 | No 0.88 | | glass | 0.387 | No 0.48 | No 0.611 | Yes 0.395 | | thyroid | 0.696 | No 0.71 | No 0.919 | Yes 0.695 | | heart | 0.808 | Yes 0.80 | Yes 0.806 | No 0.785 | | haberman | 0.727 | Yes 0.72 | Yes 0.731 | Yes 0.731 | | ionosphere | 0.901 | No 0.87 | No 0.918 | No 0.657 | | clean1 | 0.649 | No 0.580 | No 0.819 | No 0.554 | | breast | 0.954 | No 0.959 | Yes 0.956 | No 0.942 | | wdbc | 0.952 | Yes(?) 0.955 | Yes 0.954 | No 0.932 | | r15 | 0.877 | No 0.976 | No 0.975 | No 0.437 | | australian | 0.846 | Yes(?) 0.851 | Yes 0.848 | No 0.838 | | diabetes | 0.736 | Yes(?) 0.740 | Yes(?) 0.741 | No 0.706 | | mammographic | 0.819 | No 0.772 | No 0.809 | No 0.503 | | ex8a | 0.838 | No 0.696 | No 0.855 | No 0.61 | | vehicle | 0.567 | Yes(?) 0.563 | No 0.691 | No 0.38 | | tic-tac-toe | 0.87 | Yes 0.87 | Yes 0.87 | Yes 0.87 | | german | 0.726 | Yes(?) 0.73 | No 0.732 | No 0.699 | | splice | 0.806 | No 0.793 | Yes 0.807 | No 0.542 | | gcloudb | 0.893 | No 0.882 | Yes(?) 0.895 | No 0.769 | | gcloudub | 0.942 | No 0.928 | Yes(?) 0.944 | No 0.886 | | checkerboard | 0.978 | No 0.836 | Yes(?) 0.977 | No 0.988 | | phishing | 0.926 | No 0.934 | No 0.938 | linux15-13/clais-noGPU-50 | | d31 | 0.582 | No 0.954 | No 0.955 | No 0.519 | | spambase | 0.685 | No 0.914 | No 0.914 | No 0.878 | | banana | 0.895 | No(?) 0.893 | No(?) 0.893 | No 0.738 | | phoneme | 0.822 | Yes(?) 0.821 | Yes(?) 0.821 | No 0.79 | | texture | 0.666 | No 0.966 | No 0.966 | No 0.716 | | ringnorm | 0.976 | Yes(?) 0.976 | Yes(?) 0.976 | linux15-13 | | twonorm | 0.976 | No(?) 0.975 | No(?) 0.975 | linux15-13/linux5-20 | I also list the results pass test or not of uncertainty sampling (US) in *TABLE 3*/*TABLE 5* in [Zhan2020](). | dataset | [Zhan2020] | No preprocessing | sklearn.StandardScaler | sklearn.Normalizer | | ------------ | ---------- | ---------------- | ---------------------- | ------------------ | | appendicitis | 0.845 | No 0.786 | Yes 0.839 | | | sonar | 0.76 | No 0.698 | No 0.775 | | | iris | 0.87 | No 0.928 | No 0.937 | | | wine | 0.958 | No 0.94 | No 0.966 | | | parkinsons | #N/A | 0.854 | 0.854 | | | ex8b | 0.892 | No 0.846 | Yes(?) 0.899 | | | seeds | #N/A | 0.913 | 0.913 | | | glass | #N/A | 0.491 | 0.612 | | | thyroid | 0.702 | No 0.73 | No 0.936 | | | heart | 0.765 | No 0.814 | No 0.816 | | | haberman | 0.725 | Yes 0.723 | Yes 0.729 | | | ionosphere | 0.914 | No 0.878 | No 0.927 | | | clean1 | 0.826 | No 0.624 | No 0.839 | | | breast | 0.955 | No 0.961 | Yes(?) 0.957 | | | wdbc | #N/A | 0.965 | 0.965 | | | r15 | #N/A | 0.95 | 0.95 | | | australian | 0.846 | Yes(?) 0.851 | Yes(?) 0.85 | | | diabetes | 0.744 | No 0.754 | Yes 0.745 | | | mammographic | #N/A | 0.779 | 0.814 | | | ex8a | 0.85 | No 0.711 | No 0.879 | | | vehicle | 0.539 | No 0.559 | No 0.696 | | | tic-tac-toe | #N/A | 0.871 | 0.87 | | | german | 0.742 | No 0.737 | Yes(?) 0.738 | | | splice | 0.809 | No 0.803 | No 0.82 | | | gcloudb | 0.887 | Yes 0.886 | No 0.898 | | | gcloudub | 0.951 | No 0.936 | Yes(?) 0.953 | | | checkerboard | 0.91 | No 0.85 | No 0.984 | | | Phishing | #N/A | | | | | d31 | #N/A | cl7-25 | linux7-20 | | | spambase | #N/A | cl7-25 | linux7-20 | | | banana | #N/A | cl7-25 | linux7-20 | | | phoneme | #N/A | | | | | texture | #N/A | | | | | ringnorm | #N/A | | | | | twonorm | #N/A | | | | Example. **Yes** *appendicitis* `sklearn.StandardScaler` ![](https://i.imgur.com/aMjkfxp.png) Example. **No** *seeds* ![](https://i.imgur.com/T6ew5Ni.png) Example. **Yes(?)** *wine* `sklearn.Normalizer` ![](https://i.imgur.com/6b2erjX.png) Example. **No(?)** *twonorm* `sklearn.StandardScaler` ![](https://i.imgur.com/F8bMFYL.png)