owned this note
owned this note
Published
Linked with GitHub
# 2022 S.T. Yau High School Science Award (Asia)
> Research Report
## The Team
Registration Number: Comp-102
Name of team member: Chuang Po Cheng
School: Kaohsiung Municipal Senior High School
Country: Taiwan (The R.O.C)
Name of team member: Chiu Lu Ming
School: Kaohsiung Municipal Senior High School
Country: Taiwan (The R.O.C)
Name of team member: Hsu Yu Chia
School: Kaohsiung Municipal Senior High School
Country: Taiwan (The R.O.C)
Name of supervising teacher: Lu Cheng Liang
Job Title: Physics teacher
School: Kaohsiung Municipal Senior High School
Country: Taiwan (The R.O.C)
**Date** 31, August, 2022
## Title of Research Report
**Calculate when the stunning colorful cloud will appear.**
> Chuang Po Cheng, Chiu Lu Ming, Hsu Yu Chia
## Abstract
Cloud is one of the amazing natural phenomena on Earth. However, we often miss it because of the daily work or so. Therefore, we want to create a system to predict the stunning colorful clouds.
In simple words, we use the data from the internet to calculate it in our physics model. The data or variants we use, such as temperature, air pressure, humidity, and, the most critical, height of cloud, which we have attempted, to make an AI model to classify it, in order to estimate the height of the cloud, could be accessed automatically by Python web-crawling. As for the physics model, we use Cauchy’s equation to calculate the refraction index, a method presented by Thailand researchers, and some tricky method to recreate the route of the light travelling through the atmosphere.
After we calculate our physics model by the data and variants we accessed, we can present a time when the colorful cloud would appear, and appreciating it.
+ Keywords:
> Web-crawling, Physics model, Cloud, Light Refraction, Light Scatter, Nature Beauty.
--------------------------------------------------------------
## Acknowledgement
The authors would like to thank everyone who contributed to this study.
First of all, we are greatly thankful to the Mother Nature to give us such an incredible view, both appreciated and relaxed. Second, we’d like to appreciate the researchers who share the equations and the results about their researches in different subjects, including but not limited to physics, meteorology, and computer science, which help us to do this research. We also have great appreciation to Kaohsiung Senior High School for providing us with the environment to finish this research. Our thanks also go to Teacher Chung Yen Lin for giving us advice when we were facing difficulties. Last but not least, thanks for our supervisor, Teacher Lu Cheng Liang, helping us and giving us the opportunity to do this project.
--------------------------------------------------------------------------------------------------------------------------
## Commitments on Academic Honesty and Integrity
We hereby declare that we
1. are fully committed to the principle of honesty, integrity and fair play throughout the competition.
2. actually, perform the research work ourselves and thus truly understand the content of the work.
3. observe the common standard of academic integrity adopted by most journals and degree theses.
4. have declared all the assistance and contribution we have received from any personnel, agency, institution, etc. for the research work.
5. undertake to avoid getting in touch with assessment panel members in a way that may lead to direct or indirect conflict of interest.
6. undertake to avoid any interaction with assessment panel members that would undermine the neutrality of the panel member and fairness of the assessment process.
7. observe the safety regulations of the laboratory(ies) where the we conduct the experiment(s), if applicable.
8. observe all rules and regulations of the competition.
9. agree that the decision of YHSA(Asia) is final in all matters related to the competition.
**We understand and agree that failure to honour the above commitments may lead to disqualification from the competition and/or removal of reward, if applicable; that any unethical deeds, if found, will be disclosed to the school principal of team member(s) and relevant parties if deemed necessary; and that the decision of YHSA(Asia) is final and no appeal will be accepted.**
--------------------------------------------------------------------------------------------------------------------------
## Table of Contents
[2022 S.T. Yau High School Science Award (Asia)](#2022-S.T.-Yau-High-School-Science-Award-(Asia))
[The Team](##The-Team)
[Title of Research Report](##Title-of-Research-Report)
I [Abstract](##Abstract)
II [Acknowledgement](##Acknowledgement)
III [Commitments on Academic Honesty and Integrity](##Commitments-on-Academic-Honesty-and-Integrity)
[Table of Contents](##Table-of-Contents)
1. [Introduction](##Introduction)
- 1.1. [How the colorful cloud appears](###1.1-How-the-colorful-cloud-appears)
- 1.2. [Theory](###1.2-Theory)
- 1.2.1. [Light refraction and scatter](####1.2.1-Light-refraction)
- 1.2.2. [Refraction index](####1.2.3-Refraction-index)
- 1.2.3. [The relation between time and solar altitude](####1.2.3-The-relation-between-time-and-solar-altitude)
- 1.3. [Panoply](###1.3-Panoply)
2. [Approach](##2.-Approach)
- 2.1. [Motivation](###2.1-Motivation)
- 2.2. [Data Accessing - Sunrise and Sunset Times](###2.2-Data-Accessing---Sunrise-and-Sunset-Times)
- 2.2.1. [Data.gov.tw](####2.2.1-Data.gov.tw)
- 2.2.2. [Data access](####2.2.2-Data-access)
- 2.3. [Data Accessing - Pressure and Dry Temperature](###2.3-Data-Accessing-–-Pressure-and-Dry-Temperature)
- 2.3.1. [FORMOSAT-7](####2.3.1-FORMOSAT-7)
- 2.3.2. [Download the file](####2.3.2-Download-the-file)
- 2.3.3. [Analyze the file to extract the Information](####2.3.3-Analyze-the-file-to-extract-the-Information)
- 2.4. [Data Access - Cloud Height](###2.4-Data-Access-–-Cloud-Height)2.4 Data Access – Cloud Height
- 2.4.1. [AccuWeather](####2.4.1-AccuWeather)
- 2.4.2. [Data access](####2.4.2-Data-access)
- 2.5 [Data Access – Humidity](###2.5-Data-Access-–-Humidity)
-2.5.1 [Open Weather Data Platform](####2.5.1-Open-Weather-Data-Platform)
-2.5.2 [Data access](####2.5.2-Data-access)
- 2.6. [Refrction index](###2.6-Refraction-index)
- 2.7. [Simulation](###2.7-Simulation)
- 2.7.1. [Light route in atmosphere](####2.7.1-Light-route-in-atmosphere)
- 2.7.2. [Time-incident angle relation](####2.7.2-Time-incident-angle-trlation)
3. [Result](##3-Result)
4. [Refrences](##4-Reference)
5. [Reflection](##5-Reflection)
- 5.1 [Idea creator](###5.1-Idea-creator)
- 5.2 [Team leader](###5.2-Team-leader)
- 5.3 [Physics model](###5.3-Physics-model)
- 5.4 [Simulation](###5.4-Simulation)
- 5.5 [Funny accident](###5.5-Funny-accident)
- 5.6 [Result](###5.6-Result)
- 5.7 [Conclution](###5.7-Conclusion)
6. [Certification](##6-Certification)
-------------------------------------------------------
## Introduction
### 1.1 How the colorful cloud appears
First, we have to know how the color cloud appeared. The light from the sun is scattered during traveling through the atmosphere, and the light with a longer wavelength, usually red light, has a greater refraction angle. Therefore, the different wavelengths of light will reflect from the different horizontal levels of the color cloud, which makes the color different. Here comes the picture illustrating it. From the daily observation, we notice that color cloud usually appears after rain or typhoon, especially water vapor pressure.
The picture above is the sunset with some colorful clouds but without rich water vapor. How this scenery appears we think it’s because of the height of the cloud, making the light can have enough scattered before being reflected.
The picture above, in contrast, is of the rich water vapor in the atmosphere. Therefore, a multi-color cloud could be created even if in a low-level atmosphere. This kind of cloud is cumulonimbus, which usually appears in the low-level atmosphere. Note: this is pictured at 8/23, 18:30.
In the illustration above, we can see the different positions on the Earth, which symbolize the different time of the day, and has a different light route, the red line. The different light routes are caused by the different refraction in the atmosphere. However, different refraction phenomena are not only due to the different time, but also the different atmosphere variants, such as air pressure, air temperature, humidity, and so on. Fortunately, we got access to the Formosa No.7 satellite, which contained the data we need, including the air refraction index.
### 1.2 Theory
The theories we used can be categorized with light refraction, which both scatter and refraction are caused by the refraction index. Therefore, we use the following equation:
1. $$ N_{n}\times\sinθ_{n}=N_{n+1}\times\sinθ_{n+1}$$
where $N_n$ is the current medium (level) refraction index and $\theta_n$ is the incident angle. Similarly, $n+1$ is the next medium (level). The equation is given by [4]. And the below are some detailed descriptions.
#### 1.2.1 Light refraction and scatter
Light would refract through different media. However, different air densities or air molecules can be considered to be different medium, too. Therefore, the light refract in the atmosphere is the multi-scattering. Hence, we have to use integration to calculate it, which you can see it at equation 3. Besides, the density of atmosphere is an exponential distribution.
If the sunlight beam going through the triangle glass, it will produce a rainbow-like light phase, which is called scatter, and it is caused by the different wavelength. Or just like the picture in the below.
(picture)
#### 1.2.3 Refraction index
Since the atmosphere refraction index would be majority affected by the wavelength, our purpose as well, we use Cauchy Equation [3] to calculate our refraction index to decrease the miscalculation between different wavelengths. And the below is the Cauchy Equation:
2. $$ n_{air}(\lambda, T, v, p) \approx 1+( \frac {T}{77.6 \times 10^{−6}})(1+ \frac{7.52\times10^−3}{λ^2})(p+4810\times\ \frac{v}{T}) $$
where $\lambda$ is wavelength ($\nu m$), using 700 nm or 0.7 $\nu m$ as red-light wavelength, $T$ is temperature (degree Kelvin), $p$ is air pressure (mbar), and $v$ is the pressure of water vapor (mbar). However, we couldn’t get the $v$ directly, so we use the below equation [5]:
3. $$ v=-\frac{1}{\rho g} \int_{p_2}^{p_1}{M_p dp} $$
where $v$ is precipitable water vapor (cm), $g$ is the acceleration due to the earth’s gravity ( 986.665 (cm/sec/sec), $\rho$ is the density of liquid water ($g/cm^s$), $p$ is the atmospheric pressure along altitude ($mbar$),$P_1$ and $P_2$ are the atmospheric pressure at the surface and the top of the atmosphere respectively. $M_p$ is mixing ratio at the pressure level, $p$. The integration is from the surface at $P_1$ up to the pressure designated by $P_2$ which depends on the final altitude reached by the radiosonde observation. $M_p$ can be calculated by below:
4. $$ M_p=\frac{0.622\times\\ e}{p−e} $$
where $e$ is the actual vapor pressure ($mbar$). The actual vapor pressure is obtained as the product of the saturated vapor pressure ($e_s$) and the relative humidity ($RH$) at pressure $p$, or as the equation below:
5. $$ e=\frac{RH\times e_x}{100} $$
The saturated vapor pressure value depends only on the air temperature $T$ in degree Celsius. The saturated vapor pressure (in $mbar$) is calculated according to below:
6. $$ e_X=6.112 \exp(\frac{17.67 \times T}{T+243.5}) $$
After using the equation mentioned earlier, we can use the refraction index we calculated to simulate the routes in different light wavelengths. See **(2.6)** Simulation for more details.
#### 1.2.3 The relation between time and solar altitude
To compute solar altitude angle, we need the observer’s latitude, time, and date of the year. After accessing the above variants, we can compute it by the following formula:
7. $$\theta =\arcsin (\sin\delta\sin\phi+\cos\delta\cos\phi\cos \omega) $$
where $\delta$ is the declination angle, $\phi$ is the latitude angle, and $\omega$ is the hour angle.
The declination angle changing depends on seasons, due to the earth's revolution. It is the angle between the latitude of direct sunlight and the earth’s equator. The northern latitude to the earth’s equator is positive, and the opposite is negative. If the axis of the earth were not askew, the declination angle would not exist (the following picture). The declination angle depends on the date of the year(d), not related to the observer’s location. For instance, d=1 means 1 of January.
8. $$\delta=−0.40928\cos(\frac{2 \pi}{365(d+10)})$$
Declination angle can transform the local solar time into sun’s moving angle, noted that sun travel $2𝜋/24$ $rad.$ per hour. In the midday, declination angle is zero and therefore it can be calculated easily through the below equation, which $t$ is the local time.
9. $$ \omega =\frac{\pi}{12(t−12)}$$
By the above formulas, we could easily compute with Python, and draw the data as a linear graph. The following picture is a linear graph when the observer in $23.5°N$. The horizontal axis is time of the day (unit: hour). The vertical axis is solar altitude angle computed by the above formulas (unit: rad). The vertical axis is solar altitude angle computed by the above formulas (unit: rads).
The vertical axis is solar altitude angle computed by the above formulas (unit: rad). The blue line is the day of summer solstice; the rea one is the day of spring equinox; the green one is the day of winter solstice.
### 1.3 Panoply
In this research, we use a lot of data obtained from satellites or other open data platforms. These data were usually stored in some unfamiliar type, including NetCDF, HDF, and GRIB. Although these types can be read into Python, we should understand the framework of the data before coding. Panoply (https://www.giss.nasa.gov/tools/panoply/) is the best application released by NASA for data preprocessing in this situation. It can read several data types, including most of the open data in astronomy and meteorology. Panoply is able to turn single data not only into plots but also export into CSV.
> the intrface of `Panoply`
-------------------------------------------
## 2. Approach
### 2.1 Motivation
Why we want to make this project is because we don’t want to miss the colourful cloud, which is such a practical motive. Also, in our opinion, watching sunset or sunrise accompanied with the colourful cloud is a relaxing hobby for the students who are preparing for the college admission. However, the relaxing moment couldn’t be very long, and the poem below is a great present of it.
向晚意不適，驅車登古原。
夕陽無限好，只是近黃昏。 -- by 李商隱
Which also means:
Toward evening I felt uneasy
And rode up into the hills.
The sunset was indescribably beautiful
But night was falling fast. --[7]
TTherefore, we want to create a program that can predict or calculate when the colorful cloud would appear, aiming for the precise time. Hence, the project is birth.
### 2.2 Data Accessing - Sunrise and Sunset Times
<!--  -->
The most important cause of the stunning colorful cloud is the light right from the sun. Therefore, it is important for us to know the time of sunrise and sunset.
#### 2.2.1 Data.gov.tw
Data.gov.tw is an online open data platform established by the Taiwanese government in order to release data to the public based on “The Freedom of Government Information Law”. The platform provides multiple types of data, including CSV, XML, and JSON.
```flow
op0=>operation: Access the date from [Data.gov.tw](data.gov.tw)
op1=>operation: Encode `JSON` into list
op2=>operation: Access current time
op3=>operation: Access the value by current time
op4=>operation: Export into `CSV`
op0->op1->op2->op3->op4
```
#### 2.2.2 Data access
The platform provides a permanent link for users to download the latest data. After downloading the data in JSON from the link, we encode it into a dictionary in Python. In the dictionary, it is easy for us to access the current data by importing the current time with Time module. Due to the data which has already been sorted out in cities, we export the current data in CSV files. And below are the codes and the results of accessing the time of sunrise and sunset.
```python=
# sunset and sunrise
```
The result of accessing and sunrise time.
| |花蓮縣|宜蘭縣|高雄市|臺北市|
|---------|------|-----|-----|------|
|民用曙光始 |05:01 |04:59|05:08|04:59 |
| 日出時刻 |05:25 |05:23|05:32|05:24 |
| 方位 | 289 | 289 | 288 | 289 |
| 過中天 |12:00 |11:59|12:05|12:00|
| 仰角 | 83S | 82S | 84S | 82S |
| 日沒時刻 |18:34 |18:34|18:37|18:36|
### 2.3 Data Accessing – Pressure and Dry Temperature
Based on the below paragraph (**2.3.1 FORMOSAT-7**), we can access the data of current atmospheric pressure and temperature with Formosat-7. There are two different systems that we can access target data, including Formosat-7 TDPC, and Formosat-7 TROPS. After our discussion, Formosat-7 TDPC is our first choice, according to its stability and more advanced property.
```flow
op0=>operation: Request and obtain the structure of the website
op1=>operation: Analyze and enter the latest folder `URL`
op2=>operation: Analyze and access the latest value `URL`
op3=>operation: Download the data with the `URL`
op4=>operation: Format the file to the readable form
op5=>operation: Access the pressure and temperature value
op6=>operation: Store the value in list
op7=>operation: Export in `CSV`
op0->op1->op2->op3->op4->op5->op6->op7
```
The only way we can access the latest data is through its official download site. In this situation, we tend to use Python for our coding language, due to its rich module database, consisting of wget, request, and netCDF4.
#### 2.3.1 FORMOSAT-7
The Formosat-7 is a major international collaborative space program between Taiwan and the USA, aimed at creating a high-reliability meteorological satellite system. All six satellites of the Formosat-7 system orbit the Earth for about 97 minutes, and could provide about 4000 data per day between 50 degrees north and south latitudes. By measuring radio occultation signals, atmospheric parameters, including refractivity, dry pressure and temperature, humidity, and the electron concentration of the ionosphere can be derived. Users can approach the data from Taiwan Data Processing Center (https://tacc.cwb.gov.tw).
### 2.3.2 Download the file
First, we have to get the folder URL. In this situation, we have to import the Requests Module to access the website information. Due to the characteristic of the URL under this condition, the target we need to get was only the name of the latest file. After reaching the target with Beautiful Soup Module, we combine it with the base URL to get the folder URL. After then, we do the exact same steps we did above to get the file URL. Last but not least, we download the file with the file URL with Wget Module.
#### 2.3.3 Analyze the file to extract the Information
After achieving the file, we have to analyze and extract it to get the information we need. First, we analyze it with Panoply, an app released by [NASA](https://www.giss.nasa.gov/tools/panoply/). After we got the information we need, we import and read the file into python by netCDF4 module. The data we took out would be stored in the array and exported into CSV for later simulating.
|Pres<br>Temp|Mean sea level altitude of perigee point|Pressure|Temperature|
|------------|----------------------------------------|--------|-----------|
| 1 | 79.998245 | 0.0 | 0.0 |
| 2 | 79.97825 | 0.0 | 0.0 |
| 3 | 79.958244 | 0.0 | 0.0 |
| 4 | 79.91824 | 0.0 | 0.0 |
> The result of accessing pressure and temperature
``` python
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
import pandas as pd
import os
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
opt = webdriver.ChromeOptions()
# 加入 headers 資訊
opt.add_argument('--user-agent=%s' % user_agent)
webdriver = '/Users/chen950726/Downloads/chromedriver 2'
driver = webdriver.Chrome(webdriver, options=opt)
url = ['shorturl.at/ikoq9',
'https://www.accuweather.com/zh/tw/taipei-city/315078/hourly-weather-forecast/315078',
'https://www.accuweather.com/zh/tw/new-taipei-city/2515397/hourly-weather-forecast/2515397',
'https://www.accuweather.com/zh/tw/taoyuan-city/3369297/hourly-weather-forecast/3369297',
'https://www.accuweather.com/zh/tw/hsinchu-city/313567/hourly-weather-forecast/313567',
'https://www.accuweather.com/zh/tw/hsinchu-county/3369298/hourly-weather-forecast/3369298',
'https://www.accuweather.com/zh/tw/miaoli-county/3369299/hourly-weather-forecast/3369299',
'https://www.accuweather.com/zh/tw/taichung-city/315040/hourly-weather-forecast/315040',
'https://www.accuweather.com/zh/tw/changhua-county/3369300/hourly-weather-forecast/3369300',
'https://www.accuweather.com/zh/tw/nantou-county/3369301/hourly-weather-forecast/3369301',
'https://www.accuweather.com/zh/tw/yunlin-county/3369302/hourly-weather-forecast/3369302',
'https://www.accuweather.com/zh/tw/chiayi-city/312591/hourly-weather-forecast/312591',
'https://www.accuweather.com/zh/tw/chiayi-county/3369303/hourly-weather-forecast/3369303',
'https://www.accuweather.com/zh/tw/tainan-city/314999/hourly-weather-forecast/314999',
'https://www.accuweather.com/zh/tw/kaohsiung-city/313812/hourly-weather-forecast/313812',
'https://www.accuweather.com/zh/tw/pingtung-county/3369304/hourly-weather-forecast/3369304',
'https://www.accuweather.com/zh/tw/taitung-county/3369305/hourly-weather-forecast/3369305',
'https://www.accuweather.com/zh/tw/hualien-county/3369306/hourly-weather-forecast/3369306',
'https://www.accuweather.com/zh/tw/yilan-county/3369296/hourly-weather-forecast/3369296',
'https://www.accuweather.com/zh/tw/penghu-county/3369307/hourly-weather-forecast/3369307',
'https://www.accuweather.com/zh/tw/kinmen-county/2332525/hourly-weather-forecast/2332525',
'https://www.accuweather.com/zh/tw/lienchiang-/3369309/hourly-weather-forecast/3369309']
city_name = [
'基隆市', '台北市', '新北市', '桃園市', '新竹市',
'新竹縣', '苗栗縣', '台中市', '彰化縣', '南投縣',
'雲林縣', '嘉義市', '嘉義縣', '台南市', '高雄市',
'屏東縣', '台東縣', '花蓮縣', '宜蘭縣', '澎湖縣',
'金門縣', '連江縣'
]
start = time.time()
value_store = []
for i in range(0, 22):
print("現在正在取得", city_name[i], "的雲層高度資料")
driver.get(url[i]) # 更改網址以前往不同網頁
time.sleep(2)
value = driver.find_element(
By.XPATH, '//*[@id="hourlyCard0"]/div[2]/div/div[2]/div[2]/p[4]/span').text
value_store.append(int(value[0:len(value)-1]))
print(value_store[i])
print(city_name[i], '雲層高度', value)
time.sleep(1)
try:
os.remove('cloud.csv')
print("已刪除舊檔案")
except: # 如果 try 的內容發生錯誤，就執行 except 裡的內容
print('無法刪除舊的檔案 或 不存在舊檔案')
dict = {'縣市': city_name, '雲層高度': value_store}
df = pd.DataFrame(dict)
file_csv = os.open('cloud.csv', os.O_RDWR | os.O_CREAT)
df.to_csv('cloud.csv')
end = time.time()
print("本次檔案已成功建置")
print("執行時間：%f 秒" % (end - start))
driver.close() # 關閉瀏覽器視窗
```
### 2.4 Data Access – Cloud Height
In order to calculate the route of light, we need the position of the cloud to calculate the reflect angle of the light reflected from the cloud. There are more details at 2.7 Stimulate. Therefore, if we want to simulate the stunning clouds, it is important for us to obtain the height of the cloud.
#### 2.4.1 AccuWeather
There are many different ways to access the data, including commercial weather forecasts, aviation weather forecasts, and the data released from the satellite. AccuWeather is a company aimed at providing accurate commercial weather forecasts, which also runs a free, advertising-supported website (AccuWeather.com). The website is based on weather information from numerous sources, including National Weather Service (NWS) and other reliable meteorological organizations.
#### 2.4.2 Data access
> The proccess chart of accessing cloud height
There are several modules for users to crawl on the internet, including Requests, which we used in the above paragraph. AccuWeather.com is an international website, which also runs on a dynamic web page. Therefore, selenium is our first choice in order to read the information on a dynamic web page.
Firstly, we enter the website, where we want to access the target with Python and Selenium. Secondly, we acquire it with XPath, which can target our value. After we got the value, we stored it in the list until we finish running all the cities we want. Finally, we stored all the values in CSV.
|基隆市|臺北市|新北市|桃園市|新竹市|新竹縣|苗栗縣|臺中市|彰化縣|南投縣|雲林縣|
|-----|------|------|------|-----|-----|------|-----|------|-----|-----|
| 9.5 | 7.9 | 9.1 | 9.1 | 9.1 | 9.1 | 9.1 | 9.1 | 9.1 | 9.6 | 9.1 |
|嘉義縣|嘉義市|臺南市|高雄市|屏東縣|臺東縣|花蓮縣|宜蘭縣|澎湖縣|金門縣|連江縣|
|-----|-----|------|-----|------|------|-----|-----|------|-----|------|
| 9.1 | 9.1 | 9.1 | 9.1 | 9.1 | 10.2 | 9.1 | 1.8 | 10.2| 10.4 | 10.2|
> The result of accessing the height of cloud (km)
```python
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
import pandas as pd
import os
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15"
opt = webdriver.ChromeOptions()
# 加入 headers 資訊
opt.add_argument('--user-agent=%s' % user_agent)
webdriver = '/Users/chen950726/Downloads/chromedriver 2'
driver = webdriver.Chrome(webdriver, options=opt)
url = ['shorturl.at/ikoq9’,
'https://www.accuweather.com/zh/tw/taipei-city/315078/hourly-weather-forecast/315078',
'https://www.accuweather.com/zh/tw/new-taipei-city/2515397/hourly-weather-forecast/2515397',
'https://www.accuweather.com/zh/tw/taoyuan-city/3369297/hourly-weather-forecast/3369297',
'https://www.accuweather.com/zh/tw/hsinchu-city/313567/hourly-weather-forecast/313567',
'https://www.accuweather.com/zh/tw/hsinchu-county/3369298/hourly-weather-forecast/3369298',
'https://www.accuweather.com/zh/tw/miaoli-county/3369299/hourly-weather-forecast/3369299',
'https://www.accuweather.com/zh/tw/taichung-city/315040/hourly-weather-forecast/315040',
'https://www.accuweather.com/zh/tw/changhua-county/3369300/hourly-weather-forecast/3369300',
'https://www.accuweather.com/zh/tw/nantou-county/3369301/hourly-weather-forecast/3369301',
'https://www.accuweather.com/zh/tw/yunlin-county/3369302/hourly-weather-forecast/3369302',
'https://www.accuweather.com/zh/tw/chiayi-city/312591/hourly-weather-forecast/312591',
'https://www.accuweather.com/zh/tw/chiayi-county/3369303/hourly-weather-forecast/3369303',
'https://www.accuweather.com/zh/tw/tainan-city/314999/hourly-weather-forecast/314999',
'https://www.accuweather.com/zh/tw/kaohsiung-city/313812/hourly-weather-forecast/313812',
'https://www.accuweather.com/zh/tw/pingtung-county/3369304/hourly-weather-forecast/3369304',
'https://www.accuweather.com/zh/tw/taitung-county/3369305/hourly-weather-forecast/3369305',
'https://www.accuweather.com/zh/tw/hualien-county/3369306/hourly-weather-forecast/3369306',
'https://www.accuweather.com/zh/tw/yilan-county/3369296/hourly-weather-forecast/3369296',
'https://www.accuweather.com/zh/tw/penghu-county/3369307/hourly-weather-forecast/3369307',
'https://www.accuweather.com/zh/tw/kinmen-county/2332525/hourly-weather-forecast/2332525',
'https://www.accuweather.com/zh/tw/lienchiang-county/3369309/hourly-weather-forecast/3369309']
city_name = [ '基隆市', '台北市', '新北市', '桃園市', '新竹市', '新竹縣',
'苗栗縣', '台中市', '彰化縣', '南投縣', '雲林縣', '嘉義市',
'嘉義縣', '台南市', '高雄市', '屏東縣', '台東縣',
'花蓮縣', '宜蘭縣', '澎湖縣', '金門縣', '連江縣' ]
start = time.time()
value_store = []
for i in range(0, 22):
print("現在正在取得", city_name[i], "的雲層高度資料")
driver.get(url[i]) # 更改網址以前往不同網頁
time.sleep(2)
value = driver.find_element(
By.XPATH, '//*[@id="hourlyCard0"]/div[2]/div/div[2]/div[2]/p[4]/span').text
value_store.append(int(value[0:len(value)-1]))
print(value_store[i])
print(city_name[i], '雲層高度', value)
time.sleep(1)
try:
os.remove('cloud.csv')
print("已刪除舊檔案")
except: # 如果 try 的內容發生錯誤，就執行 except 裡的內容
print('無法刪除舊的檔案 或 不存在舊檔案')
dict = {'縣市': city_name, '雲層高度': value_store}
df = pd.DataFrame(dict)
file_csv = os.open('cloud.csv', os.O_RDWR | os.O_CREAT)
df.to_csv('cloud.csv')
end = time.time()
print("本次檔案已成功建置")
print("執行時間：%f 秒" % (end - start))
driver.close() # 關閉瀏覽器視窗
```
> The code of accessing cloud height
### 2.5 Data Access - Humidity
```flow
op0=>operation: Access the current data from open data platform
op1=>operation: Encode the JSON into array
op2=>operation: Access the value of humidity, county name, county serial number and store it into thelist
op3=>operation: Purify the wrong data
op4=>operation: Calculas the average humidity by county
op5 Exoirt humidity of station and average humidity of county into CSV
op0->op1->op2->op3->op4->op5
```
> the process of accessing humidiity
In the atmosphere, refraction is caused by spatial variations, including temperature, pressure ,and humidity, with humidity being the most important. Therefore, it is important for us to access the current humidity.
### 2.5.1 Open Weather Data Platform
Open Weather Data Platform (opendata.cwb.gov.tw) is an open data platform, founded by the Central Weather Bureau (CWB) in Taiwan for users to access the current data of meteorological information. It provides multiple different data, county weather forecast and meteorological observations data was included. In this report, we will be using meteorological observations data of [automatic weather stations](https://opendata.cwb.gov.tw/dataset/observation/O-A0001-001), which provides humidity in its dataset.
### 2.5.2 Data access
It is the same way for us to access the data as we did in 2.2.2. First of all, we reached the data from the URL and encode the JSON into the array. After reaching the data, we obtained the value that we need, including humidity, county name, county number, and the automatic weather station name. Getting all the values, the next step we did is to calculate the average humidity by county. First, we extract the humidity with the same county number and store it in list. Second, we counted the quantity of the same county number at the same time. After calculations, we can get the average humidity. At last, we export two data: humidity of station and average humidity of county.
|huminity county| NaN |臺北市|臺中市|基隆市|台南市|高雄市|新北市|
|---------------|-----|-----|------|-----|------|-----|----|
| 平均相對濕度 |-99.0|0.729|0.794 |0.772|0.860 |0.826|0.808|
| 縣市 |臺北市| 宜蘭縣 |宜蘭縣|新北勢|基隆市|新北市|
|-------|-----|--------|-----|-----|------|-----|
| 鄉鎮 |士林區| 頭城鎮 |頭城鎮|貢寮區|安樂區|萬里區|
| 測站名 |科教館|大溪漁港| 石城 | 澳底 |大武崙| 野柳 |
|相對溼度| 0.69| 0.73 | 0.70 |0.81 | 0.72 | 0.81|
```python
```
### 2.6 Refrction index
Mainly, we use python to compute the whole calculation, and csv file to connect different calculations. And the below are the calculations mentioned at [1.2].
First, the Cauchy’s equation mentioned at [1.2.3.1](##1.2.3-Refraction-index)
``` python
def Cauchy_equation(air_pressure,air_temperature,wavelength,relative_humidity):
T = air_temperature
w = wavelength
a = air_pressure
v = pressure_of_water_vapour(relative_humidity,T)
return 1+(0.0000776/T)*(1+(0.00752)/w/w)*(a+4810*v/T)
```
`air_temperature` and `air_pressure` are given by [Data Accessing - Pressure and Dry Temperature](###2.3-Data-Accessing-–-Pressure-and-Dry-Temperature), which is related with the altitude, different altitude layers would have different temperature and pressure.
``` python
def pressure_of_water_vapour(relative_humidity,temperature):
RH= relative_humidity
T = temperature
es=6.112*np.exp(17.67*T/(T+243.5)) # saturated vapor
e = RH*es/100 # actual vapor pressure
Mp = 0.662*e/(a-e) # mixing ratio at the pressure level, p
return -1/986.665*Mp
```
As for the integration at (1.2.3.2), because the temperature and pressure alter in the small gap, 20 meters, we choose it as the gap in numerical method for the integration, or as the code below.
```python
for i in range(0,3899):
air_pressure = float(pressure[i])
air_temperature = float(temperature[i])+273.0
wavelength = 700*1000.0
relative_humidity = 0.7
refractive_index.append(Cauchy_equation(
air_pressure = air_pressure,
air_temperature = air_temperature,
relative_humidity = relative_humidity,
wavelength = wavelength
)
)
```
Because the colorful cloud is often caused by the red-light, we use 700 nm as the wavelength of light. As for the humidity, we use the data accessed from [2.5 Data Access – Humidity](###2.5-Data-Access---Humidity).
### 2.7 Simulation
When thinking how to present the simulation, we didn’t have any access to the professional software which can run the complex simulation. We didn’t know how to conduct it as well. Therefore, we use the simple but powerful way “Matplotlib” to conduct it. And the [link](https://github.com/zack-404/Predict_Colourful_Cloud/blob/main/light_v4.py) is connected to the code.
#### 2.7.1 Light route in atmosphere
The main idea is to use the light vector reflecting from the target cloud to reverse the light route during traveling through the atmosphere. The fact that it would work is that the height of cloud is pretty low compared with the thick atmosphere. After we reverse the light route, we can calculate the incident angle and therefore access the time from the time-angle table at [2.4.2](####2.7.2-Time-incident-angle-trlation). Also noted that the main colorful cloud reflects the near-red light, sso we use the refraction index which has been adjusted to the red-light wavelength of 700 nm by the Cauchy Equation mentioned at [1.2.3](####1.2.3-The-relation-between-time-and-solar-altitude).
How we use the refraction index is as below equation, where
$N_n$ and $N_{n+1}$ is the refraction index in the current and the next level.
1. $$ N_{n}×\sin \theta_{n}=N_{n+1}×\sin \theta_{n+1} \rArr \sinθ_{n+1}=\frac{N_{n} \times\sin\theta_{n}}{N_{n+1}} $$
But there are some initial setups you need to know before understand the code. The observer $O$ is at $(0.0)$ and the target cloud $P$ we want to observe is at $(x,y)$. If we connect the $O$ and $P$ together, we can get the vector $\vec{OP}$ or $\vec{v}$ which can be used to calculate the reflect angle $\theta$.
After we have the reflect angle $\theta$, we can use the equation above, using the current and the next level refraction index $N_n$ to calculate the next level’s refraction angel $θ_{n+1}$, and continuously doing it until reaching the top of the atmosphere. As the result, we can connect all the vector together, and get the picture below to simulate the light route.
#### 2.7.2 Time-incident angle table
We calculate the solar altitude by the formula (1.2.3). The incident angle is the solar altitude angle’s supplementary angle. And we can easily compute it and list the correspondence between the incident angle and time of the day. We build two lists, one is the time and the other is the solar altitude angle. Between two adjacent materials, there are 15 minutes intervals. Because we only discuss about the cloud of dawn, we search the data from midday to sunset. While the incident angle (rads) is given, the program would find the location of it and print two times which the incident angle given is within the data corresponded by.
```python
m=int((sunset-sunrise)/30)
m0=int((sunset-sunrise)/15)
while 1<int(m0-m) :
if vector < solaraltitude[int((m+m0)/2)-1]:
m0=int((m+m0)/2)
else:
m=int((m+m0)/2)
```
Above is how we search the location of data from two lists. While sunrise is time of sunrise, sunset is time of sunset, and vector is the incident angle given. Because our time gap is 15-minute, m is the location of midday data and m0 is the location of sunset in the lists. Noted that our predict time is a 15 minutes interval because of the unpredictable variants such as user favor, and mis simulation.
We take the following time-incident angle correspondence for example:
|Time|Incident angle|
|-|-|
|10:00|1.4114465988757618|
|18:15|1.4669588037239287|
If the best appreciation incident angle (rads) is 1.45, the code will print 18:00~18:15. In this way, the user can get the most appropriate time to watch the phenomenon.
------------------------------------------------------------------------------------------------------------------------------------------------
## 3 Result
As for the result, we use our program to predict the colourful cloud in these days. Here comes our example.
Example 1:
| Date | Picture time | Predict time | Temperature | Air pressure | Relative humidity | Incidence angle | Simulate time |
| ---- | ------------ | ------------- | ----------- | ------------ | ----------------- | --------------- | ------------- |
| 8/26 | 18:23 | 18:15~18:30 | 31° | 1008.1 | 70% | 1.509887 | |
Example 2:
| Date | Picture time | Predict time | Temperature | Air pressure | Relative humidity | Incidence angle | Simulate time |
| ---- | ------------ | ------------- | ----------- | ------------ | ----------------- | --------------- | ------------- |
| 8/27 | 18:25 | 18:15~18:30 | 31° | 1005.6 | 76% | 1.507151 | |
Example 3:
| Date | Picture time | Predict time | Temperature | Air pressure | Relative humidity | Incidence angle | Simulate time |
| ---- | ------------ | ------------- | ----------- | ------------ | ----------------- | --------------- | ------------- |
| 8/27 | 18:52 | 18:15~18:30 | 31° | 1005.6 | 76% | 1.507151 | |
Example 4:
| Date | Picture time | Predict time | Temperature | Air pressure | Relative humidity | Incidence angle | Simulate time |
| ---- | ------------ | ------------- | ----------- | ------------ | ----------------- | --------------- | ------------- |
| 8/30 | 18:17 | 18:15~18:30 | 27° | 1088.1 | 93% | 1.504601 | |
Why we use the 15 minutes as the predicted time gap is that during the 15 minutes gap, you can enjoy the diversity of the sky, which would absolutely blow your mind, and make you admire the beauty of nature with no doubt. Besides, another reason is that the colorful cloud would sometime keep longer than 15 minutes, the example 2 and 3 are great examples about it. But during the 15 minutes gap we predict, there would be some colorful clouds in the sky. Although the examples seem to perform pretty well, to be honest, there are still some conditions couldn’t be tested in this model, especially extreme weather conditions, such as typhoon, afternoon thundershower, and the worse air condition.
To sum up, mother nature is unpredictable. This model just presents a kind of method that can predict it very slightly. The colorful cloud could appear any time. This model just gives you an advice. Hope you enjoy the report and the colorful cloud.
## 4 References
[1] Bruneton, E., & Neyret, F. (2008). Precomputed Atmospheric Scattering. Computer Graphics Forum, 27(4), 1079–1086. [https://doi.org/10.1111/j.1467-8659.2008.01245.x](https://doi.org/10.1111/j.1467-8659.2008.01245.x)
[2] Lopes, Diogo A. R. Fernandes, António Ramires (Ed.). (2014). Atmospheric Scattering - State of the Art. Instituto Politécnico de Leiria. [https://repositorium.sdum.uminho.pt/bitstream/1822/30959/1/epcg2014_final%20submission_no_numbers_19.pdf](https://repositorium.sdum.uminho.pt/bitstream/1822/30959/1/epcg2014_final%20submission_no_numbers_19.pdf)
[3] Trager, Scott. "The Earth's atmosphere: seeing, background, absorption & scattering" (PDF). S.C. Trager. Retrieved 31 May 2022.
[4] Jenkins, F. A., White, H. E., & Brukhard, D. G. (1958). Fundamentals of Optics. American Journal of Physics, 26(4), 272. [https://doi.org/10.1119/1.1996127](https://doi.org/10.1119/1.1996127)
[5] Phokate, S. (2017). Atmospheric water vapor: Distribution and Empirical estimation in the atmosphere of Thailand. Journal of Physics: Conference Series, 901, 012051. [https://doi.org/10.1088/1742-6596/901/1/012051](https://doi.org/10.1088/1742-6596/901/1/012051)
[6] A Solar Altitude Angle Model for Efficient Solar Energy Predictions. (2020, March 4). In Dr. Sergio Herrería-Alonso. [https://www.mdpi.com/1424-8220/20/5/1391/pdf?version=1583316673](https://www.mdpi.com/1424-8220/20/5/1391/pdf?version=1583316673)
------------------------------------------------------------------------------------------------------------------------------------------------
## 5 Reflection
In this project, what I’m responsible can be separate to three different part: idea creator, leader, simulation and physics model.
### 5.1 Idea creator
This idea has been in my head about a year or so. It came up to me because I really like the colorful cloud, which no one may doubt it, I believe. At first, my model is that create an Artificial Visual model to classify the cloud and then estimate the height of the cloud, as the abstract said. After that, we can use the physics model to calculate it. However, I’m not familiar with it, and no interest in artificial intelligence as well because of the unpredictable process during the hidden layer, therefore I gave up, then. Fortunately, I met my teammate 莊博丞, who is interest in it. At the same time, I’m really appreciated that our supervisor brings up the opportunity which let us meet.
### 5.2 Team leader
Because I’m the idea creator, I, naturally, became the leader of the project. When I was distributing the responsibility, I distributed with the ability, disability, and interest as far as I knew. And below are the ability and interest to my teammate I knew:
| name | ability |
|-------------|--------------------------------------------------------------------------------------------------------------------------------|
| Yoga Hsu | 1. good at mathematics and physics calculation <br> 2. lack of coding and programming experience<br> 3. have a not bad friendship with me|
|Pochen Chaung| 1. familiar with python <br> 2. interest and experienced in artificial intelligence <br> 3. having more time than two of us, because of the grades|
By the way, Yoga Hsu is my friend and classmate in school. He said he want to learn programming and coding, but he lacked action. Therefore, I invited him to this project.
Conclude all the mentioned above, I distributed the responsibility as below, separated into two parts:
+ `Cloud classify model` : Pochang Chuang
+ `Physics model` : Yoga Hsu, and me
Also, this is the very first time that I lead a project. Therefore, I’m not pretty sure whether I have been a food leader in this project, so I ask them to give each other some feedback, including me. And the below are the feedback to me:
+ Yoga:
+ 莊博丞:
### 5.3 Physics model
Before making the physics model, I had to do some research in the precious research in the field we are going to, atmospheric refraction. I did find some detailed paper about it. However, it’s a bit too complicated and too hard for a non-native English reader, me. I took a lot of time on exploring how to **read the paper efficiently** rather than let the paper looking at me. In the end, I found a useful tips for me to understand it, taking note. After taking note on the paper, I began to fully understand what variable works for what. I consider it to the most vital ability for the upcoming college or academic life.
At first, Pochang had the access to the FORMOSAT-7, which contained the refractivity index in every altitude. However, after I used it in simulation, it turned out that this refractivity can’t use in red-light refraction, due to the wavelength. I was desperate, because I did not find any useful information in atmospheric refraction. In the last week before the submission deadline, something miracle happened, I found the Cauchy’s equation in Wikipedia. I knew Wikipedia’s information can’t be used in the academic report, so I go to find the relative reference. And guess what, the equation is true. After conducting it in the refraction model, it worked a lot better than the refraction index given by the FORMOSAT-7 due to the red-light longer wavelength.
All in all, I only used the three equations to simulate the light:
1. $$ N_{n}\times\sinθ_{n}=N_{n+1}\times\sinθ_{n+1} $$
2. $$ n_{air}(λ, T, v, p)≈1+(\frac{77.6\times10^{−6}}{{T}})(1+\frac{7.52\times10^−3}{λ^2})(p+4810\times\frac{v}{T}) $$
3. $$ v=-\frac{1}{ρg}∫_{p_2}^{p_1}M_pdp $$
which mentioned at (1.2.0.1), (1.2.3.1), (1.2.3.2).
### 5.4 Simulation
Except for the role of the leader, another duty is do the simulation. This is my first time that use python as the main developing language, I usually use java as my main language, instead, and this is my very first time, again. In the simulation I learned how to make a graph from matplotlib.pyplot and read the csv file by csvmodule in python.
+ `matplotlib.plot`, which I thought as the intro to some mathetic problem, statistic, and data visualization.
+ `csv` module, I see it as some both frontend and beckend basic in web development.
In this simulation, I’m not only ding it as the literally meaning, but the diverse learning for python language. Honestly, for the one who is a little familiar with the programming logic, it is quite a easy task to learn python syntax and the use of module.
I really appreciate the progress of learning.
Yet, I still can’t simulate the light route, but, however, I can use the incident angle to reverse the incident angle and therefore predict the precise time, which means it can match the result of the time-incident angle table. The light simulation is still in progress.
### 5.5 Funny accident
When I was writing this report in .md, I found out that writing the content table manually is boring, inefficient, and annoying. Thanks to the csv module we used in the program, I came up with an idea that use it to read the .md file and generate a content table. I truly finished it the day after the idea created. But something unfortunately or surprisingly happened, there is a Visual Studio Code extension that work well on that. What more, I uninstalled it a few days ago. My mind just turning into blank when I noticed that. However, LEARING is still a wonderful thing.
### 5.6 Result
As for the result, we at least passed the audition, and the further one is still unknown. It will be updated very soon.
### 5.7 Conclusion
At first, this team was built up for an activity in Israel, but some issues just came upon to us just a month before departure, which made us disable to go there in the end. However, the team just turned to this project.
During this project, we met a lot of obstacles. Every progress we made, we met obstacles at least one. The first we met was the artificial visual model problem couldn’t build up successfully. The second was that I couldn’t realize how to build the physics model to simulate the light by the approach in the paper, not believing that I could not figure out the basic physics affect. Last but not least, it seemed that my teammate didn’t have much motivation then I thought, which could be the upset of the previous obstacles, but I don’t know how to cheer them up.
Not until the last two week before the submission deadline did our team be truly ignited, discussing and progressing 'til midnight. We figure out that we can access to the AccuWeather for the height of the cloud. Refraction index can be calculated by Cauchy equation. The variants needed in the model can be access from the internet, which greatly appreciated. Thanks to my teammate and the useful internet access, it solved metrically, totally.
In the end we made it. We made the program which can predict the colorful cloud. It may be some bug in our model, but during our testing, it all work well.
## 6 Certification
Google Drive
Gist
Clipboard
Sign in with Wallet
