--- tags: 前端 --- # 台股分析系統建置 - 使用JS ## 初始化專案 ### 安裝[node.js - 12.0版以上](https://nodejs.org/en/) Javascript一般來說只會在網頁上出現,`node.js`出現使其也能在Windows 64bit等作業系統的上運行,因此又稱之為`Javascript的運行環境`,其終端機套件管理工具為npm。 ### 安裝[vscode](https://code.visualstudio.com/Download) 最強Javascript文字編輯器,請選擇Windows > System Installer > 64 bit的版本進行安裝,將資料夾拉到桌面圖示上方或`vscode [資料夾]`開啟專案 ### 依序輸入指令 - `ctrl+~` 在vscode裡開關終端機的快捷鍵 - `npm install --global yarn nodemon pm2` 安裝基於node.js開發的CLI工具 - `yarn install` 安裝`package.json`裡所有相依套件 - `npm run download` 下載台股前一百檔並存到資料庫,因爬蟲速度有時候很慢,若覺得資料足夠時可以先按`ctrl+c`中斷執行 - `npm run start` 啟動站台,`ctrl+左鍵`點擊網址來打開網頁 :::danger `sqlite3`因受到node-pre-gyp遷移到@mapbox/node-pre-gyp,且尚未支援linux,因此`sqlite3`已從5.0.2退版到4.2.0 ::: ### VSCODE推薦擴充外掛(面板開啟快捷鍵`Ctrl+Shift+X`) 1. `Markdown All in One`讓md檔可以預覽核選方塊、數學式等進階功能 2. `sqlite` 查看資料庫與下指令擴充套件 3. `Vetur` 支援擴充檔名`.vue`的編譯、樣板、格式化 4. `Python` 支援擴充檔名`.ipynb`,右上角一鍵執行或`shift+enter`執行`.py`檔 - 查詢CSS屬性請參照[CSS 菜單](https://hackmd.io/fNbgc3_LSG20SlWKjYaejQ) - 查詢Bootstrap框架請參照[Bootstrap 菜單](https://hackmd.io/fNbgc3_LSG20SlWKjYaejQ) - 更多開發工具說明請參照[GIT+VSCODE+MD 攻城獅必備工具](https://hackmd.io/T56rQ0CaSj2uuZa8R3gTFw) ## 認識NPM 初始化node.js專案的指令 - `npm init -y` 建立一個以英文命名的空資料夾後,用此指令產出設定檔 其中`-y`代表所有設定選項全部選yes,並產出`package.json`檔 - `npm install`或`npm i` 若資料夾中已經有`package.json`,此指令可自動安裝所有相依套件 - `npm install --globl yarn`或`npm i -G yarn` 若要安裝終端機(cli)的指令,像是比npm更快的套件管理工具yarn, 安裝後便可改用`yarn add [套件]`來安裝套件 `package.json`分別有`scripts`、`dependencies`、`devDependencies`三大類,其中 - `scripts` 紀錄自訂指令 如`{"start": "yarn && node scripts/start.js"}` 當輸入`npm run [自訂指令]`後,便會接連執行 - 檢查套件是否缺漏(yarn) - 執行腳本(node scripts/start.js) - `dependencies` 產品套件 輸入安裝指令`npm install [套件]`或`yarn add [套件]` 如bootstrap, react, vue等套件,會通通記錄到這邊 - `devDependencies` 開發工具 開發工具編譯或是優化的套件 `npm install [套件] --save-dev`或`npm i -D [套件]`或`yarn add -D [套件]` 比如sass, babel,通常會記錄到這邊 - `npm config` 若公司網路需要設定憑證才給放行,則需要下指令 - `npm config list` 列出所有NPM設定 - `npm config set cafile "[.cer檔的路徑]"` 設定憑證檔案的路徑 ### JS引入模組的方法 CommonJS ```javascript= const fs = require('fs'); const path = require('path'); // ---程式碼--- const getPath = (dir = '') => path.join(__dirname, dir); module.exports = { getPath } ``` AMD(需透過require.js) ```javascript= require(['fs', 'path'], function(fs, path){ // ---程式碼--- }) ``` CMD ```javascript= define(function (require) { const fs = require('fs'); const path = require('path'); // ---程式碼--- }) ``` ESMoudle ```javascript= import React from 'react'; // ---程式碼--- export default class extends React.Component{} ``` ### STEP 1 下載股市資料庫 台灣證券證交所 - [證券編碼公告](https://www.twse.com.tw/zh/page/products/stock-code2.html) - 整理上市上櫃的資料成`csv檔` ```csv 上市日,市場別,產業別,備註,代號,名稱 2003/6/30,上市,ETF,1,0050,元大台灣50 2006/8/31,上市,ETF,1,0051,元大中型100 ``` 手刻一個讀取csv的函式 ```javascript= // 路徑模組,可分離檔名、擴充名、組合路徑,__dirname代表此js檔的目錄 const path = require('path'); const getPath = (dir = '') => path.join(__dirname, dir); // 檔案讀寫模組 const fs = require('fs'); function readCSV(dir = '') { let str = fs.readFileSync(getPath(dir), 'utf8') let rows = str.split('\r\n') let cols = rows[0].split(',') return rows.slice(1).map(row => Object.fromEntries(row.split(',').map((x, i) => [cols[i], x]))) } ``` 從雅虎財經取得日線OHLCV(開高低收量)資料,這邊使用node套件`yahoo-finance` ```javascript= // 雅虎財經下載工具 const yahooFinance = require('yahoo-finance'); // 時間管理大師,將時間格式化為YYYY-MM-DD const moment = require('moment'); // 取得今日YYYY-MM-DD格式的字串 const TODAY = moment().format('YYYY-MM-DD'); // 定義一個異步函式 async function fetchOHLCV(symbol, from = '2000-01-01', to = TODAY) { return await new Promise((resolve, reject) => { yahooFinance.historical({ symbol, from, to }, (err, rows) => { // ['date', 'open', 'high', 'low', 'close', 'volume'] resolve(rows) }); }) } ``` 初始化資料庫並製作高階API ```javascript= // 資料庫 const sqlite3 = require('sqlite3').verbose(); const db = new sqlite3.Database(getPath('stock.db')); // 定義讀取資料庫成array of object的函式 async function readSQL(sql = '') { return await new Promise((resolve, reject) => { db.all(sql, (err, rows) => { if (err) reject(err) resolve(rows) }) }) } // 定義將array of object的物件寫入資料庫的函式 async function insertRows(table = 'test', rows = [{ name: '1' }]) { // 判斷欄位資料格式 let schema = Object.entries(rows[0]).map(([k, v]) => `${k} ${isNaN(v) ? 'text' : 'number'}`).join(',') // 如INSERT INTO TW2330(open,high,low,close,volume) VALUES (?,?,?,?,?); let placeholder = Object.keys(rows[0]).map(() => '?').join(',') return await new Promise((resolve, reject) => { db.serialize(() => { db.run(`CREATE TABLE IF NOT EXISTS ${table}(${schema});`) let statement = db.prepare(`INSERT INTO ${table} VALUES (${placeholder});`) for (let row of rows) statement.run(Object.values(row)) statement.finalize(); resolve() }) }) } ``` 主程式統整函釋,完成後指令`node downloader.js`執行 ```javascript= async function main() { let tickers = readCSV('代號.csv') for (let ticker of tickers) { let symbol = ticker['代號'] + (ticker['市場別'] == '上市' ? '.TW' : '.TWO') let table = 'TW' + ticker['代號'] let aoo = await fetchOHLCV(symbol) await insertRows(table, aoo) } main() // 將好用的函式輸出成模組 export.modules = { getPath, readCSV, readSQL } ``` ### STEP 2 開發後端API 後端部分採用node.js的Express框架,優點是操作容易,開啟站台程式碼僅需三行 ```javascript= const express = require('express'); app = express(); app.listen(5000, ()=>console.log('伺服器上線!http://localhost:5000')); ``` Express使用方式簡介 ``` - app.set - app.use ├ 靜態檔static ├ 中介軟體middleware ├ request ├ response └ next - app.(get|post|put|delete) ├ request(縮寫req) ├ params ├ json ├ files └ body └ response(縮寫res) ├ status 寫入HTTP狀態碼(100~500) ├ set 設定標頭參數,如Content-Type:[MIME類別] ├ send 回傳字串 ├ sendFile 讀取並回傳檔案 └ redirect 重新導向 ``` ```javascript= // 後端框架express.js,cors讓不同網域的html也能抓到API const express = require('express'); const cors = require('cors'); // 將剛剛的程式碼變成 const { getPath, readCSV, readSQL } = require('./downloader') // 初始化 const tickers = readCSV('代號.csv') const app = express(); const PORT = 3000; // 允許跨源請求 app.use(cors()); // 靜態檔轉址 app.use('css',express.static('/static/css')) // 回傳檔案 app.get('/', (req, res) => { res.sendFile(getPath('HTML相對路徑')) }) // 回傳API - 所有台股代號 app.get('/ticker/ticker', async (req, res) => { res.send(tickers) }) // 回傳API - 特定台股,冒號代表變數即/ohlcv/:ticker=>req.paramsticker=2330 app.get('/ohlcv/:ticker', async (req, res) => { let { ticker } = req.params let aoo = await readSQL(`SELECT date,open,high,low,close,volume FROM TW${ticker} ORDER BY date;`) // 轉為二維陣列 let aoa = aoo.map(x => Object.values(x)) res.send({ ohlcv: aoa }) }) const server = app.listen(PORT, () => { console.log(`伺服器上線!http://localhost:${PORT}`) }) ``` ### STEP 3 開發前端介面 #### 新增HTML檔 新增一個`trading-vue.html`檔,輸入!後按tab,會瞬間展開成完整的HTML,接著利用CDN引入套件 - `vue`前端MVVM框架 - `trading-vue-js`證券交易視覺化工具,簡稱tvjs - `tvjs-overlays`tvjs內建技術分析圖層 body底下再新增帶有id的元素,同時script裡新增一個vue的實例連結#app元素 ```diff= <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <script src="https://cdn.jsdelivr.net/npm/vue@2.x/dist/vue.js"></script> + <script src="https://unpkg.com/trading-vue-js@1.0.1/dist/trading-vue.js"></script> + <script src="https://unpkg.com/tvjs-overlays@0.3.0/dist/tvjs-overlays.js"></script> <title>Document</title> </head> +<style>html,body{margin:0}</style> <body> + <div id="app"></div> </body> +<script> + var vm = new Vue({el:'#app'}) +</script> </html> ``` #### TradingVue組件 綁定屬性,其中有冒號的必須在實例裡定義(同於v-bind:),參數說明如下: ``` - ref → 節點或組件動態索引 - width & height → canvas畫布的寬高 - toolbar → 是否顯示繪圖工具列 - data → Datacube物件 ├ agg(計算模組) ├ data(資料本體) └ chart(主畫面) ├ id ├ type: 'Candles' ├ data: [[ms,o,h,l,c,v]] └ settings └ onchart(主畫面上的圖層) ├ name: 'ALMA 10' ├ type: 'ALMA' ├ data: [[]] └ settings: { color: "#559de0" } └ offchart(畫面下方的指標折線+長條圖) ├ name: 'MACD' ├ type: 'MACD' ├ data: [[]] └ settings: { color: "#85c427ee", backColor: "#85c42711", bandColor: "#aaaaaa", upper: 80, lower: 20 } - overlays → 圖層 ``` ```diff= <body> <div id="app"> + <trading-vue + ref="trading-vue" :width="width" :height="height" + :toolbar="true" :data="dataCube" :overlays="overlays" + /> </div> </body> <script> + const { TradingVue, DataCube } = TradingVueJs var vm = new Vue({ el:'#app', + components: { TradingVue }, + data: { + baseURL: 'http://localhost:3000', + width: window.innerWidth, + height: window.innerHeight, + dataCube: new DataCube(), + overlays: [], + }, }) </script> ``` #### 串接API ```diff= <script> var vm = new Vue({ ..., + async mounted() { + this.tickers = await fetch(`${this.baseURL}/ticker/ticker`).then(res => res.json()) + window.onresize = ()=>{ + this.width = window.innerWidth; + this.height = window.innerHeight; + } + } }) </script> ``` #### `trading-vue-js`客製化圖層 type|data -|- Candles | timestamp, Open, High, Low, Close, Volume, Style (optional) Volume | timestamp, Value, Green or not? Spline | timestamp, Number Splines | timestamp, Number, Number, ... Channel | timestamp, Upper, Middle, Lower Range | timestamp, Number Trades | timestamp, Type 1 Buy 0 Sell, Price, (opt)Label Segment | settings: {p1:timestamp,Price,p2:timestamp, Price} Splitters | timestamp, (opt)Text, (opt)Side, (opt)Color, (opt)Y-pos ## 深入Express #### 相關套件 ```javascript= const iconv = require('iconv-lite'); // big5解碼 const Tesseract = require('tesseract.js'); // 文字辨識 const multer = require('multer'); // 檔案上傳 const globSync = require('glob').sync; // 產生檔案清單 ``` #### 檔案上傳 前端部分 - 使用form submit ```html= <form action="/file/upload" method="post" target="_blank" enctype="multipart/form-data"> <input type="file" name"image" onchange="handleFile(event)"/> <progress id="progress" value="0" max="100"/> <input type="submit" value="提交"/ > </form> ``` > 然而純HTML的作法有個小問題,就是必須點擊提交才會上傳檔案,因此需要js的介入 前端部分 - 使用XMLHttpRequest ```html= <script> function handleFile(event){ var file = event.target.files[0]; var formdata = new FormData(); formdata.append('image',file); var xhr = new XMLHttpRequest(); var progress = document.getElementById('progress'); xhr.upload.addEventListener('progress', function (e){ progress.value = e.loaded; progress.max = e.total; // 單位是bytes }); xhr.open('POST','/file/upload'); xhr.send(formdata); } </script> ``` 前端部分 - 使用axios+es6語法 ```html= <script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"/> <script> const handleFile = async (event)=> { let formdata = new FormData(); formdata.append('image', event.target.files[0]); let progress = document.getElementById('progress'); return axios.post('/file/upload', { data: formData, responseType: 'json', onUploadProgress: (e)=> { Object.assign(progress, {value:e.loaded, max:e.total}) }, }); }; </script> ``` 後端部分 - 使用express+multer ```javascript= const fs = require('fs'); // 檔案重新命名 const multer = require('multer'); // 檔案上傳 const upload = multer({ dest: '上傳檔案儲存的資料夾' }); // 若確定只有上傳一個檔案,使用upload.single('檔案的表單欄位名') app.post('/file/upload', upload.single('image'), (req,res)=>{ // 檔案主要欄位 // originalname 檔案在前端上傳時的原始名稱 // path 檔案經過multer處理後會變成一個沒有副檔名的hash值,並對應設定的dest位置 // size 檔案大小bytes let { originalname, path, size } = req.file; let pngName = path + '.png'; fs.renameSync(path, pngName); // 對pngName這個檔案做壞壞的事,然後回傳給前端 res.set({ 'Content-Type': 'image/png'}).sendFile(pngName); }); ``` #### app.use - 中介軟體(midddleware) 若想要在每次收到request的時候在終端機中列印 ```javascript= app.use(cors()); app.use((req,res,next)=>{ console.log(req.path); next(); }); ``` - 靜態檔管理 `app.use('API路徑', express.static('檔案位置'));` 如檔案放在`/dist/css/bootstrap.css` 而html裡寫`<link rel="stylesheet" href="css/bootstrap.css">` 需新增一行 app.use('css', express.static('dist/css')); #### 結合Websockect 若前端畫面想自動更新,除了定期向後端發送API之外,最好的方法是採用Websocket,由後端推播訊息驅動前端動作,可做到像是「檔案更新後直接推播重新整理」的功能。 ```javascript= const fs = require('fs'); const wss = new SocketServer({ server }); const CODE_INJECTION = ` var protocol = window.location.protocol === 'http:' ? 'ws://' : 'wss://'; var address = protocol + window.location.host; var socket = new WebSocket(address); socket.onmessage = function (msg) { if (msg.data == 'reload') window.location.reload(); }; ` wss.on('connection', (ws) => { fs.watch('dist', { recursive: true }, (e, filename) => ws.send('reload')); fs.watchFile('dist/index.html', (cur, pre) => ws.send('reload')); }); ``` #### 爬蟲 puppeteer + cheerio 取得網頁後直接列印出pdf - [puppeteer](https://pptr.dev/) 以高階的簡易語法操作瀏覽器 - [cheerio](https://github.com/cheeriojs/cheerio/wiki/Chinese-README) `node.js`解析html結構的套件 ``` const puppeteer = require('puppeteer'); async function printPDF(url = '', pdfName = '') { const CHROME_PATH = 'chromium的下載位置' const browser = await puppeteer.launch({ executablePath: CHROME_PATH, headless: true }); const page = await browser.newPage(); await page.goto(url, { waitUntil: 'networkidle0' }); const pdf = await page.pdf({ format: 'A4' }); await browser.close(); /fs.writeFileSync(pdfName, pdf, 'binary'); return pdf; } ``` ### SQL指令 #### 資料表處理 - 建立資料表`CREATE TABLE IF NOT EXISTS Member (...);` - 資料格式`[INTEGER|REAL|TEXT|BLOB]` - 資料限制式`[NOT NULL|PRIMARY KEY|AUTO_INCREMENT|UNIQUE|DEFAULT|CURRENT_TIMESTAMP]` - 外部鍵結 `FOREIGN KEY (...) REFERENCES [TABLE] (...) ON DELETE CASCADE ON UPDATE NO ACTION` - 建立索引`CREATE UNIQUE INDEX id ON Member (...);` - 修改欄位或資料表名稱`ALTER TABLE [TABLE] RENAME [COLUMN?] TO [TABLE|COLUMN];` - 建立觸發器 `CREATE TRIGGER [TRIGGER] [BEFORE|AFTER|INSTEAD OF] [INSERT|UPDATE|DELETE] ON [TABLE] WHEN [condition]` - 移除表格或觸發器 `DROP [TABLE|TRIGGER];` #### 資料列CRUD - 新增create `INSERT INTO Member(...) VALUES (...)`; - 讀取read `SELECT * FROM Member WHERE account = {account};` - 修改update `UPDATE Member SET password = {password} WHERE id = {id};` - 刪除delete `DELETE FROM Member WHERE id = {id};` #### 條件式 - 排序 `ORDER BY ASC DESC LIMIT10;` - 等於不等於 `WHERE a<>'a' AND b='b';` - 介於範圍 `WHERE a BETWEEN 0 AND 100;` - 字串篩選 `WHERE a LIKE '%_![a]'` - 包含集合 `WHERE a IN ('a','b')` - 群組化篩選 `GROUP BY (...) HAVING [condition]`; #### 函數 - 字串類 `ASCII CHAR LENGTH REPLACE TRIM CONCAT UCASE LCASE` - 數字類 `AVG COUNT MAX MIN SUM ABS FLOOR CEOL POWER ROUND SQRT PI EXP LOG` - 統計類 `DISTINCT MID` #### 日期 - 時間 NOW CURDATE CURTIME DATE - 時差 DATE_ADD DATE_SUB DATEDIFF TIMESTAMPDIFF - 格式 DATE_FORMAT #### 欄位合併(水平合併) ```sql= SELECT ... FROM table_a AS a [LEFT|RIGHT|INNER|OUTER] JOIN SELECT ... FROM table_b as b ON a.[欄]=b.[欄] ``` #### 行列合併(垂直合併) ```sql= SELECT ... FROM table_a UNION SELECT ... FROM table_b ``` #### 腳本 ```sql= BEGIN TRANSACTION; ---SQL語法內容--- END; COMMIT; ``` #### 檔案I/O - 執行SQL檔`source C:/[路徑+檔名.sql]` - 匯入CSV檔(MySQL) ```sql= LOAD DATA INFILE '[檔名.csv]' INTO TABLE [表格] FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 ROWS; ``` - 匯入CSV檔(PostgreSQL) ```sql= COPY DB.[表格]([欄位]) FROM '[檔名.csv]' WITH CSV HEADER DELIMITER AS ','; ``` - 匯出CSV檔 ```sql= SELECT * FROM [表格] INTO OUTFILE '[檔名.csv]' FIELDS TERMINATED BY ',' ENCLOSED BY '' LINES TERMINATED BY '\n'; ``` ### 登入認證機制(passport.js) ```javascript= const mongoose = require('mongoose'); const crypto = require('crypto-js'); const passport = require('passport'); const LocalStrategy = require('passport-local').Strategy; const GoogleStrategy = require('passport-google-oauth20').Strategy; const User = mongoose.model('User', { local: { email: String, password: String }, google: { id: String, token: String, email: String, name: String }, }); passport.use( new LocalStrategy((username, password, next) => { User.findOne({ username }, (err, user) => { if (err) return done(err); if (!user) return done(null, false); if (!user.verifyPassword(password)) return done(null, false); return done(null, user); }); }) ); passport.use( new GoogleStrategy( { clientID: GOOGLE_CLIENT_ID, clientSecret: GOOGLE_CLIENT_SECRET, callbackURL: 'http://www.example.com/auth/google/callback', }, (accessToken, refreshToken, profile, next) => { User.findOrCreate({ googleId: profile.id }, (err, user) => { return next(err, user); }); } ) ); let auth = passport.authenticate; let successRedirect = 'profile'; let failureRedirect = 'login'; const checkAuth = (req, res, next) => { if (req.isAuthenticated()) next(); else res.redirect('/'); }; app.get('/login', (req, res) => res.render('login.html')); app.post('/login', auth('local-login', { successRedirect, failureRedirect })); app.get('/auth/google', auth('google')); app.get('/auth/google/callback', auth('local-login', { successRedirect, failureRedirect })); app.get('/logout', (req, res) => { req.logout(); res.redirect('/'); }); app.get('/signup', (req, res) => res.render('signup.html')); app.get('/profile', checkAuth, (req, res) => res.render('profile.html')); ``` ## 前端函式庫 - fontawesome - ICON圖示 - jquery - 簡化HTML與JavaScript之間的操作 - lodash / underscore - JS物件操作擴充庫 - moment - 時間格式化套件 - Cleave - 輸入框格式化 - bootstrap / materialize - UI框架 - axios - 前後端通用的請求發送套件 - xml2json - xml轉json - docx - 編輯word檔 - sheet.js/xlsx - 編輯excel檔 - math.js - 矩陣運算 - crypto-js - 加密演算法,如RSA, sha1, sha256, md5 - dom-to-image - 輸出螢幕快照圖片 - pdf.js - 預覽PDF - jspdf / pdflib / pdfkit - 編輯PDF(無法輸出中文字) - split.js - 可伸縮畫面切割 - popper.js - 響應式彈跳方塊 - sweetalert - 一行搞定的彈跳視窗 - three.js - 3D畫面渲染 - matter.js - 物理引擎 - frabic.js - canvas畫布操作 - tone.js - 音訊操作 - Anime.js - DOM動畫框架 - aos.js - Animate On Scroll,捲軸特效 - chart.js - 簡易又生動的圖表 - leaflet.js / openlayers.js - 網頁地圖 - algolia - 地址搜尋 - fastest-levenshtein - 文字模糊比對 #### 前端框架 - angular - react - 基本引入套件`react`+`react-dom` - 路由管理`react-router`或`react-router-dom`(推薦後者) - 狀態管理`redux`或`mobx`+`mobx-react` - APP開發`react-native` - SSR伺服器端渲染`next.js` - UI框架`material-ui`或`ant-design` - vue - 路由管理`vue-router` - 狀態管理`vuex` - SSR伺服器端渲染`nuxt.js` - UI框架`vuetify`或`element-ui` - svelte - 相較於MVVM結構控制runtime,svelte則採用compiler - 內建狀態管理 #### 打包工具 > 何謂打包?請參照:[Webpack打包範例 - 我全包了](https://hackmd.io/tkWGj4THScCtgumJRMkhXA) - gulp 採用pipeline的語法設計模式 - rollup - snowpack 講究零設定 - parcel 零設定打包 - webpack 具有最完整設定,但學習門檻高 - vite 基於esmodule的打包工具