Crawler - HackMD

# Crawler ## Prerequisites - Rocky Linux 8.6 - Nginx 1.20 - PHP 8.1 - Mysql 8.0 - Laravel 9.19 - Chrome 106.0.5249.119-1 - Chrome Driver 106.0.5249.61 ## Basic Installation ### Install Extra Packages for Enterprise Linux (EPEL) ``` dnf install epel-release -y ``` ### Install remi Library ``` dnf install dnf-utils http://rpms.remirepo.net/enterprise/remi-release-8.rpm -y ``` ### Install PHP 1. Clean PHP Install Version ``` dnf module reset php -y ``` 2. Setting PHP Version ``` dnf module enable php:remi-8.1 -y ``` 3. Install ``` dnf install php -y ``` 4. Setting php.ini ``` max_execution_time 600 max_input_time 600 ``` *Note* You can command `php --ini` find the php.ini 5. Setting php-fpm ``` user = nginx group = nginx ``` >path /etc/php-fpm.d/www.conf ### Install MySQL 1. Install php-mysql ``` dnf install php-mysql -y ``` 2. Clean Mysql Install Version ``` dnf module reset mysql -y ``` 3. Setting Install Mysql Version ``` dnf module enable mysql:8.0 -y ``` 4. Install ``` dnf install @mysql -y ``` ### Install Nginx 1. Clean Nginx Install Version ``` dnf module reset nginx -y ``` 2. Setting Nginx Version ``` dnf module enable nginx:1.20 -y ``` 3. Install Nginx ``` dnf install nginx -y ``` #### Setting Nginx Sever 1. Change your default root ``` root /var/www/html/scrapebaidu/public; ``` > path /etc/nginx/nginx.conf 2. Change keepalive_timeout ``` keepalive_timeout 600 ``` 3. Append parameter ``` fastcgi_read_timeout 600; proxy_read_timeout 600; proxy_connect_timeout 600; proxy_send_timeout 600; ``` #### Setting Nginx PHP Add code at the bottom ``` location / { try_files $uri $uri/ /index.php?$query_string; } ``` > path /etc/nginx/default.d/php.conf ### Install Git ``` dnf install git -y ``` ### Install Composer ``` dnf install composer -y ``` ### Install wget ``` dnf install wget -y ``` ### Install Unzip ``` dnf install unzip -y ``` ### Install Chrome ``` dnf install https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm -y ``` > Check Version `google-chrome --version` ### Install Chrome Driver 1. Enter the specified directory ``` cd /usr/bin ``` 2. Download chromedriver_linux64.zip ``` wget https://chromedriver.storage.googleapis.com/106.0.5249.61/chromedriver_linux64.zip ``` 3. Unzip chromedriver_linux64.zip ``` unzip chromedriver_linux64.zip ``` > Check Version `chromedriver --version` 4. Optionally, delete chromedriver_linux64.zip file ``` rm -rf chromedriver_linux64.zip ``` ## Disable SELinux 1. Set enforce ``` setenforce 0 ``` 2. Change SELINUX ``` SELINUX=disabled ``` > path /etc/selinux/config ## Enabled Nginx ``` systemctl enable nginx --now systemctl start nginx ``` Mysql ``` systemctl enable mysqld systemctl start mysqld ``` PHP ``` systemctl enable php-fpm systemctl start php-fpm ``` ## Setting Project 1. Clone the repository ``` git clone git@125.227.40.229:php/scrapebaidu.git ``` *Note* or use unzip code 2. Setting .env ``` cp .env.example .env ``` *Note* You need to configure database connection settings here 3. Composer ``` composer install ``` 4. Migration ``` php artisan migrate ``` *Note* You must add a new database first, otherwise you will get an error 5. Crontab ``` * * * * * php /path/to/artisan schedule:run >> /dev/null 2>&1 ``` 6. Chown ``` chown -R nginx storage chown -R nginx bootstrap/cache ``` ## Setting Proxy Whitelist Please contact tw developer --- Get Start Crawler