# CPENT iLab筆記 - Appendix E: Bash Environment and Scripting

## Exercise 1: Basic BASH Queries
Objectives
- Create basic BASH queries, extract data, and explore the features and components of BASH
Lab Duration: 20 Minutes

1. By default Bash-Machine machine selected, click Student profile to login.


2. Type password in the Password field and click Sign In.


3. In this exercise, we will review the different methods of extracting data using BASH. First, we want to create several files with passwords in them. You can create your own or open one of the files on the machine, copy parts of it, and create multiple files. We need at least three files to work with.

4. In this exercise we have already created wordlists.txt file in the Home directory and made a copy of those files in the Home directory, as shown in the screenshot.


5. First, try the command cut -d' ' -f2 file.txt. Launch a terminal, type cut -d' ' -f2 file.txt and press Enter. An example of the output of the command is shown in the screenshot.


6. Next, you can view the info on the CPU. Type cat /proc/cpuinfo and press Enter. An example of the output of the command is shown in the screenshot. Take a few minutes and review the results.


7. Next, we will review the sort command. Type sudo ls / -R -s | sort -n -r | head -5 and press Enter. Type password and press Enter.

- R option for ls, which will cause it to recursively list the files under the specified directory.
- The power of piping is demonstrated. An example of the output of this command is shown in the following screenshot.

8. There are many different ways to work with the output. If you have time, feel free to explore more.



9. Next, type sudo find /home -mmin -5 and press Enter. The output of this command is shown in the following screenshot.


10. To view the files modified in the last 24 hours, type sudo find /home -mtime -1 and press Enter.


11. Another command we can use is the tool xxd. Type sudo xxd -s 35 -l 50 wordlists.txt and press Enter. An example of the output of the command is shown in the following screenshot.


12. Next, type printf ‘A’ | xxd and press Enter to convert A into hex.


13. To convert from hex to ASCII, type printf 0x41 | xxd -r and press Enter.


14. To output in binary, type printf 'A' | xxd -b and press Enter. The output of the command is shown in the following screenshot.


15. We will now explore the string search capability. Type sudo egrep -a -o '\b[[:print:]]{2,}\b' somefile.exe – string search and press Enter.

16. By replacing the somefile.exe with a binary file, you can use the file command to find one if needed or use /sbin/ifconfig as shown in the following screenshot. An example of the output of the command is shown in the following screenshot.



17. Another useful feature is removing duplicates. Type sudo egrep -a -o '\b[[:print:]]{2,}\b' /sbin/ifconfig | sort –u and press Enter. We are piping the output into sort, which will remove any duplicates. This is another common feature of the files found during testing or competitions.



18. The lab objectives have been achieved.

-------------------------

## Exercise 2: Basic cURL Queries
Objectives
- Create basic cURL queries, extract the data, and explore the features and components of cURL
Lab Duration: 20 Minutes

1. cURL is a command line tool and a library that can be used to receive and send data between a client and a server or any two machines connected over the internet. It supports a wide range of protocols such as HTTP, FTP, IMAP, LDAP, POP3, and SMTP.

2. Due to its versatile nature, cURL is used in many applications and for many use cases. For example, the command line tool can be used to download files, testing Application Program Interfaces (APIs), and debugging network problems. In this lab, we shall look at how the cURL command line tool can be used to perform various tasks.

3. As a penetration tester, you need to be familiar with this tool. We will cover some of the usage examples in this lab, but this tool can be used in many ways.

4. In this exercise, we will review the different methods of extracting data using cURL.

5. By default Bash-Machine machine selected, click Student profile to login.

Note: If you are already logged in skip to step 7.


6. Type password in the Password field and click Sign In.


7. For our first example, we will connect to the website. Type curl 192.168.177.200 and press Enter. Type password and press Enter.



8. In this form, the cURL tool is acting as a simple client. We have the capability to do this using Telnet and netcat, but the tool makes it in one go.

9. Let us now see what it looks like at the packet level. We need to know this in case we use it in testing. An example of the Wireshark session is shown in the following screenshot. The default User-Agent shows that the request comes from cURL. This could alert monitoring on different networks, so you might want to use the tools capability to customize the UA if this is included in the scope of work (evasion) or you are acting as a red team member.


10. Similar to most tools, the cURL utility has the ability to output to a file. cURL uses the -o option. The tool can download, and when it does, a progress bar shows the download statistics. An example of this is shown in the following screenshot.

11. If you have a partially downloaded file, you can resume the file download with the -C – option.


12. We can test this now. Log in to the Bash-Web machine with the username as root and password as owaspbwa.


13. Once you are in the machine, you can use the dd command to create the file. Type dd if=/dev/zero of=testfile_10MB bs=10485760 count=1 and press Enter.


14. Next, we need to copy the file to the root folder for the web server. First, we need to identify its location. In most cases, the root folder is in a default location. So let us see if this is the case. Type ls -lart /var/www/index.html and press Enter. An example of the output of the command is shown in the following screenshot.


15. Now that the file has been verified along with the path, we can copy our created file there. Type cp testfile_10MB /var/www/ and press Enter.


16. Swithc to Bash-Machine machine and type sudo curl 192.168.177.200/testfile_10MB -o testfile.bin and press Enter.

Note: 192.168.177.200 is the IP of the Bash-Web machine.


17. Before we delve further into the features supported by cURL, we will discuss HTTP requests and responses in more detail. If you are familiar with these concepts, you can skip to Step 20. As per the RFC, to request a resource such as a web page or to submit data to a server, an HTTP client (such as a browser or cURL) makes an HTTP request to the server. The server responds with an HTTP response, which contains the “contents” of that page. An example of this is shown in the following screenshot.


18. HTTP requests contain the request method, URL, some headers, and some optional data as part of the “request body.” The request method controls how a certain request should be processed. The most common types of request methods are “GET” and “POST.” Typically, we use “GET” requests to retrieve a resource from the server and “POST” to submit data to the server for processing. “POST” requests generally contain some data in the request body, which the server can use.

19. HTTP responses are similar and contain the status code, some headers, and a body. The body contains the actual data that clients can display or save to a file. The status code is a three-digit code, which tells the client if the request succeeded or failed, and how to proceed further. Common status codes are 2xx (success), 3xx (redirect to another page), and 4xx/5xx (for errors).

20. To review the request headers and connection details, you can use the verbose option. Type curl -v http://192.168.177.200 and press Enter. An example of the output from the command is shown in the following screenshot.


21. Take a few minutes to review the output. If you so prefer, you can output the command to a file and review it there as well later. The output contains request data (marked with >), response headers (marked with <), and other details about the request such as the IP used and the SSL handshake process (marked with *). When the HTTPS protocol is used.

22. Most often, we are not interested in the response body. You can simply hide it by “saving” the output to the null device, which is /dev/null (the bit bucket).

23. If you want to perform the recon without seeing the output of errors, you can use the -s option. Type curl -svo /dev/null http://192.168.177.200/testfile_10MB and press Enter.


24. The -s option is slightly aggressive, since it hides even error messages. For your use case, if you want to hide the progress bar but still view any errors, you can combine the -S option.

25. Therefore, if you are trying to save the cURL output to a file but simply want to hide the progress bar, you can type curl -sSvo /dev/null http://192.168.177.200/testfile_10MB and press Enter.



26. Now, we address the fact that the command shows the name curl in the User-Agent. When testing APIs, you may need to set custom headers on the HTTP request. You can use the -H option of cURL for this purpose. Ensure that you are capturing on Wireshark before entering the next command. If you want to send the custom header X-My-Custom-Header, type curl -H ‘X-My-Header-Test: CLIENTZZZ’ http://192.168.177.200 and press Enter. The output of the command is shown in the following screenshot.



27. Wait a minute! We have our custom header, but we still have the announcement that we are using curl. This needs to be addressed. Restart the Wireshark capture and then type curl -H ‘User-Agent: TEST’ ‘X-My-Header-Test: CLIENTZZZ’ http://192.168.177.200 and press Enter.



28. As you can see, we have now successfully obfuscated our client connection. You can do a lot more and are encouraged to explore. By default, cURL sends GET requests, but it can also be used to send POST requests with the -d or --data option. All fields must be given as key=value pairs separated by the ampersand (&) character. Type curl --data "firstname=boolean&lastname=world" https://httpbin.org/post and press Enter.

29. The output clearly indicates that we have posted two parameters that are identified with the form data.


30. As a refresher, characters that are considered special characters must be encoded. This can be done manually, but it is easier to use the cURL tool. Now, type curl --data-urlencode "email=[e-mail]" --data-urlencode "name=Penetration Testing" https://httpbin.org/post and press Enter.


31. If you want to upload files using a POST request, you can use the -F (“form”) option. Here, we will submit the file.txt under the parameter name file.

- Type echo "This is a test file for pentesting practice" > file.txt and press Enter.
- Type curl -F file=@file.txt https://httpbin.org/post and press Enter.


32. Previously, we have seen how to send POST requests with cURL. Sometimes, you may need to send a POST request with no data at all. In that case, simply change the request method to POST with the -X option.

33. The request method can also be changed to anything else such as PUT, DELETE or PATCH. One notable exception is the HEAD method, which cannot be set with the -X option. The HEAD method is used to check if a document is present on the server, but without downloading the document. To use the HEAD method, we use the -I option. Type sudo curl -I 192.168.177.200 and press Enter. Type password and press Enter.



34. As the above screenshot shows, we have only returned the header. Using the HEAD method has been demonstrated.

35. We can also access the Firefox developer tools. Launch Firefox browser and open any web site. Press F12 to open the developer tools, and then click on the Network Tab. Select a GET request. Right-click on this to bring up another menu.

36. Click Copy as cURL. We have the option to copy for both Windows and POSIX, select the appropriate one, and then paste it.


37. Switch to cURL terminal and paste the copied cURL in the terminal and press Enter. You can try this by connecting to any web server machine.



38. Sometimes, you may wish to ensure that all cURL requests use the same options. Passing these options manually is not a feasible solution. Therefore, cURL allows you to specify options in a configuration file. The default configuration file is located in ~/.curlrc in Linux/MacOS. An example of a configuration file is shown in the following screenshot.


39. If you want to use a custom configuration file instead of the default one, then you can use -K option to point curl to your configuration file. To learn more about the tool, enter man curl. Take a few minutes to read about the tool.

40. We have shown multiple ways to use the tool, and there are many more. The lab objectives have been achieved.

-----------------------

## Exercise 3: Log Analysis with BASH
Objectives
- Create BASH queries, extract data from log files, and modify log data
Lab Duration: 20 Minutes

1. In this exercise, we will review the different methods of extracting data from log files using different BASH methods.

2. By default Bash-Machine machine selected, click Bash-Web to switch.

Note: If you are already logged in skip to step 4.

3. Login to the machine with the username as root and password as owaspbwa.


4. In the terminal, type tail -f /var/log/syslog and press Enter.


5. Switch to Bash-Machine and in the terminal type nikto -h 192.168.177.200 and press Enter.

Note: If the machine is locked, move your cursor in the upward direction and type password in the Password field and press Enter.


6. Switch to Bash-Web and press Ctrl+C. The main log for Apache is located in the folder /var/log/apache2. Change into the directory with cd /var/log/apache2.


7. Next, type tail -f access.log and press Enter and leave the machine intact.


8. Now, switch to the Bash-Machine and run the Nikto scan again. Then switch to the Bash-Web and review the output of the scan. This is shown in the following screenshot.


9. The scan results are being tracked in our access.log file. However, since we only scanned the default page, we do not see a lot of information. Again, the above screenshot reveals that we are being logged when we access the web server and not only that our IP address is being logged as well. Therefore, we need to identify a method to modify the logs and change what is located there. We could delete the log, but this will be obvious and not ideal in a professional test. If the log is removed, it could cause a problem for our clients.

10. In cases of logs with a smaller size or if we are looking for a specific keyword, then we can spend some time observing the logs manually using things such as grep expressions.

11. As we have done earlier, we can use the cat command to pipe the output into grep and look for specific keywords. Press Ctrl+C in the Bash-Web machine to stop the traffic. Type cat access.log | grep index.php | more and press Enter. The output from this command is shown in the following screenshot. Press Enter to review until you get the terminal.



12. As the above screenshot shows, one of the problems is that we have a header with the User-Agent of nikto as well. In addition to the IP address and the obvious attack strings, this is another concerning aspect.

13. Let us now take a look at all logs in the folder. Type ls /var/log and press Enter.


14. As the above image shows, there are several log files, and investigating them is beyond the scope of this lab. However, the process of log analysis is the same, and which log you review is irrelevant.

15. The dmesg command prints the kernel ring buffer. By default, the command will display all messages from the kernel ring buffer. In the terminal, type dmesg and press Enter, and the entire kernel ring buffer will print. An example of the output is shown in the following screenshot.



16. The output file can be long, so you can pipe it to less and review it there. We can also pipe the log into the grep tool. Type dmesg | grep network and press Enter. If anything has been logged, it will print. An example of this is shown in the following screenshot.


17. Next, we will review the auth log. In the terminal type cd /var/log and press Enter to change the directory to log. Type tail -f auth.log and press Enter. This will show the login sessions.


18. Switch to Bash-Machine machine, and type ssh root@192.168.177.200 and press Enter. Accept the storage of the key message, and enter yes.


19. Enter the wrong password.


20. Now, switch to Bash-Web machine, and type awk '/sshd.*Failed/ { print $9 }' /var/log/auth.log and press Enter. An example of the output of the command is shown in the following screenshot.

21. As the screenshot shows, the login has failed.


22. We are now ready to modify the log and change our access so that it does not show us making access. Type the following command and change the address if required to your address:

cat '/var/log/apache2/access.log' |grep -v “192.168.177.82” > cleaned.log

Note: 192.168.177.82 is the IP address of the Bash-Machine.


23. Now, you want to grep for your IP address to see if it is still there. Type cat cleaned.log | grep "192.168.177.168" and press Enter. The output of the command is shown in the following screenshot.


24. As the above screenshot shows, we are successful, and there is no evidence of our IP address in the log.

25. The last task of this lab is using BASH to connect to a socket. This is accomplished by entering the code shown in the following screenshot.


26. Depending on the version of the OS, the Transmission Control Protocol (TCP) location may or may not be the same.

27. The lab objectives have been achieved.