# CPENT iLab筆記 - Appendix F: Python Environment and Scripting

## Exercise 1: Basic Python
Objectives
- Create basic Python scripts and explore the features and components of Python
Lab Duration: 20 Minutes

1. In this lab, you will review the Python framework and create simple Python scripts.

2. By default Python-Machine machine is selected, type password in the Password field and press Enter.


3. Launch a terminal and type python --version and press Enter. Then, type python3 --version and press Enter.

4. You have two different versions of Python, which is common, because some codes work with one version and others work with another. It is best to keep them both. Until you get to penetration testing specific scripts, the difference is not meaningful.


5. Create a simple conditional script. Click Leafpad icon to launch the Text Editor. Type the following code as shown in the screenshot, and save it as myfirst.py on Desktop.



6. In the terminal type cd Desktop and Enter to change directory location. Type python myfirst.py and press Enter. This is a simple piece of code that shows the “if else” statement, which works similarly in all languages.


7. Next, observe the “while” loop; Open a new Leafpad editor and type the code as shown in the screenshot and save it as while.py on Desktop.



8. Type python while.py and press Enter.


9. Continue and move to the actual scripts for testing. Start small and work your way up. If you are new to “if” statements and loops, then you will need to explore more and practice. First, master the network communication, which is based on sockets. This way, you start with a simple socket.

10. Since you are on the attacking machine, you are primarily concerned with creating the network connection capability within the own machine. The sockets consist of an Internet provider (IP) address bound to a port, and this is how you extract the data as well as send data into the machines. You have a Transmission Control Protocol (TCP) as well as a User Datagram Protocol (UDP) type of socket. Design these based on how you open the socket when you create it. To create a socket, enter the code below:

- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- To build a socket, specify two options: the socket family and the socket type. The socket family, in this case, is AF_INET, which is an IPv4 socket. Other families are AF_INET6 for IPv6, AF_UNIX for local sockets, and AF_RAW for raw sockets.
- The second option is the socket type, which, in this case, is a SOCK_STREAM socket. SOCK_STREAM sockets are TCP-style sockets, but you also have the option of using SOCK_DGRAM for User Datagram Protocol-style sockets or SOCK_RAW for raw sockets.
11. Once you have created the socket, use the functions within the socket family to make a connection.

12. Create a simple banner grab using Python. In the terminal type python3 and press Enter.


13. In the Python terminal type the following code as shown in the screenshot:

import socket
bangrab = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
bangrab.connect(('127.0.0.1', 60000))
bangrab.recv(4096)
bangrab.close


14. The Secure Shell (SSH) server is bound to port 60000 on the example machine. We can add the capability to interact with a web server. However, to do this, you have to expand on the code used. An example of how you could do this is shown in the next sequence of statements.

s.send("GET / HTTP/1.0\nHost: 192.168.177.200\n\n")
page = ""
while 1:
data = s.recv(1024)
if data == "":
break
page = page + data

Note: 192.168.177.200 is the IP address of Web-Python machine.

15. The socket send method takes a single argument: the string that you wish to send. In this case, you are sending the request to the OWASP BWA machine. First, initialize the page with an empty string to ensure you do not pick any garbage that could be left over in memory. Once you have set the empty string, create a code to use a loop to continuously retrieve the data up to the amount of data that you receive, which, in this example, is 1024 bytes. Since you are setting the value in the recv function, the “read” will continue until there is an empty string read, indicating there are no contents left. Add the code to close the socket and print the data using the commands below:

s.close()
print page

16. Open a new Leafpad editor and type the code as shown in the screenshot and save it as web-conn.py on Desktop.



17. In the terminal type cd Desktop and Enter to change directory location. Type python web-conn.py and press Enter.


18. However, there is one downside to the example above: the code is for Python 2 not 3, and it is also only for IPv4. The first part is inevitable when working with different versions. You need to syntactically address such issues. Such predicaments also necessitate having more than one version. The second part, with respect to only working with IPv4, is common to encounter. If you do want to modify the code, then enter

af,type,proto,name,conn = socket.getaddrinfo("www.eccouncil.org", 80,0,0, socket.SOL_TCP)[0]

19. Regarding 6, consider that you are using getaddrinfo, which determines and translates the returned data. In this case, you have passed it the hostname, the web server port of 80, protocol of TCP, and 0 for the family and socket type. This will allow it to figure those out for us. This function returns an array of possible IP addresses that can be used as well as the socket and family types of those IP addresses.

20. Next, work with the python-nmap module itself. Use Python 3 for it to work. Launch a new terminal and enter the following commands:

python3
import nmap
nm = nmap.PortScanner()
nm.scan('127.0.0.1', '53,443,3306,60000')


21. Although this step works, it is not the best format. It is important you make improvements. Check if the output is because you requested for more than one port; enter

nm.scan('127.0.0.1', '22')


22. The output is still not well formed. You can eliminate the fact that it was caused by the number of ports. Next, explore better methods.

23. Before you do so, change the scan and acquire more machines. Enter the following code:

nm.scan('192.168.177.0/24', '22-60000')


24. You are now passing arguments into the nm tool, which is followed by a scan.

25. Once the scan is complete, enter nm.all_hosts(). The results are shown in the screenshot.


26. This module contains multiple options for assisting us to make better sense out of the raw output. First, enter nm.command_line().


27. Since you have created the output in XML, the standard stylesheet allows you to view the output in a more readable form. Since you now know how the module is performing, use the function and pass it the arguments. Enter the following code:

nm.scan(arguments='-n -A -p 3306 --script=/usr/share/nmap/scripts/mysql-info.nse')


-----------------

## Exercise 2: Basic Networking and Python
Objectives
Create basic Python scripts and explore the features and components of Python
Lab Duration: 20 Minutes

1. In this lab, create simple Python network scripts: the first one is a Transmission Control Protocol (TCP) client.

2. By default Python-Machine machine is selected, type password in the Password field and press Enter.

Note: If you are already logged in skip to step 3.


3. Enter the code below in the editor of your choice and save it as TCPclient.py on Desktop.

Note: 192.168.177.157 is the IP address of WinPy-Python machine.



4. Launch a Terminal and type cd Desktop and press Enter. To run the TCPclient.py file type python TCPclient.py and press Enter. Notice the creation of the socket. Further, you now have a TCP socket. Connect to it, send a GET request, save the response, and then print it, as shown the screenshot.


5. Note that this code is not set up to error check, but it will serve the purpose. When you create scripts during testing, it is rare to create the added overhead of boundary and error check. Since you are only using it for personal testing, this should not be a concern.

6. For the User Datagram Protocol (UDP), change the socket type to SOCK_DGRAM and use sendto(). Since the UDP is connectionless, there is no connect function.

7. You are now ready to create the server. The server communication is more complex than client communications; this is the requirement to accept incoming connection. As with all socket communications, you need to create the socket, and then bind it. Once you have completed this step, sit in a loop and cycle, and wait for connections.

8. Enter the code shown below. Note that this code is for the Python 2.7 and not Python 3.


9. First, pass in the IP address and port you want the server to listen on. Next, tell the server to start listening with a maximum of five connections. This is only to make the code simpler. You could make it any number of your choice. This is not a concern when you are testing.

10. Place the server into its main loop, where it is waiting for an incoming connection. Once the client connects you, handle the connection and send a message back to the client. This loop method is the same as the majority of object-oriented code. This is known as “event driven” programming, where the loop runs continuously, waiting for events to act on.


11. You can modify the client code in order to connect to the loopback, and then send the data into the server using the changes shown in the screenshot.


12. Run the client. The server will receive the string you sent.



13. Once again, this is only a basic server that you use to focus on concepts. This is acceptable from the point-of-view of the penetration testing requirements. Then, consider the following:

- There may be pauses in input from the client.
- You may want to handle multiple connections at the same time.
- To do this, add more robust coding statements—as a possible example, refer to the following enhancements:
- If you want to listen to X amount of connections—in this case 10—add the following:
- s.listen(10)
- In this statement, add an argument to the function of 10, which will represent up to 10 connections. Then, set up a simple variable to provide a list of the returned data. To do this, enter
- Input – [s].
- The next component you need is the capability to perform extractions. Within the library, you have the select module that provides this capability. Use the following statement:
- reader,output,exceptions = select.select(input,[],[])
- This module takes three arguments:
- A list of sockets to check for reading
- Socket to check for writing
- Sockets to check for errors
- In effect, this method returns three lists: read, write, and errors.
- Once you have the data, review it by cycling through the array:
- for sock in reader:
- if sock = s:
- c,addr = s.accept
- print “New connection from “ , addr
- input.append(c)
- Here, explore each socket in the array. If you find the needed socket, then you may seek a connection. Accept the connection and print a message. Then, append the connection into the input list. Evaluate whether it is a command to execute.
- Continuing the code block; add an “else” statement that reflects it is a command and not data:
- else:
- command = sock.recv[1024]
- if command:
- shell = command.rstrip().split(" ")
- If the socket was not the same as the listener socket, you have a client who has sent us information. Read that information into a string called command.
- If no data are waiting, the socket has shut down and you need to close it. If they are, clean it up to get ready to execute it. First, strip any whitespace from the end. Then, split it into individual arguments to be passed into the shell.

14. The handling of a server can be much more complex. Since penetrations testers only work with the client side, this is not a concern.

15. Next, explore the Raw Sockets. You will also explore methods to use Python and its libraries to extract packet data.

16. When you want to access the lower-level network information, use raw sockets. This will give you the capability to extract the network IP header details.

17. Windows and its access to raw sockets behave differently compared with Linux. Therefore, when you are writing the script code, determine the OS, so you can access the raw socket properly.

18. Create a small chunk of code that will consider the OS. This structure still references NT, since it is still part of the latest OS systems. They still have their roots in the Windows NT kernel. In fact, Microsoft has maintained this same kernel structure up until Windows Server 2016, which allows testers to reference other rings of the Intel architecture, and not just rings 0 and 3. The NT model has focused on these two of four available rings within the architecture. The breakdown is as follows:

- Ring 3
- User mode
- Ring 2
- Not used
- Ring 1
- Not uses
- Ring 0
- kernel
19. The problem with this approach is and always has been that all user mode processes will need to spend time in kernel mode.

20. An example of this simple sniffer that reads one packet is shown in the screenshot.


21. We start by constructing the socket object with the parameters necessary for sniffing packets on the network interface. The difference between Windows and Linux is that the former will allow us to sniff all incoming packets regardless of protocol. This is not the same method and approach for Linux, where the tester needs to specify the protocol you want to use—for example Internet Control Message Protocol (ICMP).

22. We use the extra step for input/output control for Windows in order to enable the promiscuous mode. An example of an attempt to run the code without the correct privileges is shown in the screenshot.


23. Perform the sniffing action. In this example, you are only sniffing one packet. Then, turn off the promiscuous mode.


24. While the output is in a raw format, you can see the machine that has sent the packet “192.168.177.157” and that it is generated from a Linux machine because of the contents of the packet are in numbers and punctuation. By contrast, the header of a Windows system has a–w lowercase characters as a default composition. This is trivial for us to modify using the -p option in ping as well as many other ways.

25. In its current form, the sniffer receives all IP headers along with any higher protocols such as TCP, UDP, or ICMP. The information is packed in binary form; as shown above, it is difficult to understand. You will now going to work on decoding the IP portion of a packet, so you can extract useful information such as the protocol type (TCP, UDP, ICMP) as well as the source and destination IP addresses.

26. An example of the IP header from the Network Sorcery site is shown in the screenshot.


27. Decode the entire IP header (except the Options field) and extract the protocol type, source, and destination IP address. Using the Python ctypes module to create a C-like structure will allow us to have a friendly format for handling the IP header and its member fields, as shown in the screenshot.


28. Create a complete sniffer program that decodes the header as shown in the screenshot, so you do not have to read the binary. This binary is represented in hex, which is not entirely challenging.



29. To explain further, first, store the content from the structure. The IP class simply takes in a raw buffer (in this case, what you receive on the network) and forms the structure from it. When the init method is called, new has already finished processing the buffer. Inside init.

30. Add the code to continually read and parse the packets that are read. Pass the IP header, which consists of the first 20 bytes into the structure. Then, print it out. An example of the program in action is shown in the screenshot.


31. As the number of activities increases, the process of creating your own scripts becomes more cumbersome.

32. Before scripting, explore the tool; enter sudo scapy.


33. This is an interactive interpreter that is similar to the Python interface.

34. Begin with a basic ping packet. First, explore how to determine the structure that you need. At the prompt, enter ICMP().default_fields. As you have probably determined, you are using the tools capability to extract the default fields. An example of the output from this is shown in the screenshot.


35. This is the power of the tool. You only ask it to provide the structure to write the script. Remember that you do not have to populate all fields—only the ones you need.

36. Enter the commands below into the terminal window. Remember to replace the IP address with the one for your network. Alternatively, if you have a “live” Internet connection, you can replace the dst with the site of your choice. The commands are the same.

Note: 192.168.177.157 is the IP address of WinPy-Python machine.

p = IP(dst="192.168.177.157")/ICMP()
r = sr1(p)
r


37. You sent an ICMP packet to the destination represented by dst, and then displayed the reply string, If you enter r[IP].src, then the source address will be displayed.


38. This is an extremely powerful tool and one that you should include in your arsenal of tools.

39. Next, specify the type of ICMP. Enter the following code:

Note: 192.168.177.200 is the IP address of Web-Python machine.

- p = IP(dst="192.168.177.200")/ICMP(type="echo-request")

- This allows you to manipulate the packets.
- Use this in the testing to check if the site monitoring system detects whether you have sent crafted packets.
- Then, enter here p the variable that you stored the data in.

40. Create the packet using slash notation: Each layer is separated by the slash (/) symbol. To verify that the packet is assembled correctly, print the packet and check that both the IP and the ICMP layers are correct.

41. Access each layer of the packet, similar to the way you access a dictionary: by specifying the layer as the array index to the answer packet. Access the values of each field in the layer using the dot (.) notation to indicate they are elements of a class. At times, you will want to see the whole structure in a more readable format. The show method will break down each layer and show all values that have been set.

42. Now that you are familiarized with the basics of creating packets and interacting with them through the Scapy shell, begin building a script.

43. Scapy eases network packets manipulation and allows you to forge complicated packets to perform advanced tests. Consider two examples that are difficult to express without Scapy.

44. Sending a TCP segment with maximum segment size set to 0 to a specific port is an interesting test to perform against embedded TCP stacks. Without Scapy, this takes multiple components; with Scapy, you can achieve this in one line.

45. Enter the following command:

send(IP(dst="192.168.177.157")/TCP(dport=502, options=[("MSS", 0)]))

Note: You have crafted a TCP packet of 0 bytes in size and sent it into a target port. For this example, the port is ModBus. You can use this to check how an embedded system reacts, since many employ either a TCP stack that is not complete or not standard, which can complicate the testing.


46. From the above command, you can send custom packets into the network, although this does not go into the full scope of the method.

47. The other example is for firewalking. If you do not have Internet access, this will not be visible to you.


48. Let us now explore some more examples of this tool. With Scapy, each network layer is a Python class. The “/” operator is used to bind layers together. Put a TCP segment on top of the IP, and assign it to the packet variable. Then, stack it on top of the Ethernet. Type the following command packet = IP()/TCP(), and then Ether()/packet.


49. You now have the structure of the packet in the summary. The items in red you supplied and Scapy complete the rest.

50. To know which fields, enter ls(IP, verbose=True).


51. Enter p = Ether()/IP(dst="192.168.177.157")/TCP().


52. There are useful methods within Scapy such as the summary function. Enter p.summary(). An example of the output is shown in the screenshot.

53. You will find that Scapy has interesting features. One of the methods of trying to get through a filter is source port scanning. Scapy automatically pretends to be traffic coming from File Transfer Protocol server. Additionally, the default port that Scapy selects is port 80.

54. As you review the image of the structure, anyone of these can be references. To get the fields to display, you can use the structure and the library method combined. Enter print(p.sprintf("%Ether.src% > %Ether.dst%\n%IP.src% > %IP.dst%")).


55. The “sr1()” function sends a packet and returns the corresponding answer. srp1() does the same for layer two packets, that is, the Ethernet. If you are only interested in sending packets, use the “send()” function.

56. Alternatively, Scapy can use OS sockets to send and receive packets.


57. An example of the output is shown in the screenshot.


58. The code creates a UDP socket to query the EC Council address based on the reference to the Google public DNS. To understand why you demonstrate the UDP here, you must know that when you use UDP, Scapy can create the socket without needing root privileges. This is highly important because, if you get on the box, you could load Scapy and create UDP sockets without root privileges and send data using UDP.

59. Moreover, Scapy can extend protocols—for example, you can set up DNS over TCP. This is possible because of class inheritance; it is one of the popular components to leverage in object-oriented programming.


60. Since DNS is allowed predominantly across the networks, you can use this to send messages and use TCP.

61. Another powerful feature of Scapy is automation, which allows abstraction. An example of a TCP Scanner that takes advantage of this is shown in the screenshot.


62. There are many options to build and do other tasks with this tool.

63. A table that lists some of the commands and their effects are shown in the screenshot.


64. An additional example for sending packets is shown in the screenshot.


65. Scapy also allows fuzzing because of its “fuzz()” feature. Enter send(IP(dst="192.168.177.157")/fuzz(UDP)/NTP(version=4)),loop=1).


66. In this example, you have fuzzed almost 4902 packets and sent them into the UDP port of NTP 123.


67. This enables you to quickly build fuzzing templates and sending them in a loop.

68. Next, explore an example of sr(), which allows you to send and receive. Enter sr(IP(dst="192.168.177.157")/TCP(dport=[53,80,3306,60000])).


69. The data can be retrieved using summary(), as shown in the screenshot.


70. A listing of the functions that are already coded within Scapy is shown in the screenshot.


71. You are now familiar with both Python and Scapy, which are excellent tools for testing. This concludes the lab exercise.

---------------------------

## Exercise 3: Windows Python Script and Enumeration
Objectives
- Create Python scripts and explore the enumeration of Windows

1. In this lab, you will create some simple Python scripts. First, you will perform an enumeration of the Windows machine with the "wmi python" function. Click WinPy-Python and then click Ctrl+Alt+Delete.


2. In the Password field type Pa$$w0rd and press Enter.


3. Open a File Explorer and navigate to C:\Program Files and shift + right-click on Python39 folder and choose Open PowerShell window here as shown in the screenshot.


4. In the PowerShell window, type the following commands as shown in the screenshot:

python
import wmi
conn = wmi.WMI()
for class_name in conn.classes:
if 'Process' in class_name:
print(class_name)

5. Press Enter twice. Ensure you follow the indentations.


6. To query specific system information, determine the Windows Management Instrumentation (WMI) class that can provide the relevant information. Utilize the "classes" property of the WMI object such as wmi.WMI().classes, which will return the list of WMI classes.

7. Even if you know the name of the WMI class, you will still require the exact name of the property these classes offer and methods that can perform specific operations. The process and steps to obtain this data include using the "." operator. This is a common method for extracting different components within classes.

8. To achieve this, type the following command and press Enter:

wmi.WMI().Win32_Process.methods.keys()


9. The command prints all properties from the WMI class; you must now review the methods. Type wmi.WMI().Win32_Process.properties.keys() and press Enter.


10. Once you know the properties and methods of class "Win32_Process," use the WMI class name followed by an open and close parenthesis to return the objects of the WMI class. Type the following commands:

conn = wmi.WMI()
for process in conn.Win32_Process():
print("ID: {0}\nHandleCount: {1}\nProcessName: {2}\n".format(process.ProcessId, process.HandleCount, process.Name))

11. Press Enter twice. Ensure you follow the indentations.


12. You can also list the running services using code, as shown in the screenshot.

import wmi
conn = wmi.WMI()
for s in conn.Win32_Service(StartMode="Auto", State="Running"):
print(s.State, s.StartMode, s.Name, s.DisplayName)


13. In the next example, filter out all Windows services that are stopped; in startup mode "Automatic," then use the "StartService()" method to start the Windows service. In case you want to stop the service, you can use the "StopService()" method.

import wmi
conn = wmi.WMI()
for s in conn.Win32_Service(StartMode="Auto", State="Stopped"):
if 'Update' in s.Name:
result, = s.StartService()
if result == 0:
print("Successfully started service:", s.Name)

14. Using the wmi function, you can extract the following:

Network domain
Interface MAC
Interface IP
Subnet mask
System profiles
System operation
Architecture
Computer name
Registered user

15. The screenshot shows the code to perform this task. You can enter the code to accomplish this task, or you can review the output. Minimize the notepad++ window.


16. Now, launch a command prompt and type python C:\Users\Admin\Desktop\enumwin.py and press Enter. The output and results from the command are shown in the screenshot.

Note: You can also use the wmi to connect to a remote computer if you are remote testing. To do this, you enter the following command: conn = wmi.WMI("192.168.177.1", user="username" password="P@ssw0rd@123")


17. Explore one more capability. For this, you will use the Python 3. Remember, it is always good to experiment and test your scripts with different versions. This is because, if you are on an own machine, you would be familiar with the version, which corresponds to specific functions and methods. However, always experiment with different versions in case you encounter a different version from your own. It is advisable to have multiple versions of Python on the testing machines. Create a file that contains the code shown in the screenshot.


18. Review the code and the output from the code being used, as shown in the screenshot.


19. You can improve the port scanner by adding color to the output.


20. You will use the colorama module to provide the color output. The code for the scanner is shown in the screenshot.


21. An example of the colorized output is shown in the screenshot.


22. You will find that the scanner is slow because you are not using of multi-threading concepts. To speed it up you, need to add threads.


23. You may use different techniques and functions in Python or within the tools that have been developed in Python.

24. This concludes the lab exercise.

---------------------------------

## Exercise 4: Python Web Scraping
Objectives
Use Python scripts to extract data from a website
Lab Duration: 20 Minutes

1. In this lab, you will create some simple Python scripts that take advantage of different libraries and you will explore the BeautifulSoup tool. BeautifulSoup has functions that will allow you to extract large data with little effort.

2. BeautifulSoup is a Python library designed for quick turnaround projects such as screen-scraping.

3. There are three main components of the tool:

- Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree. It is a toolkit for dissecting a document and extracting what you need. It does not take much code to write an application.
- Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You do not have to think about encodings, unless the document does not specify an encoding. Further, Beautiful Soup cannot detect one. In this case, you just have to specify the original encoding.
- Beautiful Soup sits on top of popular Python parsers.

4. The process herein involves inputting data into the tool, and then parsing it and extracting data.

5. Valuable data that were once locked up in poorly designed websites is now within your reach. Projects that would have taken hours take only minutes with Beautiful Soup.

6. By default Python-Machine machine is selected. If you are already logged in skip to step 7.


7. Let us consider the method for this exercise using the Python 3 library. In a new terminal, enter the code shown in the screenshot.

Note: 192.168.177.200 is the IP address of Web-Python machine.


8. After running the request, you will receive a Response object. This object has a status_code property, which indicates if the page was downloaded successfully. Enter the following command:

page.status_code


9. A status code starting with 2 generally indicates success, and a code starting with 4 or 5 indicates an error.

10. Print the contents of the page with the appropriately named content function. In the terminal window, enter page.content. An example of the output of this command is shown in the screenshot.


11. These data are in an extremely “raw” format; you can use BeautifulSoup to parse and clean these up. In the Python interpreter, enter the following commands:

from bs4 import BeautifulSoup
soup = BeautifulSoup(page.content, 'html.parser')


12. The process to clean up the output is appropriately named prettify. You can now print the HTML content of the page, fully formatted, using the prettify method on the BeautifulSoup object. Enter the following command:

print(soup.prettify())


13. As all tags are nested, you can move through the structure one level at a time. Select all elements at the top level of the page using the children property of soup. Note that children returns a list generator, so you need to call the list function on it.

14. In the interpreter enter the following command:

list(soup.children)


15. The command will dump all data from the list that has been generated.

16. Use the following command to extract the BeautifulSoup elements: [type(item) for item in list(soup.children)].


17. All items are BeautifulSoup objects. The first one is declared as a class; in this case, you have a series of different types of string elements.

18. If you perform the steps from earlier, you can extract specific or items of data. Enter the following command:

soup.findAll('a')

- The command will extract all items with the character “a.”


19. You have extracted all items with the character “a” in it. To extract a smaller subset, enter the following command:

soup.find('a')


20. The output shows us a one-line item.

21. Another thing to look for is the start of paragraphs. Enter the following command:

soup.findAll('p')



22. You will use different functions to scrape the data from the websites; this is dependent on how the website is built.

23. Another method to extract links is by using the following commands:

for url in soup.find_all('a'):
print(url.get('href'))


24. If you only want to grab text, enter print(soup.get_text()).



25. We can access the soup.body to get the body section. Then, grab the .text from there using the following sequence of commands:

body = soup.body
for paragraph in body.find_all('a'):
print(paragraph.text)



26. The best method to practice with this tool is to create a web page, and then query the different parameters to yield a result.

27. Enter the HTML code and save it as sample.html, as shown in the screenshot.



28. You have now created a simple HTML page to extract data. Then, enter the following Python code into a file and save it as souptest.py on Desktop:

#!/usr/bin/Python 3
from bs4 import BeautifulSoup
with open("yourname.html", "r") as f:
contents = f.read()
soup = BeautifulSoup(contents, 'lxml')
print(soup.h2)
print(soup.head)
print(soup.li)

29. The code breakdown is as follows: Print the HTML code of the three tags from the extracted data within the file. Since you have supplied the file this time, it makes it easier to observe the functioning of the different commands. A BeautifulSoup object is created; the HTML data are passed to the constructor. The second option specifies the parser.



30. In a terminal window where the file is located, enter python souptest.py. An example of the output of this command is shown in the screenshot.


31. Modify the code, so it extracts tags and text. The name attribute of a tag gives its name and the text attribute its text content. Replace the code for the print statement in your code, as shown here:


32. Once you have made the changes and saved the file; execute the code and review the output, as shown in the screenshot.


33. To extract all tags, use the recursive capability to accomplish this. One again, replace the print section with the following code:

for child in soup.recursiveChildGenerator():
    if child.name:
    print(child.name)


34. Once you have made the code changes, you now have the call that goes through the document tree and prints the names of all HTML tags.


35. This piece of code is extremely powerful. With only a few lines, you now have the capability to extract all tags from the site, which is useful for penetration testing engagements.

36. Continue and explore the tool. You have the tag for children, as shown in the next line of code:

root_childs = [e.name for e in root.children if e.name is not None].

37. You can also extract descendants, as shown in the next line of code. Get all descendants (children of all levels) of a tag:

root_childs = [e.name for e in root.descendants if e.name is not None].

38. To see this in the code, add the next two lines to the python script, and then run it:

root_childs = [e.name for e in root.descendants if e.name is not None].
print(root_childs).

39. In the text editor of your choice enter the following code:

#!/usr/bin/Python 3
from bs4 import BeautifulSoup
import requests as req
resp = req.get("http://192.168.177.200")
soup = BeautifulSoup(resp.text, 'lxml')
print(soup.title)
print(soup.title.text)
print(soup.title.parent)


40. An example of the output of the script is shown below after it is run against the OWASP Broken Web Applications (BWA) virtual machine home page.


41. This concludes the lab exercise.