COMP 4635 - W2 Lab 2: Managing Resources with Tagging

--- tags: comp4635(2025) --- # COMP 4635 - W2 Lab 2: Managing Resources with Tagging ## Code of Ethics :::danger * The lab exercises for the course should be attempted ONLY INSIDE THE SECLUDED LAB ENVIRONMENT documented or provided. Please note that most of the attacks described in the lab sheet would be ILLEGAL if attempted on machines that you do not have explicit permission to test and attack. The university, course lecturer, lab instructors and teaching assistants assume no responsibility for any actions performed outside the secluded lab. * The lab network should be regarded as a hostile environment. No sensitive information should be stored on your virtual machine that someone is able to gain access to it. * Do not intentionally disrupt other students who are working in the labs or hack into other student's physical or virtual machines. ::: ## Method of Submission In the following lab, each checkpoint will require you to submit some files **with designated names**. Put all files into a folder and compress them into a ZIP archive named `w2lab2-<your-id>.zip`, where `<your-id>` should be replaced with your student ID. Submit the ZIP archive [on Canvas](https://canvas.ust.hk/courses/63913/assignments/385049). There are in total **4** checkpoints, with **1** of them as bonus. The base mark is **6** points. ## Accessing the AWS Management Console 1. At the top of [the lab's Canvas page](https://awsacademy.instructure.com/courses/124953/modules/items/11835913) (login if necessary), click **Start Lab** to launch your lab. A Start Lab panel opens displaying the lab status. 2. Wait until you see the message "**Lab status: ready**", then click the **X** to close the Start Lab panel. 3. At the top of the lab page, click **AWS**. This will open the AWS Management Console in a new browser tab. The system will automatically log you in. **Tip**: If a new browser tab does not open, there will typically be a banner or icon at the top of your browser indicating that your browser is preventing the site from opening pop-up windows. Click on the banner or icon and choose "Allow pop-ups." 4. Arrange the AWS Management Console tab so that it displays alongside these instructions. Ideally, you will be able to see both browser tabs at the same time, to make it easier to follow the lab steps. ⚠ **Do not change the Region unless instructed to do so**. 5. The README provided in the lab's Canvas page may look similar, but submission details only exist in this one, and there may be changes that could affect your grading. Therefore, please refer to this lab sheet. 6. In the lab environments, access to AWS services and service actions might be restricted to the ones that are needed to complete the lab instructions. You might encounter errors if you attempt to access other services or perform actions beyond the ones that are described in the task. ## Objectives After completing this lab, you will be able to: - Apply tags to existing AWS resources. - Find resources based on tags. - Use the AWS CLI or AWS SDK for PHP to stop and terminate Amazon EC2 instances based on certain attributes of the resource. ## Scenario The environment for this lab (pictured below) consists of: - Amazon VPC named Lab VPC - Public subnet - Private subnet - Amazon EC2 Linux instance named `CommandHost` [AWS Command Line Interface (CLI) tools have been pre-installed and configured for you on this instance] - 8 Amazon EC2 Linux instances - Private instances have three custom tags applied to them: | Tag Name | content | | :------------- | :------------- | | Project | The project that the instance belongs to. The instances in this lab belong to one of two projects: **`ERPSystem`** and **`Experiment1`**. | |Version|The version of the project that this instance belongs to. All Version tags are currently set to 1.0.| |Environment|One of three values: **`development`**, **`staging`**, or **`production`**.| In the Task portion of this lab, you will log in to the Command Host and run some commands to find and change the Version tag on all development instances. You will run several examples that show how you can use the JMESPath syntax supported by the AWS CLI **`--query`** option to return richly formatted output. You will then use a set of pre-provided scripts to stop and re-start all instances that are tagged as belonging to the **`development`** environment. ![image](https://hackmd.io/_uploads/rydJV_r4lg.png) ## Task 1: Using Tags to Manage Resources In this task, you will log in to the Command Host, and use the AWS CLI to find a set of resources according to their tags. You will then use the AWS CLI to change the value of one of the tags. ### Connect to the Command Host 1. In the **AWS Management Console**, on the Services ⌄ or ᎒᎒᎒ menu, choose **Compute**, then **EC2**. 2. In the left navigation pane, choose **Instances**. 3. Select the **`Command Host`**. 4. Copy the **IPv4 Public IP** from the Description in the lower pane. 5. Do the following. * Choose the Details drop down menu near the top of the lab's Canvas page, then choose Show. A Credentials window will open. * Choose the **Download PEM** button and save the **`labsuser.pem`** file. * Then exit the Details panel by choosing the **X**. :::spoiler For Linux / MacOS user 6. Open a terminal window, and change directory `cd` to the directory where the `labsuser.pem` file was downloaded. For example, run this command, if it was saved to your `Downloads` directory: ```bash cd ~/Downloads ``` 7. Change the permissions on the key to be read only, by running this command: ```bash chmod 400 labsuser.pem ``` 8. Return to the terminal window and run this command (replace **<public-ip\>** with the **Public IPv4** value you copied to your clipboard earlier in the lab): ```bash ssh -i labsuser.pem ec2-user@<public-ip> ``` 9. Type `yes` when prompted to allow a first connection to this remote SSH server. Because you are using a key pair for authentication, you will not be prompted for a password. ::: :::spoiler For Windows user 6. If you are using the latest version of Windows 10 or Windows 11, it should include a built-in SSH server and client that are based on OpenSSH. If SSH is not pre-installed in your Windows machine, visit this link and follow the same process as installing SSH on "Windows Server 2022" in your Windows 10/11 machine: https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse?tabs=powershell&pivots=windows-server-2022 7. Open a Powershell window, and change directory `cd` to the directory where the `labsuser.pem` file was downloaded. For example, run this command, if it was saved to your `Downloads` directory: ```bash cd .\Downloads ``` 8. Run this command (replace **<public-ip\>** with the **Public IPv4** value you copied to your clipboard earlier in the lab): ```bash ssh -i .\labsuser.pem ec2-user@<public-ip> ``` 9. Type `yes` when prompted to allow a first connection to this remote SSH server. Because you are using a key pair for authentication, you will not be prompted for a password. ::: :::success ### Checkpoint 1 (2 marks) Submit a screenshot, named **`cp1.{png/jpg/jpeg}`**, showing that you successfully connected to the instance via `ssh`. Make sure the screenshot shows the following: - The `ssh` command you used to connect - The output once the connection is established ::: ### Finding Development Instances For The Project Now that you are logged in, you can use the AWS CLI to find the resources in your private subnet that belong to the **`ERPSystem`** project and are in the Environment named **development**. You will also see how to use the AWS CLI **`--query`** option to produce richly formatted results. 10. To find all instances in your account that are tagged with a tag of **`Project`** and a value of **`ERPSystem`**, copy the following command and run it in the Linux terminal window: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" ``` The command should output the full set of parameters available for all seven instances that are tagged **`Project=ERPSystem`**. This is a lot of output, and most of it does not apply to this lab. In the next step, you will use the **`--query`** parameter to narrow down the results. 11. Use the `--query` parameter to limit the output of the previous command to only the instance ID of the discovered instance: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" --query 'Reservations[*].Instances[*].InstanceId' ``` Your output entries will now consist of a list of instance IDs: ```bash [ [ "i-135b491e" ], [ "i-3e584a33" ], … ] ``` The **`--query`** command used in this example uses the JMESPath wildcard syntax to specify that the command should iterate through all reservations and all instances and return the InstanceId for each instance in the return results. This is an improvement over returning every property of our instances. But what if you want to include multiple fields in the output? 12. Copy the following command and run it in the Linux terminal window to include both the instance ID and the Availability Zone of each instance in your return result: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" --query 'Reservations[*].Instances[*].{ID:InstanceId,AZ:Placement.AvailabilityZone}' ``` Two name/value pairs are returned for each result. This command builds on the previous command’s use of the JMESPath syntax by using curly braces to specify a query for multiple properties on each instance returned: `object.{Alias1:PropertyName1,Alias2:PropertyName2,[…]}` As seen here, you can specify an alias for each property in order to return a more abbreviated output format. With this output, you can clearly see that your filter worked, and you are only seeing instances that are associated with the project **`ERPSystem`**. However, you still will probably not be able to identify which instances are being returned, based on this information. In the next steps, you will see how to include the value of your custom tags in the return output. 13. To include the value of the **`Project`** tag in your output, copy and run the following command in the Linux terminal: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" --query 'Reservations[*].Instances[*].{ID:InstanceId,AZ:Placement.AvailabilityZone,Project:Tags[?Key==`Project`] | [0].Value}' ``` Your output now includes the value of the Project tag: ```bash [[{ "Project": "ERPSystem", "AZ": "us-west-2a", "ID": "i-3250b838" }],...] ``` The value of a specific named tag can be retrieved via a JMESPath query, using the following syntax: ```bash Tags[?Key==`Project`] | [0].Value ``` This syntax instructs JMESPath to find all elements within the **`Tags`** array that have a **`Key`** value of **`Project`**. The output of that command—which will be a single Tags element—is then piped to another command that selects the first instance of this filtered set and selects the named parameter **`Value`**, which is the value of the **`Project`** tag. This result is then assigned the alias **`Project`**. 14. Copy and run the following command to also include the `Environment` and `Version` tags in your output: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" --query 'Reservations[*].Instances[*].{ID:InstanceId,AZ:Placement.AvailabilityZone,Project:Tags[?Key==`Project`] | [0].Value,Environment:Tags[?Key==`Environment`] | [0].Value,Version:Tags[?Key==`Version`] | [0].Value}' ``` The results will give you a fuller picture of the instances currently associated with the project named **`ERPSystem`**: ```bash [[{ "Environment": "production", "Project": "ERPSystem", "Version": "1.0", "AZ": "us-west-2a", "ID": "i-3250b838" }], ... ] ``` 15. Finally, add a second tag filter to see only the instances associated with the project named **`ERPSystem`** that belong to the Environment named **`development`**: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" "Name=tag:Environment,Values=development" --query 'Reservations[*].Instances[*].{ID:InstanceId,AZ:Placement.AvailabilityZone,Project:Tags[?Key==`Project`] | [0].Value,Environment:Tags[?Key==`Environment`] | [0].Value,Version:Tags[?Key==`Version`] | [0].Value}' ``` You should see only two instances returned by this command, both with a **`Project`** tag value of `ERPSystem` and an **`Environment`** tag value of `development`: ```bash [[{ "Environment": "development", "Project": "ERPSystem", "Version": "1.0", "AZ": "us-west-2a", "ID": "i-9552ba9f" }], ... ] ``` ### Changing Version Tag for Development Process In this procedure, you will change all of the **`Version`** tags on the instances marked as **`development`** for the project **`ERPSystem`**. You could individually set these properties on each affected instance, but an automated approach is more practical. You can use a simple Linux Bash shell script to build on the queries you built earlier and modify tag entries as a batch operation. 16. On the CommandHost, open the file **`/home/ec2-user/change-resource-tags.sh`**: ```bash nano change-resource-tags.sh ``` 17. Examine the contents of the script: ```bash #!/bin/bash ids=$(aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" "Name=tag:Environment,Values=development" --query 'Reservations[*].Instances[*].InstanceId' --output text) aws ec2 create-tags --resources $ids --tags 'Key=Version,Value=1.1' ``` This script first uses the command `aws ec2 describe-instances` to return only a list of instance IDs for the development machines that belong to the **`ERPSystem`** project. It then passes those values to the `aws ec2 create-tags` command, which either creates a new tag or (in this case) overwrites an existing tag. Notice how the first command uses the **`--output text`** option to manipulate the return results as text instead of as JSON. Using this command instead of JSON on a simple return result—in this case, a list of IDs—can make it easier to manipulate the return result and pass it to other commands. 18. Close the `nano` editor (with `Ctrl+X` on Linux) and run this command from the Linux command prompt: ```bash ./change-resource-tags.sh ``` 19. To verify that the version number on these instances has been incremented and that other non-development boxes in the **`ERPSystem`** project have been unaffected, copy and run the following command: ```bash aws ec2 describe-instances --filter "Name=tag:Project,Values=ERPSystem" --query 'Reservations[*].Instances[*].{ID:InstanceId, AZ:Placement.AvailabilityZone, Project:Tags[?Key==`Project`] |[0].Value,Environment:Tags[?Key==`Environment`] | [0].Value,Version:Tags[?Key==`Version`] | [0].Value}' ``` :::success ### Checkpoint 2 (2 marks) Submit a text file, named **`cp2.{txt/md}`**, showing the verification in step 19. Make sure the text file includes the following: - The output of the command - The instances' details ::: ## Task 2: Stop and Start Resources by Tag In this task, you will use a pre-provided script to stop and start a set of instances tagged as development instances. ### Examining the Stopinator Script 20. On the Command Host Instance, `cd` into the directory `aws-tools` in the home directory: ```bash cd aws-tools ``` 21. Open the file **`stopinator.php`** and examine its contents: ```bash nano stopinator.php ``` The **`stopinator.php`** script is a simple script that uses the AWS SDK for PHP to stop and restart instances based on a set of tags. This enables scenarios such as shutting off your development environment servers at the end of the day and restarting them the next morning. The script will look in every AWS region for instances that match the specified tags. The script takes the following arguments: - **`-t`**: A set of tags in the following format: `name=value;name=value` The script converts these tags into the format expected by the AWS PHP call `Ec2::DescribeInstance()`. If this optional parameter is absent, the script will identify and shut down all running Amazon EC2 instances in the account. - **`-s`**: A Boolean parameter; no arguments are required. When this parameter is present, instances identified by **`-t`** are started instead of stopped. 22. Exit your `nano` editor (with `Ctrl+X` on Linux). ### Stopping and Restarting ERPProject Development Process In this task, you will use the `stopinator.php` script to bring down and bring back up your development environment for the **`ERPSystem`** project. 23. From the Linux shell, run the `stopinator.php` script: ```bash ./stopinator.php -t"Project=ERPSystem;Environment=development" ``` The output should look like this, indicating that two instances will be stopped in your current AWS region. (Your results will differ depending on the region in which your lab is running.) ``` Region is us-east-1 No instances to stop in region Region is us-west-1 No instances to stop in region Region is us-west-2 Found instance i-9552ba9f Found instance i-d35fb7d9 Stopping all identified instances... […] No instances to stop in region Region is sa-east-1 No instances to stop in region ``` 24. Go back to the **EC2** landing page. 25. In the navigation pane, choose **Instances**. 26. Verify that two instances are stopping or have already been stopped. 27. Return to the SSH session for Command Host, and from the Linux prompt, restart your instances with the following command: ```bash ./stopinator.php -t"Project=ERPSystem;Environment=development" -s ``` 28. Return to the EC2 Management Console window and verify that the two instances that were previously shut down are now restarting. :::success ### Checkpoint 3 (2 marks) Submit a screenshot, named **cp3.{png/jpg/jpeg}**, showing verification in step 28. Please take your screenshot of the instance list in **Instances** page under EC2. You can repeat step 23-28 if you missed the window to grab a screenshot that fulfills the requirements. Make sure the screenshot shows the following: - The list of instances - The instances' states, where all should be **Running**, except the two targeted instances should be in **Pending** state. ::: ## (Challenge) Task 3: Terminate Non-Compliant Instances In this Challenge, you will be asked to find a way to terminate instances that do not conform to certain security guidelines. **Note**: If you are already familiar with AWS, we recommend that you try this challenge yourself **before** reading the detailed solution provided in the next section. When you have completed the challenge, check your work by reviewing the detailed solution. ### Challenge Description **Scenario**: Your company wants you to create automated processes that will automatically terminate instances that might allow a possible security breach. You have identified a list of security risks and are now deciding how to implement them efficiently by using either AWS CLI commands or the PHP SDK for AWS. **Challenge**: Your first security task is simple: find all instances in your private subnet that do not implement the **`Environment`** tag, and terminate them (i.e., a “tag-or-terminate” policy). **Hints**: - If you are not comfortable with PHP or a similar programming language (such as Python or Ruby) for which an AWS SDK is available, try to use a series of AWS CLI commands to perform this task. - The AWS PHP call `Ec2::terminateInstances()` can terminate instances. The equivalent AWS CLI command is `aws ec2 terminate-instances`. - You can use the **`stopinator`** script from section 2 as a reference for any code you write. **Challenge Solution Overview** There are multiple ways to approach this problem using a variety of programmatic or command-line solutions. The general solution to the problem consists of the following steps: - Identify all of the instances that currently have the **`Environment`** tag defined. - Compare this against the list of all available instances, and record the instance IDs of any instances that are not part of the list obtained from Task 1. - Supply the instance IDs of the non-tagged instances to AWS by using the aws ec2 stop-instance command (AWS CLI) or the **`Ec2::terminateInstances()`** API call (PHP). The following solution demonstrates how this problem could be solved using a PHP script. ### Task 3.1: Review the Tag-Or-Terminate Script 29. Open the file **terminate-instances.php** with the nano editor. ```bash nano terminate-instances.php ``` 30. Examine the **params** block for this script. Note that it takes two arguments: the current region that you are running in (**region**), and the ID of a subnet (**subnetid**). The code uses the **subnetid** argument to determine where to look for non-compliant instances. 31. Examine the first block of code, beginning with the comment `# Obtain a list of all instances with the Environment tag set`. This block of code uses the **describeInstances()** method, a filter to find all instances that have the **Environment** tag defined, regardless of the tag’s value. It stores all of the instance IDs that it finds in a hash table. 32. Examine the second block of code. This code examines all instances within your subnet and compares them to the list of instances that are tagged with the **Environment** tag. If an instance is not in the tagged list, then its instance ID is added to a list of instances to terminate. 33. Examine the last section of the script. These lines use the list of non-compliant instance IDs as an argument to the **terminateInstances** method. ### Configuring Environment to Test Script Before running the script, you will need to alter a couple of instances in your lab so that they no longer have the **Environment** tag defined. 34. Return to your EC2 Management Console and observe the instances running in your lab environment. 35. Select one of the instances in your private subnet (This means instances without public IP addresses). 36. On the **Tags** tab for the instance, choose **Manage tags**. 37. Find the Environment tag, and choose the **remove** icon. 38. Choose **Save**. Repeat this process for one other instance in your private subnet. ### Run the Script 39. In the EC2 Management Console, select one of the instances in your private subnet. 40. On the **Networking** tab for your instance, find the **Availability zone** field, and copy all but the last letter to a text file. This value will be referred to as region in a subsequent procedure. 41. Find the **Subnet ID** field, and copy its value to a text file. This value will be referred to as subnet-id in a subsequent procedure. 42. Return to your SSH session, and run the **terminate-instances.php** script (replacing the <region\> with your region and <subnet-id\> with your subnet-id): ``` ./terminate-instances.php -region <region> -subnetid <subnet-id> ``` You should see something similar to the following results: ``` Checking i-dd3a90d1 Checking i-a4248ea8 Checking i-793a9075 Checking i-a9248ea5 Checking i-aa248ea6 Checking i-da3a90d6 Checking i-a13b91ad Checking i-a23b91ae Checking i-ab248ea7 Terminating instances... Instances terminated. ``` :::success ### Checkpoint 4: Bonus 1 (3 marks) Submit a Python file, named **`{cp4/stop-instances}.py`**, showing your implementation of `terminate-instances.php` in Python with the `boto3` library. **However, instead of terminating instances, you should stop them instead**. To create a virtual environment with Python (which is ideal for `boto3` installation), you can do the following: 1. Create a virtual environment named `.venv` ```bash python3 -m venv .venv ``` 2. Source the virtual environment ```bash source .venv/bin/activate ``` You should see that your terminal now has a `(.venv)` prepended. 3. Install `boto3` ```py pip install boto3 ``` You can refer to the documentation of `boto3` [here](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html). 4. Create and write the python file ```bash nano stop-instances.py ``` 5. Run the Python file like the PHP file. ```bash python3 ./stop-instances.py -region <region> -subnetid <subnet-id> ``` To help you with your task, a skeleton is given to you below: ```py import argparse import boto3 # Default values, same as the PHP file region = "us-west-2" subnet_id = "" profile = "default" # Only needed if not using IAM roles # Parse command-line arguments parser = argparse.ArgumentParser(description="Stops all EC2 instances in a specific subnet that do not have the \"Environment\" tag.") parser.add_argument("-region", help="The AWS region to use", default=region) parser.add_argument("-subnetid", help="The subnet ID to filter instances", required=True) args = parser.parse_args() region = args.region subnet_id = args.subnetid def main(): # Initialize the EC2 client session = boto3.Session(profile_name=profile) ec2 = session.client("ec2", region_name=region) # Use the describe instances paginator to fetch all instances with the "Environment" tag good_instances = set() paginator = ... # TODO: Get a paginator that describes instances # Filter for instances with the "Environment" tag tag_filter = [{"Name": "tag-key", "Values": ["Environment"]}] for page in paginator.paginate(Filters=tag_filter): for reservation in page["Reservations"]: for instance in reservation["Instances"]: instance_id = ... # TODO: Get the instance's ID good_instances.add(instance_id) # Use the paginator to fetch all instances in the specified subnet stop_instances = set() subnet_filter = [{"Name": "subnet-id", "Values": [subnet_id]}] for page in paginator.paginate(Filters=subnet_filter): for reservation in page["Reservations"]: for instance in reservation["Instances"]: instance_id = ... # TODO: Get the instance's ID print(f"Checking {instance_id}") if ...: # TODO: The condition when an instance should be added to the list of instances to stop stop_instances.add(instance_id) # Stop instances without the "Environment" tag if stop_instances: print("Stopping instances...") ... # TODO: **Stop**, not terminate, the instances print("Instances stopped.") else: print("No instances to stop.") # Run main if __name__ == "__main__": main() ``` ::: ## Lab Complete Congratulations! You have completed the lab. Remeber to submit the necessary files [on Canvas](https://canvas.ust.hk/courses/63913/assignments/385049)! 43. Choose End Lab at the top of this page and then choose Yes to confirm that you want to end the lab. A panel will appear, indicating that "DELETE has been initiated... You may close this message box now." 44. Choose the **X** in the top right corner to close the panel.