## Note
### Improvements you would make to the product to make it easier to deploy

1. installation scripts/wizard: provide a script for different clients to accept input config for virtual machines ip address, username, private key to make it able to scp the installation file, modified config file and also try to disable ntp clock and start the corresponding ttcs-agent.

2. package repository: integrate ttcs agent package into standard repositories for easier installation via package managers like apt for Ubuntu/Debian

### Possible failure modes in the installation process and how you would troubleshoot them

1. incorrect configuration: misconfiguration for management_address and probe_address in the ttcs-agent.cfg file can lead to failures. Troubleshoot by checking the log file /var/opt/ttcs/log/ttcs-agent.log.

2. synchronization services not stop: the vm that does not nominated as a reference should stop the time synchronization services. Troubleshoot by checking the log file /var/opt/ttcs/log/ttcs-agent.log.

3. network issues: agent communication failed due to improper firewall or network configuration. Troubleshoot by tools like ping to diagnose network connectivity between each nodes in the cluster.

4. dependency issues: missing dependencies such as libcap. Ensure all required dependencies, like libcap. Troubleshoot by checking command `apt list --installed`.

### What tools you might use to troubleshoot deployment issues

1. log analysis: use log analysis command like `grep` or `awk` to parse and analyze both the installation and service running log to be more explicit about the error

2. network connectivity: use network connectivity checking tools to help ensure the port and protocol firewall setting is correct.

3. system monitoring: use system monitoring command like `htop` to monitor to ensure TTCS Agent is running as expected and all the dependency requirement are met.

### Possible tools you would develop or suggest to make troubleshooting easier

1. ttcs agent installation checker: a specialized checker tool that can automatically check common issues, such as configuration errors, conflicting services, network connectivity, and dependency.

2. installation wizard: provide integrations for tools like Ansible or Kubernetes for easier management and deployment of TTCS Agents in complex environments.

3. dashboard for service log: extend the current TTCS Agents dashboard to include real-time health monitoring of each agent with detailed error reporting.

4. alerts and notification: create a alert manager that sends alerts for critical issues detected in the ttcs running environment, such as communication breakdowns between agents and the coordinator.

## Setting up the environment

### create gcp virtual machine
1. setting up virtual machines
a. [for setting up cluster on gcp reference](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-zonal-cluster#console)
b. [for setting up virtual machine on gcp reference](https://cloud.google.com/compute/docs/instances/create-start-instance)

### enable clockwork license on gcp marketplace
1. add note for that the activation of the Clockwork license should take time.

2. [provide Marketplace link for gcp](https://console.cloud.google.com/marketplace)

3. should create a new project for using Clockwork Clock Sync?

4. naming? => "Tick Tock Clock Sync & Monitoring" should be "Clockwork Clock Sync"

### installing clockwork in virtual machine
1. scp installation file
2. manually overwrite the config file and scp
3. interface name not work?

    2024-01-27T02:01:39.647841+00:00 [ERROR] Invalid management address: nic0

4. stop chronyd

    sudo systemctl stop chronyd