---
# System prepended metadata

title: DS Project 2

---

# DS Project 2

Project text: https://docs.google.com/document/d/1Is2QFO20RjxVrZMSMCxsBa-FUgGgaIJ7e_o3CeQKN6w/edit

## Project architecture
* Client
* Name server
* Storage nodes

Layout of our project architecture can be found here:
https://app.diagrams.net/#G10YDMMuvrM17iaShf1-8isCTX-sQ_z9ys

## Client

```
docker-compose up --build -d 
```

First of all we need to understand what is the client? From this citation from the Project description we can say that the client is some sort of centralized software, we will make simle CLI (command line interface) for the client.

> Responsible for making the distributed nature of the system transparent to the user.

**Client CLI should be able to handle the next commands:**
* **Initialize.** Initialize the client storage on a new system, should remove any existing file in the dfs root directory and return available size.
* **File create.** Should allow to create a new empty file.
* **File read.** Should allow to read any file from DFS (download a file from the DFS to the Client side).
* **File write.** Should allow to put any file to DFS (upload a file from the Client side to the DFS)
* **File delete.** Should allow to delete any file from DFS
* **File info.** Should provide information about the file (any useful information - size, node id, etc.)
* **File copy.** Should allow to create a copy of file.
* **File move.** Should allow to move a file to the specified path.
* **Open directory.** Should allow to change directory
* **Read directory.** Should return list of files, which are stored in the directory.
* **Make directory.** Should allow to create a new directory.
* **Delete directory.** Should allow to delete directory.  If the directory contains files the system should ask for confirmation from the user before deletion.

**Requirements added by our team:**
* **Connect.** Before applying any of the commands above the client should be able to connect to the name server or pool of the name servers.

**What needs to be done:**
* Think about the commands and their usage.
* Think of the API calls needed in order to interact with Name Server.
* Think of the API calls needed in order to interact with Storage Server.
* Simple CLI which is able to handle all of the commands above.

**Useful links:**
* [**Fire**](https://towardsdatascience.com/a-simple-way-to-create-python-cli-app-1a4492c164b6) - library for creating python-based CLI

## Name Server

All the information you read here is found in this article: https://www.edureka.co/blog/apache-hadoop-hdfs-architecture/

**Here you can find the HDFS Name Node definition:**
NameNode is the master node in the Apache Hadoop HDFS Architecture that maintains and manages the blocks present on the DataNodes (slave nodes). NameNode is a very highly available server that manages the File System Namespace and controls access to files by clients. I will be discussing this High Availability feature of Apache Hadoop HDFS in my next blog. The HDFS architecture is built in such a way that the user data never resides on the NameNode. The data resides on DataNodes only.

**Basic functions of NameServer:**
1) Stores the distributed file system image
2) On read API call serves the client IP address of one of the storage servers.
3) On fetch API call serves the client data directly from its own knowledge of DFS.
4) On write API call serves the client IP addresses of all of the storage servers.


**Functions of Name Node:**
* It stores file system image
* It stores file system change log
* It stores the status of each data node
* It manages the data replications if needed
* Recieves updates from the Data Nodes 

## Data Node / Storage Server

All the information you read here is found in this article: https://www.edureka.co/blog/apache-hadoop-hdfs-architecture/

**Here you can find the HDFS Data Node definition:**
DataNodes are the slave nodes in HDFS. Unlike NameNode, DataNode is a commodity hardware, that is, a non-expensive system which is not of high quality or high-availability. The DataNode is a block server that stores the data in the local file ext3 or ext4.

* [x] **Initialize.** Initialize the client storage on a new system, should remove any existing file in the dfs root directory and return available size.
* [x] **File create.** Should allow to create a new empty file.
* [x] **File read.** Should allow to read any file from DFS (download a file from the DFS to the Client side).
* [x] **File write.** Should allow to put any file to DFS (upload a file from the Client side to the DFS)
* [x] **File delete.** Should allow to delete any file from DFS
* [x] **File info.** Should provide information about the file (any useful information - size, node id, etc.)
* [ ] **File copy.** Should allow to create a copy of file.
* [ ] **File move.** Should allow to move a file to the specified path.
* [ ] **Open directory.** Should allow to change directory
* [ ] **Read directory.** Should return list of files, which are stored in the directory.
* [ ] **Make directory.** Should allow to create a new directory.
* [ ] **Delete directory.** Should allow to delete directory.  If the directory contains files the system should ask for confirmation from the user before deletion.



**Functions of Data Node:**
* Stores the data in their local FS
* Sends the updates of local FS to Name Node in order to update the DFS

**что делает?**
1) Высылать файлы по запросу
2) Скачивать/принимать файлы по запросу
3) Кидать информацию о себе в Name Server
4) Кидает запрос по типу (1) на Storage Server с необходимыми данными





