~~Q1. The link provided provided for reference code of NFS implementation is wrong, can you please correct it.~~

Q2. If I understood it correctly, we are required to form an 'illusion' of file paths to the user. This is done by the naming server. Just like how physical memory addresses are virtualized, the location of files on storage servers is virtualized by the naming server.
If this is the case, how exactly should we virtualize? What should be the paths available to the user? Should the filesystem consist of directories like ss1/, ss2/, and so on where all the files in the ss1/ directory are stored on storage server 1, or do we decide on some other naming scheme, or is this scheme decided by someone else?
[SA] The point is kind of to maintain the abstraction of which storage server(SS) the file is actually stored on, which is maintained by the naming server(NS). So when a user stores a file in `path/to/file`, it may be stored on any SS, depending on your chosen policy (random/least full/whatever). The user will query the NS to find which SS the file is on and then query the corresponding SS with the same path to get the file. So even if the file moves between SS, the only thing that will change is the response of the NS. So in short, your path system should not give away which SS stores the file (since it may also be wrong). The "naming scheme" here will be done by the user as if they were storing files on their local machine.

Q3. Can we create 2 sockets for naming server??
[SJ] Elaborate on why do you need 2 sockets. You can either DM me([e-mail]) or Divyansh([e-mail]) for the same.

Q4. Are we required to play the audio file in real time in the client while the files are being streamed from server?
[DP] You are expected to redirect incoming packets to the mp4 player using system calls. How you stream can be left open ended.

Q5. Can we use `libvlc` ?
[DP] Any music player which can be embedded in the terminal and used to stream music directly is allowed. We expect a proof of concept.

Q6.While Handling the read command should the file be diplayed directly on the terminal or should we use vim or some other way entirely?
[SJ] You can show on terminal on client side the contents of the file.

Q7. Just to confirm the clients will always provide the absolute paths right ? Because no actual files / directories exists on the user end.

[DP] You specify paths as provided by NS to the client as accessible paths. NS would ideally provide an abstraction as mentioned in Q2. So if the file is in storage server 3 with path './dir1/file1.txt' it gives the path as it is.

Q8. Is it guaranteed that 2 clients won't stream on the same laptop at the same time?
[SJ] We won't test this.

Q9. Can you provide/can we assume any maximum on number of clients, storage servers, path length, etc?
[SJ] Yes you can assume.

Q10. Do we need to ensure persistence of files even after the program is stopped; should the data in the Storage servers remain even after they are shut down?
[SJ] Yes.

Q11.

What do we mean by "once all Storage Servers are initialized"? Do we assume a fixed number of SSs will connect initially?
[DP] This statement is to give you an idea about the flow of the program. After initialising a few storage servers, client can start sending requests and get tasks done. You can have any storage server arriving at any time (Hint:threads!). Also, even when there are no storage servers, your client can still send requests but it doesn't make sense at that point since there are no accessible paths. So, the crux being, you initialise a few storage servers and then send requests. However a storage server can register at any time during the execution of the program.

Q12. Can we use protobuf-c library for communication (Makes everything cleaner, easier to manage)?
[SJ] Nopes. Use standard socket libaries for communication.

Q13.Does reading and writing only deal with text? Or in write are we expected to send a local file (as in something from the client machine)
[DP] Text is fine for proof of concept but if you want you can do otherwise. Since you handle music streaming anyways to read different file types which covers it.

Q14. All our folders are managed only on the NM side, SS not concerned with any folder behavior. Is this ok? (All specification funcionalities are maintained.)
[SJ] Elaborate on what do you mean by "not concerned with any folder behaviour".

Q15. In initialization spec under "List of Accessible Paths" is it ok if this is completely managed by NM instead? (Subset of q14)
[SJ] Storage server will provide what all Accessible Paths are present in it to Naming server. How storage server gets the Accessible paths list is upto. You can take command like args or you can take input. 

Q16. To elaborate on questions 14,15 the plan is to not store any data about folder structure etc on storage servers. Storage servers are only responsible to store files (filenames can be unique identifiers to avoid clashes). NM stores all details about files also. When READ <nfsfilepath> is executed, the NM looks up the unique identifier, passes the unique identifier and the storage server ip:port, the client uses this identifier and sends a request to storage server to get the file.
Also, b) If you still require us to send a list of files, we will send the list of identifier file ids to the NM
This question partly comes up because we feel this process of sending a list is not necessary, when a file is queried and it cant be accessed it can be handled accordingly on the spot.
[DP] When a storage server registers, it needs to provide the set of files it wants to expose to the filesystem. This step is important. I am not sure what you mean by ```not maintaining a folder structure at the storage server end``` but your storage server should essentially provide some input to the NS while registering which would inform the NS which files the SS makes visible to the client. You can have any arbitrary folder structure at the SS end, you just provide the paths. NM then stores the unified set of all accessible paths present in every storage server. Suppose your storage server had a base directory with 10 files, 2 of which were storing confidential data about the storage server configurations. You wouldn't want this to be exposed to clients. Hence, the ideas of exposing accessible paths from the storage server side. Now, a storage server can have files in any arbitrary folder structure. The idea of giving the entire folder structure to the NS is that if you wanted to copy the entire folder in that storage server to another, you can specify the directory itself. If you can implement it in a way which ensures the directories at the storage server level can be easily copied, created and deleted while being in sync with the list of accessible paths, you are free to implement the above logic.
    
Q17. Do you want the ability to manipulate files and folder structures externally without using NFS (i.e fiddle around with Storage Server files when NFS not running) and then have the changes show up on boot? Or is it fine if we assume that all manipulation is done via NFS only.
If we assume this then we can make the entire folder manipulation software in the NM itself, instead of the complications of directory structures inside each storage server
b) If you really need aforementioned ability, would it be ok if we assign each StorageServer a top level directory randomly like SS1 <-> /bagel,SS2 <-> /pakoda etc , or if you expect something different please do elaborate
[DP] You can assume that all manipulations are carried out via NFS only. 
(b) If what you are asking is assigning a default top level directory for all storage servers say all files in /bagel then that's fine but there should be sub-directories say SS1 <-> /bagel/dir1 and it can also have files inside, because a file system doesn't just consist of files, we have directories too.
    
Q18. Do we need to handle backup cases when a SS registers? or is it enough if we do it when new files are written
[DP] You should backup data as soon as a storage server registers and there are more than 2. If you modify a file or create/delete that should reflect in the backup too.
    
Q19. After a storage server comes back up after going off, how can we tell if it is the same storage server ? Can we assume that the ip address and the port remain same ?
[DP] Yes you can.

Q20. With respect to Answer for Q18, this could cause potential complications
Say SS1 backups to SS2, SS3 initially
Now SS1 comes on, SS4, SS5 come on
SS1 starts backup process to SS4, SS5 although 2 copies already exist on SS2, SS3 which come on later.
[DP] Referring Q19, you also check if the storage server was initially backed up. If it was you should ideally match back up files but that is bonus. But you dont backup already backed up storage servers anyways. If you go on to implement the bonus you would ideally match all files in the revived storage server with the backup. Otherwise, it's fine to just not update the backup and also not create new backups.
    
Q21. Consider this scenario
SS1 has following files
/abc/1/hello
SS2 has
/abc/2/hi
now user decides to create a folder /abc/3/
under whom should it be created? (Or random?)
[DP] Random.
    
Q22. I did not really understand this part can u please explain clearlyy

3.5 Backing up Data [70 Marks]

    Failure Detection: The NM should be equipped to detect Storage Server (SS) failures. This capability ensures that the NFS can respond promptly to any disruptions in SS availability.
    Replication: Implement a replication strategy for data stored within the NFS. This strategy involves duplicating every file and folder in an SS in two other SS (once the number of SS exceeds two). In the event of an SS failure, the NM should be able to retrieve the requested data from one of the replicated stores. However, at this stage, only read operations should be allowed.
    Asynchronous Duplication: Every write command should be duplicated asynchronously across all replicated stores. The NM does not wait for acknowledgment but ensures that data is redundantly stored for fault tolerance.
[SJ] Should have attended the tutorial.
    
Q23. In response to the answer of Q20 If we have to store details about all files and their backups anyway, then what is the need of sending the entire list of files again to the name server whenever a storage server starts? I think the intended way of doing it is that on WRITE operations only backups take place.
b) Can any new SS connect to NM at any point in time? If so backup on SS start will again become more complicated 
[SJ] a) Let's suppose before making a WRITE request your original SS goes offline, then how can you get data from backups atleast for reading files. 
b) Yes any new SS can connect to NM at any point. It becomes complicated but you have to use concurrency concepts and this can be handled.
    
Q24. In the `CREATE <path> <name>` call, what exactly is `name`? If it's the name of the file to be created, can't it just be a part of the path?
[SJ] You are free to change request format as you like. This was just an example to demonstrate how pipeline should look like.
    
Q25. Can we use `ao/ao.h` header file for streaming audio?
[SJ] All POSIX libraries are allowed. if `ao/ao.h` is POSIX then yes.
    
Q26.Suppose a ss trying to reconnect again to nm,since we already registered all files before time,we are not registering again,In redundancy part 
a) since we are not adding new paths we don't really care any files that added,we have to accodimate only the files deleted through backup servers right?
b)is checking accessible path is sufficent or checking the actual data inside the file is also needed?
[DP] a) What do you mean by deleted through backup servers.
    b) Latter is recommended but due to its complexity we will not penalize the former. 

Q27)can we fix the storage servers which has to backup,SS1 and SS2 for any other server and SS3 for SS1 and SS2 backup or should it be random?
[SJ] The backups should be in 2 storage servers. Which SS's to assign for backup it's upto you.

Q28) For the operations issued by clients to storage servers by client the acknowledgement to client will be sent directly by stoerage servers or indirectly by SS via naming server?
[SJ] Getting from SS is fine.
    
Q29) Can we assume the File/Directory names to be unique, i.e ./DIR1/a.txt in Storage server 1 will not be in storage server 2 with the same path, i.e ./DIR2/a.txt?
[DP] Yes.

Q30) How many bonus points would we get if we give two golden tickets for coldplay to the TA's ?
[SJ] ✅. 

Q31) Can we assume ASYNC flag will be provided for asynchronous write?
[DP] Yes you can assume, but ideally should have done it by setting a size threshold.
    
Q32)Regarding Q26 if the files deleted when the ss goes off,how to handle that files when the ss comes back?Have to copy from backup ss?
[DP] You need not handle this. Refer Q17 (a).

Q33) If a folder is given as accessible path will all the files present in it be accessible, also will all the subfolders in it be accessible?
If yes, then what do you mean by NM will also dynamically add the new path in this case into the list of accessible paths when creating a file/folder, won't this be get taken care by itself.
[DP] You can take it as an assumption that if a folder is accessible it means its entire content is accessible. Otherwise instead of specifying folders you only specify exact files which are accessible. 
    
Q34) Can we use the utash library?
[DP] If this is for hashing, yes.
    
Q35) Suppose we make the client upload/write a very large file to a storage server. In this case does the client wait for the 'network send' part to finish and then recieve the write acknowledegement asynchronously? OR Do you want us to do BOTH the 'network send'' and 'wait for acknowledegement' asynchronously?

[DP] Send data to SS. SS starts writing Asynchronously after sending an ACK to client. After Async write is completed it sends ACK2 to client saying write succesful.
    
Q36) Are we allowed to use execvp and other cousins of exec?
[DP] Elaborate on your usecase. If this is for music streaming it is mostly allowed but I don't think anywhere else it should be.
    
Q37) Can we make an assumption that if a particular folder is stored on a storage server, the whole subtree under that folder is stored in the same storage server?
[DP] Yes.
    
Q38) Can I assume a maximum of 65536 servers?
[DP] Yes and please stop asking such questions. This project has been left very open-ended and any assumption which doesn't directly contradict a requirement in the course project document need not be validated from the TAs everytime. It just creates a bottleneck for even you to wait for the TAs response and we have our own deadlines to meet so it isn't possible to update doc so often and for such small queries.
    
Q39) In my understanding, there are 3 types of writes 1) synchronous write 2) asynchronous write 3) normal write , can u please differentiate between synchronous write and write? 
[DP] It's either synchronous or asynchronous. What is a normal write?
    
Q40) How are we supposed to handle the locking cases, should a write block when another write is running, or should the second write print an error?
[DP] You can either wait till a timeout period for write to be over and then tell client or deny write permission immediately.
    
Q41) Adding on to Q27, just to confirm, we need to have 2 backup servers for n storage servers or does each storage server have it's data backed up in two different storage servers (For example if we have 4 storage servers SS1-SS4, do I need to have dedicated servers SS1-SS2 for backup such that all of them backup to these 2 only, or SS1 can store to SS2 and SS4 and so on) ?
[DP] There aren't dedicated backup servers. You utilise whatever storage servers you have to backup others. If you have 5 storage servers you choose 2 storage servers for a third storage server which you want to back up. Backing up all data in just two specific servers is bad because if they fail all the backup is gone.
    
Q42) Are system commands allowed for copy?
[DP] Elaborate.
    
Q43) Consider the situation where we have /abc/1/ stored in SS1, backed up in SS2 and SS3. a) SS1 is down. COPY /abc/1/ /123/xyz/ is executed. Are we required allow COPY in this case?
[DP] If a SS is down, you only allow READ on it's files. Nothing else.
    
Q44) Can we use zip command and zip a folder, then send it over for copying an entire folder?
[DP] Does it copy each file with it's contents?
    
Q45) Can we assume that if file /abc/123/a.txt is being read, DELETE /abc/ wont be executed?
[DP] You can solve it using concurrency.
    
Q46) Can we use execvp commands?
[DP] For what exactly.
    
Q47) For client it says it should be able to do various file-related operations such as reading, writing, deleting, streaming, and more. What all is meant by "more" ?
[DP] Copy, create, delete....it's in the doc :) .

Q48) Refering to Q39 if Sync flag is not provided and data is not large (not greater than threshold for async) I'm thinking it as normal write… correct me if I'm wrong please
[SJ] You are correct.
    
Q49) can we exec or related functions to get the file info?
[SJ] Utilise your knowledge of MP1(Hint:- Reveal)

Q50) So can the client wait for the data to be first sent to the storage server as packets in case of an asynchronous write? Or do we need to create a separate thread in the client responsible for sending the data, making sure the client doesn't have to wait for the packets to be sent over the network? In the first case, I am assuming we are allowed to assume a maximum file size to be written.
[SJ] Former should be done.

Q51) The answer to Q29 of this document said that we could assume that file/dirpaths would be unique. But we can copy files/dirs from one SS to another. So if I copy a directory from SS1 to SS2, and then later try to access a file in that directory, how do i know which server to go to, since there are now 2 copies of that file? Sure i could include info in each path about which server it's located on, but it would render using an optimized search function for files/dirs useless (and we were also told not to do this in the answer to Q2 of this document).
So how are we supposed to deal with this?    
[SJ]
Consider 

_SS1_ running in `root_directory/dir_pbcui`

The accessible paths are

    /dir_bzaca
    /dir_bzaca/file_ofr.txt
    /dir_bzaca/file_qsw.txt
    /dir_bzaca/file_uag.txt
    /dir_bzaca/dir_ccftl/dir_hbsfj
    /dir_qkdez/dir_htcrv/dir_mrrcz
    /dir_qkdez/dir_htcrv/dir_mrrcz/file_imz.txt

_SS2_ running `root_directory/dir_gywnw/dir_fzxpq`

The accesible paths are

    /dir_rzuni/dir_eqfej
    /dir_rzuni/dir_eqfej/file_wux.txt
    /dir_rzuni/dir_aqmhk

A command such as `COPY /dir_qkdez/dir_htcrv/dir_mrrcz/file_imz.txt /dir_rzuni/dir_eqfej` copies the file `file_imz.txt` from SS1 into the `/dir_rzuni/dir_eqfej` directory of SS2. You may change the name to something like `file_imz_copy_<TIMESTAMP>.txt` to ensure unique names (so that your data structure for search doesn't fail).

You can also copy directories in a similar fashion. Copy might also happen within the same SS.
    
Q52) Can we implement logging mechanism by writing everything to a file?
[SJ] Yes as long as we can see logs are being stored and print them on terminal in NS side.
    
Q53. Can we implement the write operation through naming server rathan than storage server
[SJ] No

Q54) Do we need to make everything efficient or only the search that naming server does?? In our implementation other than the search we are using O(n²) approach. Is it fine if only search for ss is efficient and not the others?
[SJ] Where are you using O(n^2) approaches though. As long as searching for accessible paths is as intended to do this should be fine.

Q55) In response to the zip question, if it does infact copy all contents, are we allowed to use zip?
And are we also allowed to use rm, rm -rf command for deleting files and folders (this is where execvp would be used)?
[SJ] You can use it if copy all contents. For folders yes, for files use a system call.

Q56) Can we use a shell (bash) feature to implement logging?
[SJ] Elaborate on what do you mean by this.

Q57) Are we expected to store logs even after program ends?
[SJ] As long as NS keeps working store logs.

Q58) If a file is being read from or written at /abc/123, and DELETE /abc/123 is executed, or while /abc/123 is only being written to, COPY /abc/123 is executed, We decide the reject the command. Will any marks be cut for this? (OR) do you expect us to wait and check for a while and then reject the command for full marks?
[DP] This is fine.

Q59) When say a directory /DIR1 is stored in SS1, backed up in SS2 and SS3. SS2 is down. Do we need to allow write to a NEW file called /DIR1/123.txt Or can we reject the request? If you want us to allow the write then we have to backup the new file to say SS4? (or) we only backup to SS3 and then whenever SS2 comes online, backup to SS2?
[DP] If you can keep track of backups of individual files, backing up to SS4 works. If it's complex, you can inititate the same backup command again when SS2 comes back online, sort of like re-backing up files in SS1 to SS2 and SS3 again, it will overwrite the previous backup and will be fairly straightforward to implement.
    
Q60) Do backup operations have to be immediate? (After list is uploaded or file is written) Or can we simply run a background thread to check and initiate backups
[DP] Yes, because a client can ask for a request at any time, and a storage server can go down at any time. But we have to ensure consistency, hence the bg thread is a good idea.
    
Q61) Suppose the folder /DIR1/ is initially only in SS1. Now a request WRITE /DIR1/abc.txt is sent. Is it ok if we write the new file in SS2? (This helps us generalise everything and handle things on a per file basis).
[SJ] No.
    
Q62) If only /DIR1/ exists and it is empty, a command called WRITE /DIR1/DIR11/abc.txt is sent. Should we send an error or can we recursively generate folders and insert the file anyway.
[SJ] Anyway as you want is fine.

Q63) Is create only for files? or is it also for folders?
[DP] Both.