# File system usage in Windows NT 4.0
study Windows NT 4.0 file system usage.

1. compare with pervious studies on BSD, Sprite OS file system.

2. study the detail of NT 4.0 file system.

3. study the complexity of NT 4.0 file system interaction.

4. Study the overall distribution of the usage data.

## 3. Collecting the data
1) snapshots of the state of the local file systems.
2) all I/O requests sent to the local and remote file systems.
### File system snapshots
trace agent scans the file system trees periodically.
maintain a squence of records containing attributes of each file and directory.
walk record: file name, size, creation, last modify, last access time.
directory: name, number of files, number of subdirectories.
transfer these records to trace collection server, in a compressed format.

it is not enought to trace the syscall only, but need trace the request sent to drivers that implement file system.
a trace driver attached to driver of a local file system to collect each request to file system.
## 5. File system content characteristics

two main difference in local file system, user's profile, application packages Microsoft Office, Adobe Photoshop, Microsoft platfomr SDK...

major change appears when a new user logs onto a system, it will download the profile, or when a new application is installed.
### 6.1 User activity

Compare user acitivity on Windows with BSD and Sprite.

BSD and Sprite, are multi-user systems, while NT 4.0 is single-user system.(?)

A user and system are active when any file system activity in a interval is attributed to the user, since Windoes is designed for single user.

The tracing period divided into into 10 minutes and 10 seconds, 10 minutes represents steady state, while 10 seconds represents bursts of activity.


The average throughput per user has increased threefold since the 1991 Sprite measurements in 10 minutes.

Sprite not has this increasement is due to executables and large file such as fonts are stored in local disks.

Windows does not produce same high peak load, because applications in Windows read small portion of file at a time, and use memory-mapped file to handle large-files.

### 6.2 File access patterns
#### Sequential access is much faster in real hard disk access.
Most file access in BSD and Sprite is sequential(over 80%).

sequential access has two classes: complete file access and partial file access.

Near 80% of access patterns to the file system are read-only accesses, near 20% are write-only accesses, and only about 3% are read/write accesses.

#### Read-only access pattern is the main access pattern in Windows NT.

68% of read-only accesses are whole file accesses, 20% are partial file sequential accesses, 12% are random accesses.

Percentage of read-only access and sequential access of Windows NT is little lower than Sprite.

#### Hard disk performs well in sequential accesses due to the physical distance.

#### Random access pattern will slow down the system, since disk needs to move the head to the target sector every accessess, which is very slow.

#### In read/write access patterns, almost all accesses are random access, which may also affect the system performance.(?)

68% of read-only access in windows is whole-file sequential, while 78% in Sprite.

Windows has more random access than sprite.

### sequential runs
#### A portion of file is read or written in a sequential manner.

#### important for effective caching strategies.

80% of the sequential run is below 11 Kbytes.(Figure1)

#### Most bytes were transferred in the longer sequential runs.(Figure2)


#### Most operations are to short files.
40% of files are shorter than 2 KBytes.

80% of files are shorter than 10 KBytes.

Rest of 20% of files are ranging from 10 KBytes to 100 MBytes.
#### Most bytes are transferred to large files.

#### 20% of files are larger than 4 Mbytes. Mostly executables and dynamic loadable libraries.

### file open times(only with data transfer)
windows: about 75% of the files are open less than 10 milliseconds.

prite: about 75% of the files are open less than quarter-second.

BSD: about 75% of the files are open less than half-second.

The difference between Sprite and BSD is Sprite access files through network to remote storage, while BSD access files in local storage.(?)

In Windows NT, there is no significant difference in access times between local and remote storage.(?)

### Small conclusion

#### Random access pattern will slow down the system, since disk needs to move the head to the target sector every accessess, which is very slow.

#### Most operations are to short files.

Do lots of open file operation.

#### Most bytes are transferred to large files.

Long data bytes to tranferred in each file.

## 6.3 file lifetimes
### (newly create files)
Sprite: 65%~80% of the new files were deleted within 30 seconds.

Windows: 80% of the new files are deleted within 4 seconds.

### three sources for deletion:
#### 1. existing file truncated(37%).
#### 2. a file new created or truncated was deleted(62%).
#### 3. temporary file(1%).

### Delete-through-truncate
75% of the delete-through-truncate case file was overwritten within 4 milliseconds after it was create.

75% of these files are overwritten within 0.7 millisecond of the close.

Top 10% of these files have lifetime at least 1 minute, and up to 18 hours.

#### 75% of files being truncated have very short lifetime, while some files have extremely long lifetime.

### Explicitly deleted file
#### Higher latency between create and delete action.

72% of these files are deleted within 4 seconds after they were created.

60% is deleted within 1.5 seconds after they were closed.

### Possible factors for the different lateny
1. In 94% of the overwrite case, the process that overwrites the file also created it in the first place, while in 36% of the DeleteFile cases the same process deletes the file.

2. No other actions posted to overwritten file, while in 18% of the e DeleteFile cases, the file is opened one or more times between creation and deletion.

### Temporary file
#### Files created to temporarily contain information, and should be deleted at close time.

At least 25%-35% of all the deleted new files could have benefited from the use of this attribute.

23% of the cases where a file was overwritten, unwritten pages were still present in the file cache when the overwrite request arrived.

Only 4% of the deleted files are over 40 Kbytes and 65% of the files are smaller than 100 bytes.

#### No statistical proof for a correlation between size and lifetime of temporary files.

## 7. Data distribution
Most file accesses still occur in a read-only, whole-file sequential manner.

Resource predictions are often made based on the observed mean and variance of resource requests.

#### Incorrect analysis could lead to wrong system design, which could not handle extreme variance in input parameters.

User behavior has very reduced influence on most of the file system operations.

More than 92% of the file accesses were from processes that take no direct user input, even though all the systems were used interactively.

User input is mostly handled by explorer.exe, the graphical user interface, and is not affects user request to file system.

Ex: length of TCP sessions are controlled by process, with limited human factors involved.

#### Be very careful in describing file system characteristics using simple parameters such as average and variance, as they do not accurately describe the process of file system access.
#### Need to consider extreme variances situation.
#### System design needs to be prepared for the heavy-tail characteristics of the access patterns.

## 8. Operational characteristics

### 8.1 Open and close characteristics

In windows NT, file need to be opened, before it can be deleted or loaded to memory.

### interval time period of open file request

40% of the requests arrive with in 1 millisecond of a previous request, while 90% arrives with 30 milliseconds.

Only up to 24% of the 1-second intervals of open requests are recorded by user session.

Between 24% and 40% of the files that are opened read-only are opened multiple times during a user's session.

### reopen files

4% of opened write-only files are opened for another write-only session, while 36%-52% are re-opened for reading.

94% of files that were open for reading and writing are opened multiple times.

#### frequently opening file in short time is the burstiness in the system.

### user open file lifetime

#### Important for resource tuning.

40% of the files are closed within one millisecond after they were opened, while 90% are open less then one second.

Sessions with only control or directory operations 90% closed within 10 milliseconds.

70% of read-write access happens in periods of less then 1 second, while read-only and writeonly accesses have this 1 second mark at 60% and 30%, respectively.

#### It is difficult to predict when a file is opened.

### 8.2 read and write characteristics

70% of file open operations closed again after read/write operation.

80% of read operations, if read was not at the end-of-file, a follow-up read will occur within 90 microseconds.

#### Frequently doing read/write operations is the burst of performance.

80% of write operations have a next write operation less than 30 microseconds.

The difference between read/write is probably related to the fact that process performs some processing after each read, while write is pre-processed before doing write.

59% of read operations request size is either 512 or 4096 bytes, probably related to the buffer size in kernel or standard library.

The size of write operations is diverse in the lower byte range(<1024 bytes), probably related to the data-structure.

### 8.3 Directory & Control operation

74% of the open requests were to perform directory or file control operation.

There are 33 major control operations on files available in Windows NT.

ex1: control operations that used to test whether path, names, volumes and objects are valid. These operations are triggered by Win32 runtime libraries.

ex2: Control operations that used to test volume is mounted.

ex3: SetEndOfFile control operation, used to set file size after write or truncate operations.

#### Main operations performs on opening files, also the burst of performance.

### 8.4 errors

12% of open requests fail.

8% of control operations fail.

52% of open failure is due to the file does not exist.

31% of the file creation fails since the file already exists.

0.2% of the read operations fail.

no write operations failure found.

#### Need to handle operation failure.

## 9. cache manager
cCache manager needs to interact with file system through memory-mapped sections, since cache is at logical address level, while files system is at physical level.
#### Introduce two strategies for optimizing file caching

Process can disable read caching for the file at file open time.

Only 0.2% of all file read/write requests used this option, 76% of these file were opened by system process, and also disabel write file caching.

### 9.1 read-ahead

#### cache manager moves data from file system before the process requests the data.

usually 4096 bytes.

31% of reads completed in noe single IO operation, which means that the data copied to cache by read-ahead only used one time.

#### read-ahead performs well when sequential accesses with multiple reads.

40% of reads used read sizes smaller than 4Kbytes and 92% smaller than 65Kbytes.

#### read-ahead can cover 92% of all read requests.

#### Need better predict strategy, to prevent useless read-ahead data movement.

If process has specified that it will sequential access data only, cache manager doubles the size of the read-ahead requests.

Only 5% of sequential read access specified this option, and 99% of these access size were smaller than 4096 bytes, the option has no effect.

read-ahead is performed when the third of a sequence of sequential requests arrives, and 7% of all request needed data beyond the initial read-ahead.
### 9.2 write-behind

#### Cache manager does not immediately write new data to disk.

When the bursts occur, they are in groups of 2-8 requests, with sizes of one or more pages up to 65 Kbytes.

1.4% of file with write operation disabled caching.

4% of enabled caching used the flush request explicitly, and 87% of these applications flushed after each write operation, may better disable the cache directly.

#### Need to be careful about cache data lost when power failed.