# CS374 Design Project | DP 5 : Usability Testing | Report
---
### Project : Build the train mobile ticketing service for elderly people.
**Team Name : EasyKORAIL:station::ticket:**
**Team Members : Hyunseo Lee, Jinwoo Lee, Daekeon Song, Youngmin Yang**

---

## POV

**We met** seniors who tried to book a train and board it using the remote ticketing service.
**We were amazed** to realize that they want a train ticketing process that help them book and board on a train without any confusion and inconvenience that they can get used to right away when they see it for the first time.
**It would be game-changing** if seniors overcome their fear of using unfamiliar technologies in preparing for train travel.

## Tasks
Users will be assigned two main tasks at the time of payment for tickets, when the first task of setting up a ticket is done. One is the task of requesting payment from another person, other is the task of confirming ticket reservation by paying by oneself.

The Core Task is divided into three categories, but we expect that the small tasks in the process of doing this task will give us a deeper information on how the user uses our prototype.

>Task 1. Reserve a train ticket for you. You can freely choose your departure point, destination, time, and number of people.
>Task 2. Ask an acquaintance to buy a ticket instead of using your own payment method.
>Task 3. Revoke payment request and make payment directly, Check your ticket after payment.

## Updates from DP4
* **Figma -> Web Prototype**
* The existing prototype was made of Figma, so there was a disadvantage that free input was impossible. This causes a mismatch between the user's input and the actual output, which can cause confusion for our target users, seniors. Therefore, we converted the existing prototype into an HTML-based prototype and added several features that were not possible in Figma.
* **Revision in Tasks**
* As the prototype changed to the web, the tasks also changed as more functions could be performed
* Remove Task 1 : "Using your phone number, log in to the application." is a minor task of our prototype, Not core task. so we remove this task
* Modify Task 2 and set to Task 1 : "Book a train from Daejeon to Seoul where you will ride alone" -> "Book a train ticket for you, You can freely choose your departure point, destination, time and number of people. This task can be repeated over and over again".
* move Task 4 to Task 2
* Add new Task 3 : By canceling the payment request and encouraging user to pay directly, user can complete the ticket reservation process and let them know that there are two ways to pay
* **Introduce new features**
* Users can now view the appropriate data for all possible options (dates, seats etc.) with the web prototype.
* Users are free to check or delete the reserved tickets. Also, they can send or revoke payment request when they want.

## Written protocol

### Instructions for preparation and setting up the testing environment
**(1) Tutorial : [Figma Tutorial](https://www.figma.com/proto/bs3AAaRDdABMy3ambEnkys/Tutorial?node-id=36-360&t=dc3ThmsSjg5SOos5-0&scaling=min-zoom&page-id=0%3A1&starting-point-node-id=36%3A360)**
>This is link for tutorial our prototype
>
>1. Open the link above. (You may also use Figma application.)
>2. Click "present" button on the top right.
>3. Check the quick tutorial about reservation progress.

**(2) Web : [Web Prototype](https://youngmin-yang1.github.io/EasyKORAIL-prototype/)** **(Nightly build version is attached as it required consistent design iterations. It may be different from the one that we actually used in the DP5 usability testing, however, you may also use this version with the same written protocol.)**

>1. **Open the link above using the Internet accessible Android smartphone. If you are not accessible whith Android smartphone, choose between two options below.**
- Additional option 1 : Install **Chrome / Firefox browser in iOS App Store and activate the "immersive mode"** before conducting the test. (Otherwise, due to the resolution problem, some of the animation might not be seen as we desired.)
- Additional option 2 : Use **latest verison of Chrome browser in Mac OS or Windows** and open the link. (If you use any extension in browser, please unable all of them temporarily for the best experience. Extensions and web prototypes might cause conflicts.)
(Please note that our protoype models smartphone application specifically designed for senior users. Desktop environments can be considered when mobile environments are not available. Also, as our prototype requires typing some dummy informations such as card number or phone number, webbrowser might warn our webpage as scam. You can simply neglect such warning and notify it to the user before the test.)
>2. **Start testing from the main page. Log in first, and then follow Task 1, 2, and 3 in order.**
>3. **While conducting the experiment, there are some required inputs for the web prototype. Use the information below.**(This number changes depending on the specific situation of the actual program, however, we hard-coded them due to the absence of appropriate client server.)
>- **Verification code : 123456**
>- **Card CVV : 1234**

### Introduction and informed consent
**Introduction**

Introduction about our team and testing
- "Hello, we are EasyKORAIL team, taking CS374 course at KAIST.
While doing our research, we found that many senior want to book train tickets by using their phone and board the train. However, we got to know that many seniors felt inconvenience in mobile train booking system. This is primiraly attributed to their relatively low exposure to IT-technology. Also, as many existing programs aim to be versatile, they don't reflect the characteristics of seniors.
So in order to tackle the inconvenience of the seniors, we designed a program that allows seniors to comfortably book train tickets and board the train.
Today, we will introduce a prototype and carry out the usability testing. Before the prototype, you will be given a simple tutorial to get familiar with it. Then, you can test our prototype with our explanations on the tasks. Moreover, there will be relevant interview after the test. Your participant and kind feedback will be invaluable assets when improving our prototype for future users.

**Participants' consent**

- "To facilitate the development of our prototype, we would like to record your comments and feedback using the voice recording device installed here. Would you be comfortable with us recording your voice for the purpose of future design iterations? If not, we can alternatively document the process through handwritten notes. Please let us know your preference."

### Tutorial or training

We prepared a simple tutorial (best experience in smartphone web or application) so that users can quickly adapt to our prototype. **The content of the tutorial is the 'reservation progress' with progress bar**, which is one of the most central part of our prototype. **User can move to various steps by using the progress bar captioned with "reservation process"** on this page. Complex functions in the actual prototype were excluded, and **only the function of navigating through each step using the progress bar and simple click is remaining.**

- "Our prototype is specifically designed for seniors to quickly adapt to the process of booking and boarding the train. However,as our prototype is slightly different from application you may have used before, for the best experience, we prepared the quick tutorial."

**[Tutorial : Reservation progress](https://www.figma.com/proto/bs3AAaRDdABMy3ambEnkys/Tutorial?node-id=36-360&t=dc3ThmsSjg5SOos5-0&scaling=min-zoom&page-id=0%3A1&starting-point-node-id=36%3A360)**
> 1. Try to open "Reservation progress" (Drag down with your finger!)
> 2. Feel free to navigate through different stages by clicking "edit" button of each stage.

### Task list & instruction
Here is the prototype webpage shown in the prepared device. Necessary settings have been configured by the facilitator, and you may begin testing by simply logging into the service first."

>Task 1. **Reserve a random train ticket for you. You can freely choose your departure point, destination, time, and number of people.**(Child-only ticket can not be issued since currently it is prohibited by law.)
- "Your mission is to set the ticket information according to the procedure. Please proceed until payment is made. Feel free to choose your departure point, destination, time, number of people, and **book (not purchasing but just reserving) a train ticket.**"
>Task 2. **Ask one of your acquaintance to buy a ticket instead of using your own payment method. It is called "payment request."**
- "Next, your second task is asking an acquaintance to buy a ticket instead of your own payment method. You can choose any ticket you reserved."
> Task 3. **Revoke "payment request" and make payment directly**, check your ticket after payment.
- "Finally, your task is buy a ticket yourself. Revoke the request for the ticket you made in task2 and pay with your card. Please **use CVV number : 1234 for the payment.** Other values can be inserted randomly."

### Recording strategy
For seniors, using the record in seniors' smartphone such as the screen recording might be difficult. Many seniors we met before "worried about being confused or making mistakes" during the test. Thus, during the session, we will record our observations and findings with voice recording feature in our smartphone. Also, our processes and intermediate findings will be recorded as hand-written notes and sketches.
About our members' role, since seniors needed familiar member for the test, each of us will play a role of a facilitator (both for the test and interviews) for the test session of the target user they contacted, and accordingly, other personnel will act as observers to fully record the test and interview process.

### Questionnaires, Interview questions
After the usability test session, we gathered qualitative comments about the task experience from our particiapnts. We prepared total 15 questions in advance. They are categorized by themes. However, for each participant, we also asked some addiditonal questions based on individual response and action.

**General Questions**

- Remote Ticketing Platform Experience:

- Have you utilized a remote ticketing platform for any existing service (e.g., transportation, restaurant, barbershop)?
- If yes to the first question, what were the main challenges that you encountered?
- From that perspective, how is our prototype different (in UI design, simplicity, accessibility, learnability, functionality, etc.)?
- Log-in Process Evaluation:

- Was there any difficulty in the log-in process?
- Compared to your previous experience with log-in processes on any existing service, which do you prefer?
- If you have a particular preference and comment on the log-in process, please give us any advice.
- Design and Text Recognition:

- Were the design features (e.g., button, bar, region) and texts easily recognized? If you experienced inconvenience, please let us know.
- Station Selection Experience:

- When choosing the departure/arrival station, was our lexicographic-order list convenient enough?
- Understanding Progress Bar and Tutorial:

- Did you understand the existence and the functionality of the progress bar after watching the tutorial?
- If so, was traversing between each step and making modifications at certain steps easy enough?
- Reservation Process Challenges:

- If you had a particularly struggled moment during the reservation process, what was it? Could you explain how you felt during that moment?
- Payment Request Understanding:

- Did you understand the concept of "payment request" right away?
- Could you easily notice when the request has been sent?
- Revoke Payment-Request Process:

- To revoke the payment-requested ticket, did you easily find the right place to click?
- Payment Interaction Evaluation:

- When making the payment, did the interactive card animation help you follow the progress in typing the necessary information?

### Debrief prompt
After the interview session, we prepared a debrief to thank the participants and organize the overall process. In specific, debrief prompt is as following.

* This concludes the usability test and interview on our working prototype of 'Easy Train for Seniors' in the KAIST CS374: Introduction to Human-Computer Interaction course. Our aim is to build a senior-friendly remote rail ticketing service that helps seniors use the service easily, regardless of their prior IT knowledge.
* We have conducted DP 5: Usability Testing with target users, consisting of multiple seniors who attempted to book a train using the remote ticketing service. Through usability testing on the working web prototype, we gathered insights and thought processes from participants' responses. We also asked some qualitative questions to obtain more concrete feedback about their experience.
* Your invaluable responses and feedback will be aggregated to improve our prototype's user experience. As we notified you at the beginning of the test, your responses will only be used for design improvement purposes, and raw data will be discarded at the end of the design project. Additionally, when presenting the results, we will ensure your anonymity. We sincerely thank you for your participation; it will greatly help us in further developing the project.

## Session observations
>Task 1. Reserve a train ticket for you. You can freely choose your departure point, destination, time, and number of people.
>Task 2. Ask an acquaintance to buy a ticket instead of using your own payment method.
>Task 3. Revoke payment request and make payment directly, Check your ticket after payment.
- P1

> About P1 : 8 minutes (for test)
>
> The interview was conducted in person. We provided the test set up, and talk about a task and receive response.
>
> She is 76 years old and has trouble using smartphones other than KakaoTalk and YouTube. Therefore, she has never booked trains online. However, she usually enjoys taking the train to travel once a month.
>
> Task 1
> - When choosing station, she didn't notice that it was a lexiographic order. So she spent a lot of time to search "Daejeon"
> => The English station name may be the reason, but more important reason is she didn't notice that it was a lexiographic order. So it is better to indicates how the alignment is done.
>
> Task 2
> - Back on the main screen, she was confused about which to press
> => Make difference in button after reservation
>
> Task 3
> - It took a lot of time to input the number in payment stage.
> - There didn't seem to be anything particularly difficult for her.
>
> Total : She didn't notice there are "Reservation progress".
>
- P2

> About P2 : 9 minutes (for test)
>
> The interview was conducted through face-to-face. We introduced her our prototype, and talk about a task and receive response.
>
> She is 70 years old and has some access to smartphones. She sometimes makes payments through a banking app. However, the cycle of train travel is as small as one or two times a year.
>
> Task 1
> - When choosing the seat, she doesn't recognize there were 1 seniors already, so she reserved ticket for two people. Also, ticket confirmation is not intuitive.
>
> Task 2
> - payment request system was unfamiliar task for her.
>
> Task 3
> - When she check her ticket after payment, she didn't know the existence of easy access at the bottom of main page.
>
> Total: Some instructions and words are unfamiliar and non-intuitive.
- P3

> About P3: 6 minutes (for test)
>
> The interview was conducted in person. We provided the test set up, and talk about a task and receive response.
>
> He is 66 years old and mainly uses his smartphone to take pictures. He has to take the train about once a month for business reasons, and he wants to book tickets in advance, but he has never booked a ticket online.
>
> Task 1
> - He quickly completed each process and moved on, but did not notice the reservation progress bar above. Also, when he went to the home screen directly after booking a ticket, he wondered if it was properly booked.
>
> Task 2
> - He tried to find a button to pay directly instead of a request.
> => Payment requests should be made more natural by seniors.
>
> Task 3
> - He completed the ticket reservation after quickly entering only the card information without looking for other functions.
>
> Total: He followed the overall flow well but did not use many useful features.

## Usability lessons
After the test and interview, we found 10 usability issues. Please note that some insights in them are obtained from the interview after the test.
* Seniors didn't find the progress bar. (High critical) (P1, P3)
* Most seniors didn't notice the reservation progress bar while booking, and some didn't think it was a button.Since this greatly affects the user's convenience, it is necessary to change the design of the progress bar or add simple help.
* Seniors can't confirm that the ticket was booked normally. (High critical) (P1, P2)
* This problem occurred when the user entered all ticket information, pressed the reserve button, and immediately went to the home screen. It should be corrected so that errors can be determined by adding a completion mark immediately after booking.
* The seniors spent a lot of time checking the tickets they had booked. (High critical) (P1)
* The Ticket confirmation button is a phrase that is weakly intuitive for seniors. We need to increase intuition with expressions such as "booked tickets", and add a simple tutorial to the home screen.
* Seniors could not immediately send a payment request for a reserved ticket. (Mideum critical) (P2, P3)
* It is recommended that the user's options be as wide as possible. We need to provide a variety of options by correcting the transition to the home screen immediately after booking the ticket.
* Seniors had a hard time finding the name of the station on the train station ticket. (Mideum critical) (P1)
* The list of train stations is arranged as lexicographical order, but it is difficult to find a specific station due to its vast volume. Therefore, it is necessary to increase visibility by adding a header for each alphabet.
* Seniors tried to choose another element without canceling the one they had already chosen. (Mideum critical) (P1, P2, P3)
* This prototypes could be released by clicking on the previously selected content again for safety, and then selecting other elements. However, many seniors have experienced inconvenience with these functions, and we will adjust them to prioritize usability.
* The seniors did not know whose contact information the send request belonged to. (Mideum critical) (P2)
* In the send request function, a phone number can be selected from a contact, but it was not known whose phone number belonged to. We need to add a name under the phone number so that it doesn't depend on the user's memory.
* Seniors were not good at reading letters from dark backgrounds. (Low critical) (P2)
* All dark backgrounds are assigned to objects that cannot be clicked, so there is no effect on the actual functionality. However, this has a bad effect on the user's sense of use and needs to correct the color of the handwriting.
* Some seniors asked which way the train cars were headed. (Low critical) (P1)
* When selecting a seat, it is necessary to add information such as direction or size.
* The initial number of people is set in advance can cause confusion. (Low critical) (P2, P3)
* Choosing the initial number of people has its own pros and cons. It will get more opinions and take direction.

### Studio Feedbacks & Reflections Summary
> The core ideas addressed to organize high-level topics in usability Lesson are as follows. It was composed mainly of problems with a high critical level.

- **Visibility and Intuitiveness - The functions required by the user should be visualized.**
- **Information Organization - Listed lists should be organized systematically.**
- **Accessibility and Readability - Ensure that the user does not experience any inconvenience in using the prototype for accessibility or visual reasons.**

## Studio Reflections
After the DP5 presentation, we have received several feedbacks from our students. Here, we list invaluable feedbakcs and how we tried to use them to improve our project in the upcoming "design iteration".

### Presentation (Written Protocol)

- I wish you would mention the case when users don’t agree with the recording. (if you don’t agree with that, we will stop the user test.. like this)
- Even though we briefly mentioned the contents of the interview in advance of the appointment, such concern seems probable, especially for our target users. Reflecting your feedback, we slightly modified the "informed consent" and "recording strategy" sections. Rather than simply aborting the scheduled usability testing and interview if the interviewee does not agree to voice recording, we chose to use handwritten memos about the interview contents. Since this won't significantly affect the overall process of the interview, we applied the same criteria for both voice recording and memo.

- What if you instruct the seniors to record their screen? Screen recording may be helpful in many aspects, so you may at least try so.
- Many seniors we met "worried about being confused or making mistakes" during our previous interviews and usability testing. Since screen recording can still track their behaviors while using our prototype, we chose to record the interesting moments (which can only be captured by human observation rather than voice recordings) with handwritten memos from observers.

- What if debrief prompt is simpler since the target users are seniors.
- We understand that seniors may be feel confusing when they are given with long debrief, however, we spent this time for organizng the process, notifying the relevant informations (e.g. privacy issue) and thank to them.

- I wonder how the task logging in is related to your POV and solution.
- We have updated the tasks to align with our POV and solution. We also agree that the log-in feature may not differ greatly from existing applications, so we focused on more necessary daily behaviors for the task.

### Studio Session (Pilot Session)
- "Make it possible to close the train station selection screen, open contact screen."
- The lack of a backward function when accidentally clicking on the train station entry compartment causes considerable inconvenience. We will add backward function to the train station list.
- "In the person selection section, there is 1 senior already checked at the beginning. This make me confuse."
- This is based on the minimum number of reservations for convenience. However, we have confirmed that the confusion is greater than the convenience, and the prototype will be modified.
- "I didn't recognize there is a progress bar."
- As we mentioned earlier, we will inform you of how to use the progress bar through a simple tutorial.
- "I had a hard time making a final reservation on the train information screen."
- It seems that the problem was that the reservation button at the bottom right is not written as "reserve", but "next". Therefore, we will modify this to "reserve".
- "I've got some problem about where to make a payment request."
- We will add a tutorial or a description to tell user where the payment request is.

### Studio Feedbacks & Reflections Summary
> To organize the feedbacks in a high-level theme, core ideas that were addressed are given as follows. We reflected invaluable feedbacks on our report and our future design.

- **Written Protocol - Concerns about the edge case for informed consent.**
- **Written Protocol - Concerns related to the characteristics of our target user group (senior passengers).**
- **Pilot Session - Issues with the design choice of the prototype (e.g. consistency, visual cue)**
- **Pilot Session - Lack of understanding about the concept of payment request before we are explaining them.**
- **Based on these feedbacks, even though there was not a criticial issue about both the protocol and prototype, we reflected them on "design iteration" and also modified taks and written protocol.**

## Plan for iteration
- Need to specify the option of 'Payment / Payment Request', increase feasibility.
> Make seniors to recognize that it can interacts with a simple animation(bounce of apple pay/face id) during the payment process.
- The station is shown in the lexiographic order, so tag the station with capital letter(A/B/../Z) to make it easy to distinguish.
> It would be better for the seniors to list stations in the order they are more frequently encountered and familiar with in reality. Such tag will help them easily find the station.
- Make the sold-out ticket more recognizable.
> Leave an orange sign next to ticket that is close to soldout.
- Show simple description about Payment Request function.
> As mentioned in the studio session, seniors can easily understand what the payment request is.
- Show the Payment Expiration time when user reserves the ticket. (ex. "payment Expiration time : 15m30s")
> After Reservation, there will be a payment expiration time, and seniors may not notice it. Therefore, in order to clearly display this, the color will be changed and displayed as time decreases.
- Add the home button in the reservation progress bar.
> For seniors, it may be not intuitive to go back to the station page and press the back button to return to the main screen.
- Add a badge that tells user the number of tickets with short payment time remaining. (less than 5 minutes)
> This function also needs for seniors to notice a payment expiration time.
- Add an alert message to remind that the remaining payment time for user's current ticket is imminent. (less than 1 minute)
> This function also needs for seniors to notice a payment expiration time.
- Show train direction when seat is selected.
> It is intended to make design more realistic and prevent the possible errors.
- Add QR code and platform map that can be checked in the train.
> For seniors, the location and congestion of toilets and exits in the train are important. Therefore, it is a convenience function to make this known. Also, quick access to QR code of the ticket would help them when they should show their ticket to crews.
- Add an animation when user completed a reservation, payment request, and payment.
> Whenever seniors preform a particular task, such as a reservation, payment request, or payment, informing it that the action worked correctly helps seniors become confident in their action.
- Add "assistance request" which allows seniors to request personalized help by simply tagging their phone on the NFC at the assistance hub installed in the station.
> There is also the inconvenience of train stations that seniors are afraid of using trains. It is a function that can be easily solved the inconvenience.