Santoku Knife Store
Izaz Zubayer | Nishant Gadihoke | Leona Mugova | Gurmastak Singh | Salil Agnihotri
Design Evaluation
April 17, 2023
Project Overview
Introduction
This study aims to evaluate the Santoku Knife Store website to improve its user-friendliness, efficiency, and ease of navigation. The study will help the client understand the website's design evaluation and identify areas for improvement to motivate customers to purchase products or sign up for services.
Goal
Heuristic evaluation
Think-aloud study (with 10 participants)
System Usability Scale (SUS) (with 10 participants)
Conceptual model extraction (with 5 participants)
My Role
Research
Interface Design
Interaction & Prototype
Evaluation
Challenges
Ensuring independent and unbiased evaluation
Identifying usability issues based on ten crucial Heuristic usability principles.
Analyzing multiple usability issues within a short time frame with better efficiency.
Addressing the multiple usability issues faced by all visitors
Introduction
1.1 About the client
In the evaluation study project, we coordinated with ‘Santoku’, a Japanese knife store located in Vancouver. The store is known for offering special services such as knife sharpening workshops, blade repairs and custom-made knife services. In addition, the client reflects the touch of Japanese traditional craftsmanship in the modern cooking industry. As per the client, most of their emptors visit their website to obtain information on the products and services offered related to knife craftsmanship and repairs.
1.2 Client needs and expectations
The client needs the following from our study plan:
The website should persuade customers to buy its products or engage with its services.
The website should be efficient, user friendly and highly lucid.
1.4 Goals for evaluation
The evaluation goals revolve around the following three crucial domains:
Understand how users interact with the website
Understand the main frustrations/issues users face while using the site
Analyze the current interface - Does it satisfy the different attributes of usability?
1.3 Study context
The study is focused upon the evaluation of the ‘Santoku Knife Store’ website, which provides most of the crucial information on the types of knives offered and services available, therefore the client would like their website to be more user-friendly, efficient and easy to navigate. In fact, the client would also like their website to motivate the customers to either buy their products or sign up for their services. The study will be helpful for clients in understanding the design evaluation of their website which will further emphasize the need for improvements in the current design interface.
Study Methods
2.1 Evaluation Method
The following study methods were chosen for this evaluation:
Heuristic evaluation
Think-aloud study (with 10 participants)
System Usability Scale (SUS) (with 10 participants)
Conceptual model extraction (with 5 participants)
2.1.1 Heuristic evaluation
About heuristic evaluation
The Heuristic evaluation method is used to detect usability problems in interface design based on ten usability heuristics guidelines. In the process, the interface design is evaluated by a group of individuals to understand and analyze the usability problems, which are further used to improve the most pertinent usability issues.
Reasons for using heuristic evaluation
We employed Heuristic evaluation for our study plan due to following reasons:
Independent and unbiased evaluation
Identification of usability issues based on ten crucial Heuristic usability principles.
Low level of complexity.
Analyzing multiple usability issues within a short time frame with better efficiency.
Our client’s website has multiple usability issues that are faced by all visitors.
Participant pool
We assigned three evaluators from our team to be experts who were able to successfully identify usability issues in different sections of the website. The evaluators had prior experience in UX/UI design and a core understanding of Nielsen’s usability heuristics.
Study location
Online
Using laptop/desktop devices
Constructs/Research Questions and Data Collection
The evaluators were given a 30-minute time frame to scrutinize usability issues based on ten of Neilson’s usability heuristics. The evaluators observe the issue and categorize the issue based on the heuristic violated (From Neilsen Norman Group). Furthermore, the evaluators were also asked to write brief descriptions of issues and why they disliked the aspect. Then the severity of the issue was marked on a scale of 1 to 5 where 1 denotes low-level severity (low complexity issue) and 5 denotes high-level severity (high complexity issue, requires immediate fix or modification).
After issues were identified, they were categorized into the following:
Heuristic violated (From Nielsen Norman Group),
Description of issue
Severity of the issue (from 1 to 5)
Reliability & Validity
In order to ensure reliability and validity in our study we followed the following steps
In the heuristic evaluation study, we ensured that the number of evaluators is more than one. This is a crucial step as it establishes independent and unbiased opinions and observations.
The evaluators were given an abundant time frame (about half an hour) to find usability issues, this ensured that evaluators felt minimum pressure and anxiety during the task.
In a half an hour time frame, ten minutes were provided to evaluators for getting used to
the interface and navigation.
In order to enhance reliability we addressed controlled variables and confound variables
as mentioned in Appendix 5.
To ensure validity we reduced novelty and enhanced learning.
Results
Multiple issues were found, with some of the most pertinent being:
Visibility of system status violation with lack of context on the navigation bar
User control and freedom violation by not allowing customers to complete purchases via
the online store.
Recognition rather than recall, violated by making users remember where certain
navigation options are rather than showing them upfront.
2.1.2 Think-aloud study
About think-aloud study
In the think-aloud study, the participants are given a specific task and the evaluators record their thoughts which they speak about while performing the task.
Reasons for choosing Think-aloud study
The think-aloud method delivers real-time user insights and unfiltered feedback.
The method helps in promoting a deeper understanding of user behavior and needs.
It is a cost-effective and efficient testing technique.
It facilitates user-centric design improvements.
It is simple to understand and perform therefore it is highly used in industry.
Participant pool
Professional chefs
Restaurant staff (e.g., kitchen staff, managers, servers)
10 individuals aged 22 to 29, interested in purchasing knives
Study location
Professional kitchens: Approach chefs and restaurant staff during their breaks or after
their shifts to recruiting participants.
Restaurants: Connect with restaurant management to coordinate recruitment and
participation sessions in a suitable space within the establishment.
Constructs/Research Questions and Data Collection
The group of 10 participants took part in the think-aloud study and were further asked to cooperate in the System Usability Scale (SUS) test. The participants were selected from a pool of professional kitchens and restaurants who have an acquaintance with using and buying knives. The participants were given tasks in three time slots, in the first time slot (4 minutes) participants were asked to find the price of a 165mm Deba knife called ‘Kurouchi Ryo-Deba -Warikomi-’, whereas in the second time slot (4 minutes) participants were asked to search for repair sharpening service cost of 151-210 mm Boning Style knife. In the third time slot (2 minutes) participants were asked to make an e-purchase of a ‘Kurouchi Ryo-Deba -Warikomi-’ knife. In addition, participants were asked to explain the step they take while performing the task, and their comments, thoughts, impressions of the user interface, and navigation or difficulty observed while performing the task were minutely recorded. Moreover, immediately after the task was over the participants were asked to answer Post-Session Questionnaires and fill the System Usability Scale rating list (10 minutes approx, based on participants' comfort and constraints on shift-timing) as described in section 2.1.3. The results and observations obtained from the think-aloud study and System Usability Scale were helpful in understanding the user perspective of website usability.
Data collection and analysis methods
Think-Aloud Protocol: Participants shared thoughts while using the website.
Post-Session Questionnaires: Gathered feedback and ratings using SUS.
Data Analysis: Identified usability issues and patterns from collected data.
Post-Session Questionnaires
In the study we used 15 Post-Session Questionnaires as depicted in Appendix 2, the questions were asked related to their experience on the website, its design interface, color theme, navigation, usability, their average rating on specific domains, and purchasing experience. In addition, we also asked for some recommendations on which participants would love the website to add or upgrade.
Issues faced
Participant Hesitation: Discomfort in sharing thoughts affected data quality.
Observer Effect: Awareness of observation may have led to biased results.
Time Constraints: Limited session time restricted depth of insights.
Incomplete Data: Some participants didn't explore all website aspects.
Reliability & Validity
In order to ensure reliability and validity in our study we followed the following steps
The participants were provided guidelines and help during the task.
In order to operate the webpage, participants were also provided additional 2 minutes if
they needed to explore the website and learn about functions.
The task was performed in a minimum time frame to avoid fatigue and low
concentration.
In order to enhance reliability we addressed controlled variables and confounded variables
as mentioned in Appendix 5.
To ensure validity we reduced novelty and enhanced learning.
Results > Areas for Improvement
Incorporate a full-size page with a larger font size and zero margin/padding
Reconstruct the page theme for a more dynamic, minimalistic, and aesthetic design
Implement an efficient operational e-store
Improve font size and color theme
Add customer reviews and feedback section
Fix broken links and spelling errors
Incorporate alternate color themes for color-blind users
Adjust or improve navigation tab placement/content and improve the footer section.
Maintain a traditional connection with product details
Enhance newsletter incorporation and provide emails about events and new launches.
Satisfactory Aspects
The simple and familiar navigation
Minimal need for memorizing website content
Sufficient and informative content
Overall average satisfactory website experience
2.1.3 System Usability Scale (SUS)
About System Usability Scale
The SUS scale is used to find the usability based on 10 questionnaires, where participants fill questionnaires from a scale of 1 (strongly disagree) to 5 (strongly agree) based on their experience of using any interface. A SUS scale above 68 is considered above average score while a score below 68 represents poor usability and design of the interface. In the project, we performed SUS with the think-aloud study.
Reasons for choosing the SUS method
The SUS method delivers real-time user insights and unfiltered feedback.
The method helps in promoting a deeper understanding of user behavior and needs.
It is a cost-effective and efficient testing technique.
It facilitates user-centric design improvements.
It is simple to understand and easy to use.
It can be used for small groups of participants.
Highly valid and reliable.
Participant pool:
Professional chefs
Restaurant staff (e.g., kitchen staff, managers, servers)
10 individuals aged 22 to 29 interested in purchasing knives
Study location
Professional kitchens: Approach chefs and restaurant staff during their breaks or after their shifts to recruiting participants.
Restaurants: Connect with restaurant management to coordinate recruitment and participation sessions in a suitable space within the establishment.
Constructs/Research Questions and Data Collection
Steps we used to calculate System Usability Scale:
The participants were given 10 minutes to analyze and scroll across the website to find issues.
A 10-item Likert scale known as the SUS scale is given to participants immediately after a think-aloud task.
Participants can choose between 1 (Strongly Disagree) to 5 (Strongly Agree), with 3 (as mid-range or neutral agreement-disagreement)
As shown in figure (1), items 1, 3, 5, 7, and 9 (marked with green) scores are calculated by subtracting 1 from the scale position (between 1 to 5) whereas, for items 2, 4, 6, 8, 10 (marked with red) scores are calculated by subtracting scale position (between 1 to 5) from 5.
Item 1, 3, 5, 7, 9 (Marked with Green): Scale Position-1
Item 2, 4, 6, 8, 10 (Marked with Red): 5-Scale Position
Figure 1: SUS scale representing odd questions with green and even questions with red.
**Total score is calculated by adding the obtained points, through total score SUS score is calculated by multiplying the total score by 2.5
Data collection and analysis methods:
The data is collected from 10 participants labelled as participants A, B, C, D, E, F, G, H,
I, and J as represented in Appendix 1.
The analysis methods used are mean and standard deviation.
SUS Questionnaires
There are ten post-questionnaires marked from 1 (strongly disagree) to 5 (strongly agree) as represented in Appendix 2.
Reliability & Validity
It was performed in at least 2 minutes to avoid fatigue.
In order to enhance reliability we addressed controlled variables and confound variables
as mentioned in Appendix 5.
To ensure validity we reduced novelty and enhanced learning.
Results
As observed from Figure 2, the SUS score of the majority (about 70 per cent) is above 68 which means that most of the participants felt that the website was above average, although if we glance at the average score of participants (average score: 70), it depicts that website is neither excellent nor worst in fact it needs a lot of improvements as mentioned in think aloud study.
Figure 2: Graph representing SUS score of 10 participants
The graph in Figure 3 illustrates 10 SUS questionnaires, in which the average rating is represented for each question with their specific standard deviations as error labels. The crucial result from the question signifies the following:
Question 1: (Average rating: 2.9) A average scale was below the neutral scale level (SUS rating 3), which indicates that participants will need motivation and improvement in the website to increase their frequency of using the website.
Question 2: (Average rating: 1.5) The participants did not find the system highly complex.
Question 3: (Average rating: 4.3) The participants felt that system was easy to use. Question 4: (Average rating: 2.2) The participants would be independent in using and navigating across the website, which means that participants will not require much help from any technical person.
Question 4: (Average rating: 2.2) The participants would be independent in using and navigating across the website, which means that participants will not require much help from any technical person.
Question 5: (Average rating: 2) As the average rating was below the neutral SUS level it means that participants observed that various functions were not well integrated. The question may indicate non-functional online store service and improvement in the navigation panel.]
Question 6: (Average rating: 3.7) The participants highly agree that there was too much inconsistency in the website and it needed improvement. The results are also interconnected with post-interview questionnaires which indicated that most of the participants felt improvement in theme, alignment and colour.
Question 7: (Average rating: 4.6) The participants felt that the system was easy to use and most people will learn it easily.
Question 8: (Average rating: 1.4) The system is very cumbersome.
Question 9: (Average rating: 4.5) The participants felt that they were confident using the website(system).
Question 10: (Average rating: 1.5) The participants were not required to learn many things before using the website.
Figure 3: The graph representing the average level of agreement and disagreement in 10 SUS questions among participants
2.1.4 Conceptual Model Extraction
About Conceptual Model Extraction
This study aims to evaluate the usability of a small business knife website that is poorly organized, has no navigation, is cluttered, and is confusing. The goal is to extract a conceptual model of the website and provide recommendations for improvements. The conceptual model extraction of raw data for each participant is represented in Appendix 1.
Aim
Assess the usability of a small business knife website with various issues
Focus on problems like ambiguity, content organization, and functionality.
Extract a conceptual model to provide detailed recommendations for improvements
Reasons for using conceptual model extraction
Easy to use and understand.
The model also recommends customer solutions.
Participants pool and methods
Participants: 5 chefs/cooks with diverse experience in online shopping for knives
Think-Aloud Protocol: Participants verbalize thoughts while using the website
Post-Session Questionnaires: Collected feedback, ratings, and suggestions
Data Analysis: Identified usability issues, patterns, and preferences to inform recommendations
Reliability and validity
In order to enhance reliability we addressed controlled variables and confound variables.
For our Heuristic Evaluation, Think Aloud Study & System Usability Scale: our
controlled variable was the use of a mouse, and the confounding variables were hand-eye coordination, concentration level, energy level, size of hand, vision variation, skills variations
To ensure validity we reduced novelty and enhanced learning.
Constructs/Research Questions and Data Collection
To perform this study, we took note of all 10 of Nielson’s Heuristics. We then went through the website while taking note of the heuristics violated, and the severity of each violation.
The following data were collected:
Interface Design: Layout, navigation, visual elements
Navigation & Information: Menu structure, categorization, filters
Color Theme & Accessibility: Consistency, contrast, color-blind support
Broken & Irrelevant Links: Navigation issues, user confusion
Visual Design: Image quality, typography, overall appeal
Online Store Functionality: Checkout process, payment errors
Product Information: Details, materials, dimensions, care instructions
Resources for Professional Chefs/Cooks: Dedicated sections, articles, videos
Mobile Experience: Responsiveness, usability on smartphones and tablets
Results >Recommendations
Simplify the interface and layout to create a more user-friendly experience
Improve navigation and filtering options to facilitate product search and browsing
Enhance visual design with high-quality images, modern typography, and a visually appealing layout by using hierarchy.
Fix online store functionality, ensuring a secure and seamless payment process.
Offer resources, guidance, and customer support tailored to different user needs.
Make it responsive. Optimize the website for mobile devices, ensuring a consistent
experience across platforms
Provide clear documentation - such as return and shipping policies for transparency and
ease of understanding Conclusion
The usability study provided valuable insights into the website's strengths and weaknesses
Implementing the detailed recommendations will create a more user-friendly and visually appealing website
Enhancements cater to the needs of various users, from first-time buyers to experienced chefs, improving overall customer satisfaction and promoting business growth
Discussion and Conclusions
The following implications can be drawn from the results of our study:
The results from the heuristic evaluation signify that users felt a lack of context in the
navigation bar, which further leads to remembering sections from the navigation bar. In addition, there was also a restriction in making online purchases. The results are analyzed based on expert point of view who have prior experience with UX/UI design.
The first crucial results of the SUS study implies that most participants observed that there was a significant number of inconsistencies present on the website, i.e. interconnected with think aloud study, participants felt that the website should be implemented in full page with aesthetic and minimalist design. Moreover, the website needs improved navigation features with improved themes. The second result of the SUS study indicated that all the functions of the website are not well integrated which can be understood from think-aloud comments and questions, i.e. participants observed broken links, spelling errors, incorporation of customer feedback and newsletters section. The results are analyzed by a group of participants who are interested in purchasing knives.
The Conceptual Model Extraction study represents that a website should incorporate functional online store service, transparent documentation, a simplified user-friendly interface with enhanced visual designs and themes, necessary improvement in navigation and filtering options and resources as well as customer service guidance. The results are analyzed by a group of participants (professional chefs) who are interested in purchasing knives.
As mentioned above we aimed to cater to 3 main goals
Understand and conceptualize how users interact with the website
Understand the main frustrations/issues users face while using the site
Analyze the current interface - Does it satisfy the different attributes of usability?
The essence of using the above test successfully helped us in identifying and concluding solutions from the perspective of both participants (customers/users) domains as well as team (expert) domain. The studies brought multiple minute and major usability issues which further helped our team in developing improved and efficient mockup. The results are taken in consideration while creating a mock up, each usability issue was either improved or completely updated to meet the customer and client satisfaction level. We were able to successfully fulfill our main goals through the study.
Figure 4: The representation of suggestions and steps to be implemented moving forward based on the results of study. The reference of the image is represented in Appendix 1. It uses the psychology behind design such as gestalt, hierarchy, colors, and contents followed by context.
Figure 5: The mockup for Desktop interfaces referenced in Appendix 1.
Our Work
How did we redesign the website based on our study? (key revisions based on study results)
As the major usability issue relates to the navigation bar, therefore we implemented a top-screen navigation rather than the old sidebar with a drop-down menu. The products in the dropdown menu were properly sorted and categorized based on different applications. In addition, we implemented a hamburger menu for the mobile interface which ensured proper responsiveness of the website. The aim was to enhance and improve the aesthetic,vibrant and minimalist look of our website.
The study indicates an inconsistency in pages and elements of a webpage, therefore we
improved the colour scheme, font scheme, and product layouts in pages (aligned according to flex boxes and grids to provide it modern look), improved the footer and header of website, improved spelling errors, updating contact details, additions of store locations through maps, customer feedback section and removed broken links across pages.
The website was implemented on full screen both in mobile and PC interface with proper responsiveness.
Implementation of proper online store services with checkouts, price tags, offers and product information.
Instead of having vertical and horizontal navigation layout, we completely revamped the site by using nested navigation for every item to it’s corresponding category.
4.1 P-test (This is an additional part of the project with data in Appendix 1)
In order to examine the success of our mockup improvement we performed a p-test on the same participant pool of 10 users.
Hypothesis
The navigation of the new mockup website is better than the old version of the website.
The newer website is more efficient than the older website.
Null-Hypothesis
The navigation of the new mockup website is not better than the old version of the website.
The newer website is less efficient than the older website.
Reason to use p-test
Helps in proving hypotheses based on statistical data.
Highly efficient and reliable.
Participant pool
Professional chefs
Restaurant staff (e.g., kitchen staff, managers, servers)
5 individuals aged 22 to 29 interested in purchasing knives
Study Location
Online
Constructs/Research Questions and Data Collection
The data was collected from the same 10 participants of the think-aloud study. In the p-test study we gave participants 2 tasks where the first task was to find the cost of ‘Sujihiki’ knife, and the second task was to find the cost of ‘Inbayama’ knife. The participants performed the study in PC interface in both the old version of the website and the newer version mockup (link of website sent to participants) website. The time of completion of the task was recorded for both experiments and unpaired t-test was calculated for the same.
Result
The value of p=0.03217909282 (reference Appendix 1) is less than 0.05 therefore we can reject the null hypothesis. The study proved our hypothesis that (1)The navigation of the new mockup website is better than the old version of the website and (2)The newer website is more efficient than the older website. The data analysis methods are mean and standard deviation as shown in Figure 7.
Variables in study
The ‘dependent variable’ i.e. DV in the study is ‘task completion time’ in the old website and new mockup website while the ‘independent variable’ i.e. IV is an old website interface or a new mockup website interface.
Figure 7: Mean and Standard deviation (error bar) for task completion time in the new mockup and old website interface reference Appendix 1 for p test.
The result of the p test signifies that the new mockup website is efficient and better than the old website therefore our efforts to redesign the website based on solving the usability issue was successful.
Why we didn’t perform the following studies:
IMI: As the main goal was to understand usability issues, we needed more in depth affective evaluation rather than four sub-scale used in IMI. As well as we needed user feedback/recommendations to improve usability issues, which IMI isn’t sufficient for.
Cued debrief: This study requires a lot of setup (two cameras, a confined space) to and review sessions, and since we had limited time to conduct studies (approx. 20 mins), we chose not to conduct this study.
A/B testing: We needed quantitative data to support our study, therefore we went with a simple p-test rather then A/B testing, more importantly, we needed concrete data rather then fluctuating user feedback
Recommendations for client for Improving Website Design and Functionality
These are the recommendations to improve the design and functionality of the Santoku Knife Store website. Our suggestions aim to enhance user experience, increase customer satisfaction, and improve the overall performance of the website.
Incorporate a full-size page with a larger font size and zero margin/padding:
We recommend incorporating a full-size page with a larger font size and zero margin/padding to improve readability and make your website more accessible.Reconstruct the page theme for a more dynamic, minimalistic, and aesthetic design:
To make your website more visually appealing, we suggest reconstructing the page theme for a more dynamic, minimalistic, and aesthetic design.Implement an efficient operational e-store:
We recommend implementing an efficient operational e-store to provide customers with a seamless and hassle-free shopping experience.Improve font size and colour theme:
To make the website more user-friendly, we suggest improving the font size and colour theme to enhance readability and improve aesthetics.Add customer reviews and feedback section:
To improve customer engagement and trust, we recommend adding a customer reviews and feedback section to the website.Fix broken links and spelling errors: We suggest fixing any broken links and spelling errors on the website to improve its professionalism and reliability.
Incorporate alternate colour themes for colour-blind users:
To make the website more inclusive, we recommend incorporating alternate colour themes for colour-blind users.Adjust or improve navigation tab placement/content and improve the footer section: To improve the overall user experience, we suggest adjusting or improving navigation tab placement/content and improving the footer section.
Maintain a traditional connection with product details: We recommend maintaining a traditional connection with product details to provide customers with relevant information about your products.
Enhance newsletter incorporation and provide emails about events and new launches: To keep customers engaged, we suggest enhancing newsletter incorporation and providing emails about events and new launches.
Simplify the interface and layout to create a more user-friendly experience: To improve usability, we recommend simplifying the interface and layout to create a more user-friendly experience.
Improve navigation and filtering options to facilitate product search and browsing: To improve the shopping experience, we suggest improving navigation and filtering options to facilitate product search and browsing.
Enhance visual design with high-quality images, modern typography, and a visually appealing layout by using hierarchy: To make the website more visually appealing, we suggest enhancing visual design with high-quality images, modern typography, and a visually appealing layout by using hierarchy.
Fix online store functionality, ensuring a secure and seamless payment process: To provide customers with a safe and hassle-free shopping experience, we recommend fixing online store functionality, ensuring a secure and seamless payment process.
Offer resources, guidance, and customer support tailored to different user needs: To improve customer satisfaction, we suggest offering resources, guidance, and customer support tailored to different user needs.
Make it responsive: Optimize the website for mobile devices, ensuring a consistent experience across platforms. To provide a seamless experience across platforms, we recommend making the website responsive and optimizing it for mobile devices.
Provide clear documentation - such as return and shipping policies for transparency and ease of understanding: To improve customer trust, we suggest providing clear documentation, such as return and shipping policies, for transparency and ease of understanding.
These recommendations would be very useful and will help improve the website's design and functionality
Appendix
Link of raw data for Heuristic evaluation study docs file
Link of 10 SUS excel sheet with SUS SCALE Analysis
Link of raw data for Conceptual Model Extraction study docs file
P-test Excel file link for hypothesis
Presentation slide website link (figma link)
The prototype mockup of new website link (figma link)