{"id":1131,"date":"2018-01-30T23:03:14","date_gmt":"2018-01-30T23:03:14","guid":{"rendered":"http:\/\/robertvnewell.com\/?post_type=portfolio&#038;p=1131"},"modified":"2020-10-25T03:33:34","modified_gmt":"2020-10-25T03:33:34","slug":"right-sizing-user-testing","status":"publish","type":"portfolio","link":"http:\/\/robertvnewell.com\/?portfolio=right-sizing-user-testing","title":{"rendered":"Rightsizing Usability Testing"},"content":{"rendered":"<div class=\"wpb-content-wrapper\"><p>[vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<\/p>\n<h2>Rightsizing User Testing<\/h2>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]Test design, delivery, and cadence depends not only on the questions the team wants to ask, but also the priorities and ways of working we\u2019ve established.\u00a0 Scientific rigor is weighed with business goals to define acceptable risk. A researcher can own the insights or facilitate team ownership and synthesis. Here are some approaches I\u2019ve taken to align with goals and expectations across the product team.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<\/p>\n<h3>Robust Summative Testing<\/h3>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]Dell Data Protection is a suite of enterprise security management tools. \u00a0The program team needed to assess the usability of Dell&#8217;s local security management offering, Security Tools (DDP | ST), and understand where it sits in the competitive landscape.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1194&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]I designed and conducted a competitive benchmark for installing and using industry-leading security offerings. \u00a0I researched primary competitors, procured two competing systems to test, and ensured valid, statistically significant test data with a within-subjects design and counterbalanced tasks. \u00a0I led the study, but shared moderating across two comparative rounds with a contract usability engineer who I supervised. I owned the meta analysis and reporting across both.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1169&#8243; img_size=&#8221;full&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1167&#8243; img_size=&#8221;full&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<strong>Impact<\/strong><\/p>\n<ul>\n<li>Established performance benchmarks including task success, time on task, average non-critical errors, SUS-based ratings, and NPS scores<\/li>\n<li>The assessment directly informed new capabilities and helped establish differentiators that moved DDP|ST from being regarded as bloatware to a best-in-class security offering<\/li>\n<\/ul>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<\/p>\n<h3>Faster and Leaner Iterative Testing<\/h3>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]OpenStax prioritizes usability testing all new capabilities and changes. However, like with all Agile shops, testing needs to be lean, fast, and flexible, and testers need to be resourceful.<\/p>\n<p>As UX research lead I wanted to establish a usability testing protocol at OpenStax to improve consistency, effectiveness, and turnaround times in a sprint-based environment.<\/p>\n<p>I ensured all test plans, no matter how lean, start with hypotheses and\/or objectives and critical research questions. \u00a0That leads to a script with explicit language and &#8220;stage directions&#8221; to ensure an easy handoff to multiple facilitators.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1179&#8243; img_size=&#8221;full&#8221; css=&#8221;.vc_custom_1590774359145{border-radius: 1px !important;}&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]I always built around scenarios that captured real world contexts and placed features within larger task flows. \u00a0For example, when analytics told us users were abandoning our book adoption form, I was asked to test the form&#8217;s micro-interactions. \u00a0I built a scenario, however, that started from the form&#8217;s touchpoint, a marketing email. \u00a0We found users didn&#8217;t expect to be led to a form, but rather, to more information on book adoptions. We then revisited content and CTAs to more clearly telegraph the form, significantly reducing dropoffs.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1181&#8243; img_size=&#8221;full&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]We stayed flexible with test plans, sometimes performing rapid iterations between test sessions based on early findings (RITE testing). \u00a0Findings reports were no frills Google Docs that led with takeaways and recommendations, followed by a detailed summary and itemized usability issues.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1183&#8243; img_size=&#8221;full&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]Since I worked closely with the product manager, I didn&#8217;t need illustrations or complicated slide decks to communicate findings. \u00a0So I kept documentation in Google Docs. \u00a0This also allowed for easy linking between user stories and video clips. \u00a0And it encouraged discussion and decisions by stakeholders directly in the report.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1185&#8243; img_size=&#8221;full&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]Finally, I created test plan and findings report templates to speed up planning for other team members and ensure a consistent protocol across tests.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<strong>Impact<\/strong><\/p>\n<ul>\n<li>Faster and more frequent iterations let to better experiences &#8212; When OpenStax Tutor Beta was launched in Fall 2017, customer support saw a 20% <em>drop<\/em> in support requests<\/li>\n<li>Established 3-6 hour turnaround times for findings reports after most testing<\/li>\n<li>Templates and explicit test plans ensured reliability across facilitators and tests<\/li>\n<li>Established rapid iterative prototyping (RITE testing) within tests<\/li>\n<li>Enabled remote and live testing with multiple facilitators beyond the UX team<\/li>\n<\/ul>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<\/p>\n<h3>Team Insights in a <del>1-Week<\/del>\u00a04-Day Design Sprint<\/h3>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]Pearson moved to 1-week design sprints to accelerate iteration and handoffs to development. While the designer led the ideation and I led the testing, we all moved through the sprint activities as a group. We collaborated remotely through several Mural boards.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1189&#8243; img_size=&#8221;full&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]I built the test plan in one afternoon from a rough storyboard in parallel with the UXD as he built the prototype. \u00a0The next morning I moderated all the test sessions while the team took notes in a Mural template I created.[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1191&#8243; img_size=&#8221;full&#8221; css=&#8221;.vc_custom_1590778616253{border-radius: 1px !important;}&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]I led the synthesis with the group that afternoon. \u00a0We documented success rates and ease ratings and clustered qualitative observations into themes. Rather than own the insights as an analyst, I shared that job with the entire team as a facilitator. \u00a0We pulled out key findings and highlighted them on the Mural board. I documented them later in a simply formatted Word doc for easier reading and access by non-participating stakeholders[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_single_image image=&#8221;1226&#8243; img_size=&#8221;full&#8221; css=&#8221;.vc_custom_1590792827620{border-radius: 1px !important;}&#8221;][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]<strong>Impact<\/strong><\/p>\n<ul>\n<li>Problem to validated solution in four days with visual design handoff to development within a sprint&#8211; sometimes 2X within a sprint; This increased velocity and moved our process further away from waterfall.<\/li>\n<li>Moved to testing all in one morning with synthesis in an hour that afternoon<\/li>\n<li>The team owned and shared the insights, improving alignment and requiring less documentation<\/li>\n<li>Broke down silos by bringing stakeholders across departments together to ideate and validate as one team for the first time<\/li>\n<\/ul>\n<p>[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row]<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>[vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text] Rightsizing User Testing [\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/6&#8243;][\/vc_column][\/vc_row][vc_row][vc_column width=&#8221;1\/6&#8243;][\/vc_column][vc_column width=&#8221;2\/3&#8243;][vc_column_text]Test design, delivery, and cadence depends not only on the questions the team wants to ask, but also the priorities and ways of working we\u2019ve established.\u00a0 Scientific rigor is weighed with business goals to define acceptable risk. A researcher can own the insights or facilitate team&hellip;<\/p>\n","protected":false},"author":1,"featured_media":849,"comment_status":"open","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"portfolio_category":[20,3,5],"portfolio_tag":[],"acf":[],"_links":{"self":[{"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/portfolio\/1131"}],"collection":[{"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/portfolio"}],"about":[{"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/types\/portfolio"}],"author":[{"embeddable":true,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1131"}],"version-history":[{"count":76,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/portfolio\/1131\/revisions"}],"predecessor-version":[{"id":1246,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/portfolio\/1131\/revisions\/1246"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=\/wp\/v2\/media\/849"}],"wp:attachment":[{"href":"http:\/\/robertvnewell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1131"}],"wp:term":[{"taxonomy":"portfolio_category","embeddable":true,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fportfolio_category&post=1131"},{"taxonomy":"portfolio_tag","embeddable":true,"href":"http:\/\/robertvnewell.com\/index.php?rest_route=%2Fwp%2Fv2%2Fportfolio_tag&post=1131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}