Here’s what happened when we ran usability testing with staff members using Learn for the first time. From four videos we found 20 usability issues, and a wide variety of strategies to complete the same basic tasks.
Archive — User experience
We had developed an information architecture and tree tests as part of our programme of user research for Learn Foundations. The next step was to use first click tests to pit the new template against existing courses.
The latest post in my series for the Website and Communications blog about our user research work around the University of Edinburgh’s virtual learning environment.
Slides from my Edinburgh UX meetup talk on Monday 2 September 2019, about the user research we have been conducting around the needs of students and staff working with course materials digitally at the University of Edinburgh. See the more detailed blog posts about this project over at the Website and Communications team blog. Read full articleComment
Note — 2019-09-02
I’m doing a couple of talks this week. They are both about the user research we’ve been doing for the Learn Foundations project.
This evening I will be presenting at the Edinburgh UX monthly meetup. It’s a friendly meetup and it’s free, so do come along if you’re interested.
Then on Wednesday I’ll be presenting with my colleagues Karen Howie and Paul Smyth at the Association for Learning Technology (ALT) Annual Conference.
After completing the top tasks survey and the card sort as part of the Learn Foundations project, our next step was to create a prototype information architecture and test it.
It’s always great to see advice from Indi Young. Here are tips on how to better identify and synthesise patterns in qualitative data.
…when you’re looking at data, don’t group things together by noun. Group them together by verb. I’ve done a lot of work with the healthcare industry, and one thing I often see research teams do is bring together insights that are all about a noun — here is all of the data that we got about how people feel about the doctors. But when you do that the intent behind what people are really saying ends up all over the place.
How we used card sorting to help us devise a consistent information architecture for Learn VLE courses at the University of Edinburgh.
775 students participated in the study — and no two students submitted the same card sort. This highlights the great challenge faced by the Learn Foundations project in attempting to create a more standardised template that meets the wide variety of needs across the University.
✔️ Love stationery
✔️ Love workshops
This is a great guide to workshop essentials. I’m impressed that this kit contains a wider variety of materials, and yet seems so much smaller than the workshop bag we use at work. Maybe we rely too much on mountains of sticky notes!
I’d be tempted to add planning poker cards to this list. Planning poker is usually thought of as a technique for estimating work in agile projects. But it can also be used as a prioritisation technique in workshops.
I’ve been thinking and reading a lot about co-design recently (as well as doing some of it too). This website, Beyond Sticky Notes, has provided me with even more food for thought.
I am particularly struck by the table describing various approaches from transactional to transformational. In this model, “Anything ‘centred’ — human, user, patient etc.” is little better than “Designer as expert”. Meanwhile, what I thought of as co-design may actually be more like participatory design. There’s so much more to do.
But one line of warning is familiar to any good user experience practitioner, and is worth repeating until the cows come home.
Co-design builds long term commitment. By contrast, consultation often gives the illusion we’ve bought people on board — only to have them then fall overboard. With consultation, we pay later — often in costly, public and damaging ways.”
Make sure you also see Mindsets for co-design, another enlightening article on how to do co-design better.
This website is in support of a book due to be published in 2020. I am now looking forward to it.
Thanks to Alison Wright who retweeted the latter article and brought it to my attention.
As part of our programme of user research in support of the Learn Foundations project, we have carried out a top tasks survey to understand what students need when accessing course materials online.
What we found was that students value three items much more than everything else. Those items are all to do with lectures.
See the full post to find out more.
As part of the Learn Foundations project, we have carried out a programme of quantitative research to ensure a user-centred approach to solution development.
The Learn Foundations project team wanted to develop a new template using a user-centred approach. This template would be designed to introduce more consistency between different courses in Learn. But it also had to support a diverse variety of needs across different courses, supporting different schools, colleges and teaching needs. It also had to be developed quickly.
We took inspiration from a classic user experience diagram to ensure this new template could be built on firm foundations.
This post introduces the steps we took. Forthcoming posts will describe each step in more detail and some of our key findings.
Summarising the key findings from a set of user interviews I conducted with students on their needs around accessing course materials digitally. Just one of the strands of the Learn Foundations project, which I still have much more to write about.
After analysing and synthesising the insights gathered through the interviews, we built up a picture of how and why students’ experience with Learn varies throughout the year as students attempt to complete different tasks. This is presented as a semester in the life of students using Learn.
My awesome colleague Lauren Tormey wrote this blog post about a brilliant project she’s been involved in. She has been collaborating with our Information Services Helpline to reduce unnecessary support calls by iteratively improving content with a regular cycle of usability testing.
Over two summers, we had done work to improve content related to getting a student ID card. This was another case of turning long pages with giant paragraphs into concise step-by-step pages.
From July to September 2017, the IS Helpline received 433 enquires related to student cards. For this same period in 2018, they received 224, so the figure nearly halved. I repeat: halved.
An article published yesterday in The Washington Post demonstrates the danger of design’s failure to broaden popular understanding of our craft.
The article pinpoints Nest’s focus on reducing friction as the reason for their cameras’ weak security.
Khoi Vinh points out that…
…the concept of user experience writ large is not to blame here; what’s actually at fault is bad user experience practice.
The point being that good security is fundamental to good user experience. As any good designer would know, they are not in conflict. Quite the opposite, in fact.
It strikes me that Nest are using ‘reducing friction’ as a poor excuse for not implementing better security. I’m sure they’re not the only ones guilty of this.
On another point, this article got me thinking about journalism. Khoi Vinh refuses to blame the Washington Post’s perspective on “lazy journalism”, perhaps correctly.
But any time I read a mainstream/non-specialist journalist write about a topic I know a little about (motorsport, the web, whatever), I’m always astonished at how many basic errors are made. It’s a challenge if designers want the help of journalism when “explaining what it is that we do to the world at large.”
Since September, my main focus at work has been to carry out a comprehensive programme of user research for a project aiming to improve services surrounding Blackboard Learn, the University of Edinburgh’s main virtual learning environment.
I wrote this blog post providing a high-level overview of all the work that’s taken place this academic year. More detailed blog posts about each of the strands of research will come in due course.
This is been a brilliant project to be involved in. We’ve been given a lot of time and freedom to do large amount of research in support of one of the university’s most important digital services, used daily by most of our students, and many staff members.
We have made some really important discoveries. This work is ensuring that improvements are based on a strong understanding of users’ behaviour and needs when working with course materials digitally.
Check out this video, where I describe the work and some of the findings in a bit more detail, and keep an eye out for the forthcoming blog posts.
Personas are one of the most popular techniques in the user experience toolkit, but they also remain among the most controversial. It is often still unclear to some what value personas can bring, and how to avoid the pitfalls of bad personas.
This article brings one of the clearest explanations I’ve seen of how to make good personas. It is a lengthy but must-read article if you make personas and want to make them work.
This article is particularly useful at explaining why obsessing over demographics is bad, and why you should instead focus on “thinking styles”.
Statements-of-fact, preferences, and demographics frequently serve as distracting barriers. They kick off all kinds of subconscious reactions in team members minds.
My colleague Alex Burford from the University of Edinburgh School of Informatics has written this great blog post about some usability testing we have conducted in support of the Learn Foundations project.
I thoroughly enjoyed working with Duncan Stephen on this mini project. The feedback was informative, encouraging, and a call to action. I’m looking forward to embedding similar practice across the School for alternative platforms for content delivery.
Each month we are working with a different school to conduct usability testing in Learn, the virtual learning environment, to inform improvements to the Learn service.
This is just one strand of a huge amount of user research I’ve been carrying out for the Learn Foundations project. It’s been a fascinating and very enjoyable project to work on. I’ve been pretty lax at writing about it yet — but I’ll be posting much more about it soon.
A very useful contribution to the debate surrounding the usefulness/harmfulness of net promoter score. Jeff Sauro transcends the often polemical nature of the debate, by analysing actual research on the effectiveness of net promoter score.
The news still isn’t all that great for proponents of net promoter score. But at the same time, it’s not quite as bad as its detractors make out.
Kudos to Jeff Sauro for doing some actual research on this.
Dial in the feedback — Gregg Bernstein
UX your life: Applying the user-centered process to your life (and stuff) — JD Jordan, Smashing Magazine
I’m always in two minds about whether people should use work-based techniques on personal problems. I have heard of people using Trello boards at home to organise tasks, which sounds as nightmarish as it sounds sensible. I’ve even heard of people running scrum-style weekly planning meetings with their family, which definitely sounds overboard to me.
But I do like the look of some of the ideas here. For instance, I’m keen to map out out my life in weeks.
And I already know that affinity mapping can work great at home and for other stuff.
When we did the MoRun in November, Lauren and I made an affinity map to decide which of two runs to enter. My gut feeling told me another run would be better. But writing down all the pros and cons of each race, and grouping them, made it clear that my gut feeling was actually wide of the mark.
An enjoyable and informative history of user experience. Some familiar themes, but not entirely your standard take. A reminder that people have been doing something like user-centred design for longer than we sometimes think.
…UX is not really a new thing. It might seem new to your organisation and its design process, but in fact it’s been emerging since before the dawn of the internet, back in the 80s, and people have been looking to solve similar problems for almost 140 years.
Keeping it weird
Or, more accurately, stopping it being weird. This refers to the problem that most psychology research is conducted on people that are western, educated, industrialized, rich and democratic.
Tim Kadlec considers the implication this has on our understanding of how people use the web.
We’ve known for a while that the worldwide web was becoming increasingly that: worldwide. As we try to reach people in different parts of the globe with very different daily realities, we have to be willing to rethink our assumptions. We have to be willing to revisit our research and findings with fresh eyes so that we can see what holds true, what doesn’t, and where.
Making work meaningful: A leader’s guide
McKinsey report on how to engage employees.
People who find meaning at work are happier, more productive, and more engaged. Four practical interventions can help make the search more likely to succeed.
I am struck by how two of the four interventions listed are fundamentally about understanding your users better.
Talk with employees about who their customers are, and encourage each employee to connect with one.
Build regular, face-to-face interactions with customers into existing processes, stimulating employees to learn who is most affected by their work.
Help people grasp the impact of their work
Invite customers who have had the best—and worst—experiences with your products to talk with employees in person so your team can see how their work affects customers.
Another reason why user experience is worth it.
Distracted driving — UX’s responsibility to do no harm
More on the need for (UX) designers to consider ethics in everything they do.
I urge you to consider your own design priorities and choices in the same way that responsible physicians do when they take the Hippocratic Oath, saying “first, do no harm.” So, I ask the UX community at large: what is an equivalent code of ethics for our discipline?
Security design: Stop trying to fix the user
On the tendency of security approaches to rely on somehow educating users on this complex problem.
I’ve read dozens of studies about how to get people to pay attention to security warnings. We can tweak their wording, highlight them in red, and jiggle them on the screen, but nothing works because users know the warnings are invariably meaningless. They don’t see “the certificate has expired; are you sure you want to go to this webpage?” They see, “I’m an annoying message preventing you from reading a webpage. Click here to get rid of me.”…
We must stop trying to fix the user to achieve security. We’ll never get there, and research toward those goals just obscures the real problems. Usable security does not mean “getting people to do what we want.” It means creating security that works, given (or despite) what people do.
The same could be said for usability of any kind — but it seems especially vital in this case.
6 mistakes that prevent UX teams from having boardroom influence
A good list of don’ts when you’re trying to set up an effective user experience function.
In particular, the pitfalls of “cargo cult usability” could do with being more widely understood. But I also enjoyed this point about being too insular.
Newly formed UX teams have a tendency to quickly turn inwards and focus heavily on their own practices, tools and methods: heads down, working in a vacuum, doing great work that doesn’t actually influence anything. As a result, we hear frustrated stakeholders say things like: “I don’t involve the UX team because they always seem too busy”. We’ve even heard UX team members themselves complain that, “We’re so busy and so mired in the day-to-day that we don’t have time to work alongside the development team.”
This reminds me of the (hilarious but true) story of the Staffordshire UK bus company. In 1976 it was reported that the buses on the Hanley to Bagnall route were not stopping to pick up passengers. People complained that buses would drive right by long lines of waiting passengers. The complaints prompted Councillor Arthur Cholerton to make transport history by stating that if the buses stopped to pick up passengers it would disrupt the timetable!
The hunt for missing expectations
Jared Spool tells the story of a bookkeeper who became frustrated using Google Sheets because it didn’t have a double underline function.
To keep [usability] testing simple and under control, we often define the outcomes we want. For example, in testing Google Spreadsheet, we might have a profit and loss statement we’d want participants to make. To make it clear what we were expecting, we might show the final report we’d like them to make.
Since we never thought about the importance of double underlines, our sample final report wouldn’t have them. Our participant, wanting to do what we’ve asked of her, would unlikely add double underlines in. Our bias is reflected in the test results and we won’t uncover the missing expectation.
He suggests interview-based task design as a way of finding these missing expectations. Start a session with an interview to discover these expectations. Then construct a usability test task based on that.
I recently ran hybrid interviews and usability tests. That was for expediency. I didn’t base tasks on what I’d found in the interview. But it’s good to know I wasn’t completely barking up the wrong tree. I plan to use this approach in future.
Smart voice assistants and smart homes — from the past
A really enjoyable piece on the history of smart home devices, and how Google Home and Alexa aren’t such new ideas. The video is well worth a watch, particularly because it demonstrates 1970s technology from Pico Electronics in Glenrothes! It’s amazing to see it work so well.
The point of Thomas Baekdal’s piece here is to demonstrate how trends aren’t new, but they emerge over a long period of time. It reminds me a bit of Gartner’s hype cycle, and a recent Nile webinar about how to employ foresight to understand emerging trends. Not to forget the Nielsen Norman Group research demonstrating that intelligent assistants still have horrible usability problems.
Designers are defining usability too narrowly
Another call on designers to think more widely when they are working on digital products. Khoi Vinh saw a Nielsen Norman Group report on best practice on websites aimed at children — but he felt the report focused too narrowly on usability.
I don’t dispute the findings at all. But it’s disturbing that the report focuses exclusively on usability recommendations, on the executional aspect of creating digital products for kids. There’s not a single line, much less a section, that cares to examine how design impacts the well-being of children…
We’re moving past the stage in the evolution of our craft when we can safely consider its practice to be neutral, to be without inherent virtue or without inherent vice. At some point, making it easier and easier to pull the handle on a slot machine reflects on the intentions of the designer of that experience.
Thinking in triplicate
This is a very strong piece by Erika Hall, raising some seriously good points and questions about where user experience design is, and where it needs to go. It is well worth reading the full piece, and me pulling out a quote cannot do this justice. But here are some selections I particularly liked:
If good design entailed good business, women’s clothes would come in a wide range of sizes with usable pockets and our social media feeds would unfurl in reverse chronological order with an unremarkable absence of Nazis.
While most of the designers I know are far from objectivists, design as it is currently practiced is tantamount to Ayn Rand’s radical selfishness. We design for the experience of a single user at a time and expect that the collective experience, and the collective impact, will take care of itself.
It’s much more pleasant for designers to talk about empathy in one room and MBAs to talk about profits in the other and have marketers in the middle like an injectable filler.
This is exactly the sort of article we need to be seeing more of.
Keeping digital teams happy versus keeping customers happy
Gerry McGovern tells the story of trying to persuade a digital team of what they needed to fix.
“It would be nice to fix these problems,” one person said. “But the team needs also to be able to do exciting things. We need to be able to innovate.”
Unfortunately, people at work often place too much emphasis on their own enjoyment. But our work only has meaning if it is providing value to someone.
Work shouldn’t be exciting. There’s a job to do.
Proactive UX design: A big leap requiring baby steps
An excellent article from Jared Spool on the difference between proactive design and reactive design — and the importance of making your work more proactive.
Reactive UX design is just what it sounds like: reacting to a problem in the moment. “Oh, can you fix this?” “Help! Users are complaining this is too hard! What can we do?”
Without also having proactive UX design efforts, the design team is only fixing problems caused by decisions the product team has already made.
Interestingly, he also makes the point that it is easy for design teams to get sucked into doing reactive design, because it becomes comfortable for teams to do:
They like the wireframes and usability tests.
They believe this is what design work looks like. They believe design work always happens at the end of the process.
Keeping yourself out of the story: Controlling experimenter effects
How do you stop yourself, as a user researcher, biasing the results? An important topic for user researchers to consider. (It’s also an excellent excuse to re-tell the story about Clever Hans, the horse who everyone thought could count, until they realised he was simply reacting to subtle, unintentional cues from his trainer.)
I recently undertook some usability testing, where I was asking people to complete tasks that I didn’t know how to complete myself. This meant I was less likely to bias the participant. But it was a strange experience for me, and it made me less certain about how to conduct the test.
Why content should be published in HTML and not PDF
The crusade against PDFs has been one of my constant hobby-horses over the years. It has also led to some of my toughest battles in my work.
Users hate PDFs, because it makes it harder to use content. But content owners love PDFs, because it makes it easier for them to create content. It is the ultimate in user-hostility. “Who cares about the users? PDFs make my job easier for me.”
So it was great to see two trusted sources reiterate the importance of getting rid of PDFs, within days of each other.
This has also reminded me of a small project I promised I would do, but never got around to — to publish my dissertation as an HTML webpage. The idea was to demonstrate how versatile HTML is, even for things like technical or academic writing. Maybe I’ll return to that this autumn.
I really like this idea of crowdsourcing, and making available to the community, a set of readability guidelines based on evidence.
I see many content designers spending time talking – arguing – about points of style when often accessibility and usability show what we should do.
What if there was one place where we, as a community, shared knowledge and created a style guide that was accessible, usable and – if we wanted – evidenced?
We could then spend time on the things that matter more to our organisations.
The sound of silence: What we’re not saying about Siri and her AI gal pals
Why are digital assistants almost always given female-sounding voices?
While stakeholder preference might sound like a perfectly good reason at first, it hides an ugly reality. To make this clear, let me tell you a story about a talented young woman who I managed. She designed voice features for our clients’ prototypes. Although she created a voice that was meant to be genderless, the client kept referring to the voice in feminine terms. In other words, he heard what he expected to hear.
…BMW learned the hard way that female voices aren’t always the right route to take when German drivers of its 5 Series vehicles complained about “taking directions from a woman.” Yes, really.
Annoying online ads cost business
Results from a study of users of Pandora has quantified the effect of shoving adverts in users’ faces. As part of the experiment, a section of users were served fewer ads than normal, and another section were served more ads than normal.
…after 1.5 years of being exposed to the experimental conditions, people did use the service more, the fewer ads they were served. At the end of the experiment:
- The low-ad group listened for 1.7% more hours weekly than the control group.
- The high-ad group listened for 2.8% fewer hours weekly than the control group.
What is this thing called design?
I got out of bed and, in roughly an hour, hammered out a kind of primer on UX/UI design, which I’m publishing below. It’s a very unformed, rambly screed that I won’t pretend is at all definitive or even fully accurate. In fact it’s still basically a first draft; I literally typed it out in bullet point form, as shown below, a trick I used in order to absolve myself of the responsibility of writing a fully articulated essay.
Despite Khoi Vinh’s self-deprecation here, I think this is an excellent attempt at explaining what design is. For those who are frustrated about having to explain that design isn’t (just) about making things pretty, this blog post provides an excellent introduction to why — as well as helpfully explaining why this perception exists in the first place. Not bad for an hour’s work.