News from April, 2008

Recently, I had a discussion with Shane Nuessler from The Australian National University where he mentioned it would be nice if I could put a presentation together for the next conference on design analysis, usability testing, and any other methods for understanding how to find and document problems in Sakai's UX using user-centered methods.

I loved the idea but just not sure how my schedule will look in the coming months. So in the meanwhile, I thought I'd take a few minutes and whip together a quick blog post to summarize some of the different methods I've used.

Some of these methods can be used for testing existing products, but I like to think more in terms of testing new designs. Either way, I hope you find them useful.

----------------

1. Surveys

The trick with surveys is knowing how and what to ask. Without a direct dialog to affirm and discuss the real issues, it's easy to miss the heart of the problem.

But that's not to say they aren't useful. Just try to ask very specific questions like: Do you have dial-up or broadband where you use your system?

Asking general questions like "Do you think the portal is easy to use?" can lead to fairly mixed results.

2. Focus Groups

These are good for general discussions about a product, but they do carry the common pitfalls of group-thought, personality conflicts, shyness, etc.

It also depends on the methodology used. If a group of random people were asked to perform certain tasks in an application, then later came together to discuss their opinions, that could turn out to be a rewarding discussion.

But if the group is lead by a facilitator who walks them through a design and then asks for their opinions, the feedback is fairly tainted since the users weren't given a chance to figure out the UI on their own.

3. One-On-One Interview

When I do these I like to have a few goals in the back of my mind to help keep the discussion moving along – but not much more in terms of preparation. Bringing in a questionnaire usually makes the process too structured to really hear what the user is saying. Instead, I suggest trying to be as apathetic and philosophical with the interviewee as possible. If you can go out and grab a beer with him/her, even better. You really want to try to get them to let their hair down and share issues that go beyond the specifics of the tool, but more into the nature of why they use the tool.

But despite your best efforts, some users just don't know what the problem is. Sometimes they don't realize there are be better ways of doing something than how they're currently doing it. So you might have a lot of users tell you they like something, when in fact the UI or functionality is problematic.

4. Contextual Inquiries (observing people in their own environment)

This is a really nice way to get a glimpse into what a user is doing without relying on them to remember or explain some issue they've had with the software in the past.

If you're only at the design stage with your product, this might not help give you any insight into how your product is being used, but it's still nice to watch a user use other products in their environemtns – particualrly if the user fits your target "persona".

You'd be surprised how shadowing and quietly observing a user can clue you into all sorts of things that spur innovative improvements to software.

5 Heuristic analysis

There's a ton of examples on this all over the web, so I won't re-purpose them here. But I am working on the UX Scorecard which will have a good format for performing these types of reviews.

Basically, the idea is to get a panel of experts together (by experts, I mean designers who have an eye and vocabulary for design heuristics) and have them review the product.

This tends to be a good way to get some very general insight into the some of the common problem areas, but it rarely leads to any deep interaction analysis beyond just screen-level issues. I still think this is a great way to get your feet wet with a fairly nice method for reviewing and documenting a products UI.

6. Usability Testing

Last but not least, good 'ol usability testing. IMO, there is no better way to understand if a design is going to fly or not. There's a lot of different standards in testing – some are even ISO approved. But rather than going the fancy route, I tend to use quick and dirty tests just to get feedback early on while still in a design phase of a project.

When preparing a test, the thing you put in front of users can range from hand paper sketches to a semi-functional html prototype. Whatever is most economic for your organization. I prefer hand sketches myself since you can easily change them in mid-test and they don't take much to prepare.

To get going, I suggest you come up with a set of screens that revolve around a few tasks that you want to test. Don't try to test too much in one sitting as your prototype will get large and unwieldy. Just a few simple screens will do. Once you have your tasks outlined in your mind and have prepared the screens you'll be putting in front of the user, you're pretty much ready to go. Just schedule a few users and have them come in on a day that works well for everyone.

Oh, despite the central limit theorem (having at least 30 observations), Jakob Nielsen seems to think that 5 users is enough:

http://www.useit.com/alertbox/20000319.html

From my perspective, the statistical accuracy is not as important as simply seeing how users react to a design. From that perspective, 1 user is better than none.

Anyway, if you can get your screens into an electronic format (i.e. sketch them out in Visio, Photoshop, etc.. or scan in hand sketches), you can use a video recording tool like Camtasia to do all the documenting for you. That can be a real joy once you have everything setup properly.

A few things you may want to do to make the testing workout a little better for you:

a) Try to let the user know it's not them but rather the software that's being tested. You may need to reiterate this point a few times... and even still, you'll probably have some user's say things like "Oh, sorry! That was dumb of me!". Just let them know that they found a design flaw and that's exactly what you're looking for.

You may also want to stay away from calling it "user testing" as you're not testing them, but the software – so "usability testing" is possibly a less intimidating way to refer to it.

b) Invoke the verbal protocol (but don't also test for how long it takes for them to complete a task). Basically, keep encouraging the user to think out loud. They'll stop periodically, as it's not natural for them to talk about what they're thinking at every step, but give them a light prod and they should perk back up

c) Try to avoid hinting in anyway! Keep the questions focused on the task or goal. If the user asks how to do something, feel free to play the amateur psychologist and ask "how do you think you should do it?" If they're completely stumped, you've found a design flaw.

Sometimes I setup a system where if the user asks more than once, I ask them if they want to call tech support? If they say "yes", that's when I answer their question. You might have an interim step where you guide them to online help documentation – which makes for a nice way to test two birds with one stone: the product and the help docs.

With usability testing, you could try to focus on the following areas:

• Effectiveness – have they completed the task? This is categoric (yes or no)
• Efficiency – I'm tempted to suggest you avoid this one entirely. But basically, you could look for how long it took them to complete a task, or whether they chose the "preferred" path, etc. This one's a bit contentious though and should certainly be used with caution – especially if the verbal protocol is also used.
• User Satisfaction – at the end of a task and/or the entire test, it's a good time to get some free-form feedback about likes and dislikes.

Anyway, I hope this helps for now. As for how to capture and analyze all this data, that will have to be something I cover in a future post.

Regards!

Take a look this week at how the basic template idea is shaping up as well as how users can add widgets to their site.

Topics covered in this design include:

• Changes to the copy site screen
• Exploring templates a bit further
• Adding, viewing, and removing widgets
• Modifying the page layout

To learn more, take a look at this week's flash presentation.

Companion screens below:

Change Layout wo/ Sidebar	Change Layout w/ Sidebar	Widget Added
Add Widget to Page	Add Widget - Alt View	Predefined Templates
My Templates (conceptual)	My Templates	Copy Site (modified)

UX Project Feedback - Open Call

Several data points came out of the meeting:

A. The design should incorporate some ideation around a "Sandbox" site type as an option that lets instructors (and possibly other users) take Sakai for a test spin before committing to building a site of their own.

This is a good starting point, but it would be helpful to get specific use-cases around this idea.

B. There was a desire to see the design present more "real world" examples. There was some concern that the direction might not scale well when, for example 50+ sites or tools are added to a user's layout.

In general, there was a desire to see more detail. One request had to do with how tools might be ordered more elegantly so that once added to the layout, they would offer more options than just a first in first out ordering method.

Someone put the request for more details it in terms of seeing a before and after – like an "extreme makeover"!

My fumbling response to that was (sorry guys, I'm on practically no sleep at the moment due to exams) that the design is all broad strokes at the moment. I'm still dealing with the basic building blocks of this thing, so please be patient about the details. Believe me, I am just as anxious to dive into details and see how they fan out. So as soon as I get beyond a few more larger chunks of the product, I'll make it a point to come up with several mock-ups that show more real world examples and use-cases in action.

In part, that's the reason why we're talking implementation at the moment. We'd like to get a prototype track going where users can begin to interact with real data so we can see where the leaks are and be in a better position to address them.

C. There were comments that touched on what appears to be a recurring concern about the design offering too much!

Folks shared a bit of skepticism that the design is impractical in a real-world deployment where instructors or other users are NOT given access to build and manage sites in such a free-form way. Instead, the more common scenario is that instructors are given a site or two that are designed by someone else (like an instructional designer) and they often have little need (or ability) to dramatically modify it – much less build a site from scratch.

This point is well taken and certainly has not escaped my attention. There are two things I want to say about this issue:

1. We're not yet at a point where we're packing this thing up and driving it into the trunk. This design is just that, a design. More over, it's a design happening in real-time. You're seeing it emerge and in the process are privy to the toiling involved in the the trial and error. It's natural to be concerned that this is what the finished product will be, but let me assure you, it's far from it!

Before this project started, I thought quite extensively about how to approach the process. Either behind a curtain or out in the open. I opted for the latter as I think it better mirrors the spirit of this community. But that approach introduces confusion – so I apologize if I haven't managed the communications better.

I'll continue to emphasize this point as the process moves along.

Also, keep in mind, the reason I wanted to do it in the open was to make sure everyone who wants to have a say into the design will be given the chance to. So try not to feel skeptical, but rather look at it as a work in progress with your feedback playing a significant role!

2. I realize that the design presents MUCH functionality in the UX. There are several reasons I chose this route:

• Not all institutions lock things down for their users.
• Sakai does not necessarily (albeit it mostly is today) used only in the academic context. <-- This is not a principle driver, so don't get too hung up on this point.
• It's easier to scale down than up. If I designed a limited solution, it would be much harder to envision something more robust for those that need it. It's easier to go the other way.

And on that last point, keep in mind that there will certainly be an admin side to all of this that should allow for extensive configurations! So much of what you see will be very much controlled from institution to institution by the system admin. Ideally, I'd like to see an admin control panel that is cleaned up to support ease of use and efficiency for the admin – but that often comes down to a trade-off in priorities

D. There was a question about Fluid and involving their efforts.

Yes, I agree. This has been one of the main goals from the outset of this project. Once they get some more time, I'm confident we'll be collaborating much more.

E. And finally, a few folks had questions about Paris and any workshops there that could be offered around this project.

Unfortunately, my phone or my ears seem to be going (probably a combination of both) and I missed much of what was said, but John Norman mentioned something about preparing an event either during or after – as well as something during the JASIG conference. But again, I didn't fully catch everything so if someone can add those details here, I'd appreciate it.

Personally, my only plans for Paris are to discuss the UX Improvement Initiative and give a brief showing of progress being done. The format will be a 1 hour presentation. I'd prepare more of a brainstorming and planning workshop, but unfortunately, I just won't have the time this go-around.

Take a peak at how Sakai's new roles and permissions system works.

Topics covered in this design include:

• A completely new and robust way of handling roles and permissions!
• I added a dynamic hide/reveal option to the sidebar
• Rearranged where colors, styles, and other customization features are located
• Added a "styles" option that let's users pick between different pre-defined moods (layout not included)
• Did some light tweaking throughout

To learn more, take a look at this week's flash presentation.

*Companion screens below:

*Screen revision updates are included in this batch so you'll want to hunt through and find the screens that relate only to the video presentation.

First round of visual design.

No video this week guys (sorry). But I do have some juicy eye-candy.

From a visual design perspective, my goals were as follows:

1. Create a modern (web 2.0) look and feel.
2. Create a skin that's fairly easy to dissect and modify. In other words, the visual elements should be simple, but not so simple as to lose all personality.
3. Create a concept that takes a step forward in branding Sakai. As of today, Sakai lacks any visual ownership beyond its logo. What I tried to do is expand on the playful elements (strokes of the type and symbol, colors, etc.) while still retaining an enterprise application UI.

The initial stages of the visual design were actually formed during the interaction design work I had presented earlier. I added color, refined various elements, and introduced some conceptual messaging (meet the Sakai Owl).

Important: Please do not focus too much on the content in the widgets. While the content suggests a social network environment, it is only a concept and used purely to foster the visual design exercise.

Also, it's worth mentioning that this visual design is prepared primarily for default Sakai. Your campus will have the option to customize the style to meet your branding requirements.

Enjoy!

It recently occurred to me that some of you have questions regarding the general approach that's been taken with the UX Improvement Initiative as well as the current project: Improving the CLE v1.0. I had plans to elaborate on this in Paris, but given the questions I've received, I thought it might be useful if I shed some light on it now.

Initiative vs. project

There are two things happening here: The UX Improvement Initiative and the Projects that fall within it. While "improvement" is fairly broad, the initiative is not a catch-all for any and all UX activity. Instead, the initiative's aim is to improve Sakai in a fairly specific way (more on this further down).

The initiative will use a series of two-month projects, each producing deliverables that have an immediately useful and relevant purpose. In other words, projects will have well defined goals and concrete deliverables intent on solving current design challenges in Sakai.

Take the current project for example. The goals are well defined in the project's profile page and the deliverables will be a set of screens packaged up in a nice bundle I refer to as a "UX Kit" - which basically consists of everything needed for a developer to pick up and implement a design.

Notice that this approach offers two benefits:

The first is that it helps the project focus on tangible results, which keeps things from meandering without a specific outcome. That's not to say that meandering doesn't have its place - just not under this initiative.

The second is that it uncouples design from any dependency on implementation, and vice-versa. This way design can take place on its own schedule and with its own priorities within the context of two-month projects, so long as what's produced can be implemented. This gives UX folks a bit of autonomy in selecting projects and reduces the chance of being bogged down by indecision or delays that aren't directly related to design. In other words, designers can design what they think "should be" without worrying about how and if it will get implemented. If the design has merit, developers can pick it up and implement it at their leisure.

This shifts the emphasis from "code currency" to "quality currency".

Is two months enough time for quality research and design?

While the current project will result in "implementation ready" screens, not all projects under the initiative are required to. Some projects might result in inputs that will be used in other near-term design projects. For example, a project might be research oriented in that it will produce plans, definitions, and tools, but unlike experimental research, the materials produced in this project will be extremely focused.

All deliverables under this initiative should share the characteristics of being relevant to the current state of Sakai, prepared with the intention of being used in a near-term project, and developed to a level of completion that will enable them to be immediately put into practice.

Hopefully by now the focus of the initiative is becoming a bit clearer. The goal is to improve Sakai, not think up something new.

With that in mind, there may likely be a future UX initiative based on thinking beyond the current paradigm. In this future initiative, the projects will consist of more radical and forward thinking R&D that will likely take place over longer intervals with lower rates of screen production than the current initiative. This is marked by the yellow graph below.

The current project

The blue graph represents the current UX Improvement Initiative - with the star icon obviously estimating how far along we are (about ¾ of the way through the first project).

As you see by the angle of the blue graph, the rate of production is quite high. This is explained by the "catch up" effect. Much of today's design work is benefiting from the years of research and discussion that have already taken place. Since we don't need to start from scratch to improve what's already there, we get a nice bump in efficiency.

In terms for what's actually being designed, as you may already know, the current project is mostly concerned with "site setup" and other rudimentary parts of the UX - as opposed to some of the meatier academic aspects of Sakai (that will come later).

That's not to suggest that the work is entirely dry. While rearranging the existing design to flow smoothly, or what I refer to as "plumbing", I periodically toss in a few fresh ideas based on inputs I receive from others, as well as my own thoughts of course. But mostly, the brunt of the effort is plumbing.

Also, since many of the areas touched in the current project are fairly common to other enterprise applications, for example: managing users, setting permissions, changing styles/colors, etc., we can cheat a little by taking advantage of best practices and conventions in place of intensive up-front research. While this doesn't negate the need for usability testing, it does add another bump in efficiency in that we're not dealing with something entirely new and foreign.

Why "site setup"?

Some of you may be wondering why the site setup feature was selected for this first project. Without going too far into the history of how the UX Initiative began, I'll just say that site setup was an area that was discussed early on as an annoyance in the context of new users (in this case, users who have the ability to set up sites) trying to understand Sakai. It was clear from the feedback that this area of Sakai created an unnecessary barrier to those users.

So fixing it made a lot sense from a product perspective.

From a design perspective, I took a close look at how else we might cut into redesigning Sakai and found myself agreeing with "site setup" as an ideal starting point. For one thing, before the site setup process can be redesigned, we'd need to first understand what the resulting site should look like. This line of thinking was fairly attractive in that it removed some of the historic constraints that would sometimes limit a project like this.

Secondly, the site setup process makes an attractive design target since it touches a lot of different areas of Sakai while at the same time acting as a bit of an "add-on" feature – which allows the design effort to go as light or heavy as necessary without committing to more than anyone can handle.

Choosing the first initiative first

If you return to the graphs for a moment, you'll notice that the yellow graph intersects the blue graph where the former picks up and the latter winds down. This yellow graph represents a possible future initiative that will shift the design emphasis from mere improvement to broader and more philosophical questions about Sakai.

Seems like an okay plan, but some of you might be wondering why we don't just start with the yellow graph first and tackle design at a deeper and more rewarding level now in place of the cleanup work that's happening in the first initiative?

For one thing, not all institutions are prepared to accept a quantum leap with respects to changing Sakai. That's not to suggest the result of a deeper design process would force anyone to give up what they currently have, but let's face it, we'd be treading a path less travelled, so all possibilities would be on the table.

Secondly, Sakai needs help now, not two years from now. There's a lot of improvement that can be done in fairly short order with significant gains. Also, circumstantial innovation will undoubtedly be a byproduct of this effort, resulting in a substantially more robust solution than just incremental cleanup work.

Further, any plans for a long-term research project should be considered with caution in this industry since we're often dealing with a rather quickly moving target.

And last but not least, the process of working together as a team with a design lead is new to this community and taking on a theoretically less ambitious effort now will yield huge benefits later on in terms of developing a good team dynamic and building confidence based on our past successes.

Knowing when Sakai has been "improved"

I have a saying that goes: "The more we talk the less we see." It's not a riddle, I mean it quite literally. I firmly believe that regardless of where one is in the product development process, sketching ideas on paper and producing screens to visually support the discussion is invaluable! So the more screens and the faster we can produce them the better off the process.

At some point however screen production will slow. Some of the design work will register with users, and that's what we'll keep. The stuff that doesn't will fall by the wayside. Negative user feedback will be replaced by positive user feedback (or silence). And just as the graphs indicate, the process will reach a lull.

The best way I can describe the max point in the blue graph above is that we'll know we're there in much the same way we came to realize that we have problems in the current product. It'll be fairly obvious.

If that doesn't satisfy you and you're looking for something more concrete, you may want to check out the UX Scorecard I developed. It's basically a set of metrics I've used throughout my career that have proven to give some indication of quality. None of this is six-sigma, but it is a good start if we allow ourselves to put it into practice.

Where we go from here

While all of this is interesting (and hopefully exciting), the long-run success of trying to make a great product requires a shift in world view from being an observer of UX work to being a participant of it. Making lasting and meaningful UX change is by no means a small undertaking. The work is vast, cuts across all areas of the product, and is a feature of our development landscape that requires continuous attention.

I'm thrilled that as a community, Sakai has taken on this agenda, and to keep it moving we need usability folks, designers, developers, and anyone else who wants to see Sakai succeed in this area to get "actively" involved.

Summary

So that's it. That's the 50,000 foot view (oops... 15,240 meter view for many of you).

Today the focus is on tackling the low hanging fruit, cleaning up what we have, cranking out screens, and learning from our mistakes along the way.

Once we get to a point where Sakai's UX has improved a bit, we'll shift our focus from "usable" to "useful" and address the bigger strategic questions.

See how sites are managed using the Sites widget and tool.

Topics covered in this design include:

• Searching for sites
• Joining sites
• Editing or managing "my sites"
• Two alternative concepts for sites searching and joining
• Understanding some fundamental interaction design patterns (hub model)
• Seeing the relationship between The Site, Widgets, and Tools

To learn more, take a look at this week's flash presentation.

Companion screens below:

Not sure why the gallery function displays the images in a backwards order, but I numbered them in order of how they're presented in the video. Hope this helps.

10	9	8
7	6	5
4	3	2
1