There are two things you should know about this post before you go further.
First, I’d like your help. Please thoughtfully disagree with this approach and tell me why. References to support your argument are greatly appreciated. Why am I asking you to disagree? It helps me learn. This TED talk explains.
Second, I don’t consider myself a true Information Architect (IA). I never have and I doubt I ever will. Perhaps a novice IA at most. That is because I have had the privilege of working with two REAL Information Architects over my career.
What separates us?
Hmmm…advanced Library Science degrees, a passion for metadata, taxonomies, ontologies, search algorithms, and words in general.
I have a healthy respect and understanding of all these things, but I prefer to leave this to Art and Science to others.
Louis Rosenfeld came close to changing my mind about staying a novice when he published Search Analytics for your Site in 2011. It wasn’t a book review or a recommendation from a colleague that attracted me to the title. It was the illustrations from the book that he posted on Flickr that got my attention.
This elaborately color coded one was especially interesting to me because even without reading the book I could tell this chart was trying to demonstrate the seasonality of search queries at Michigan State University.
(By the way, we bought the book for our office and love it, Lou. You should buy it!)
My colleague, Jessica Rasmussen (@jessicapullen), and I were just starting the largest study we had ever been asked to conduct for one of our business units. They asked us to review the online content about their annual meeting (an international event that attracts upwards of 10,000 people) and make recommendations about what, if anything, they could do to improve it for their audiences.
I wondered if something similar to the Michigan State picture could help our client establish an editorial calendar.
We began using Google Analytics to monitor the top keywords and phrases that were driving traffic to the site. The amount of content on the site posed a big challenge almost immediately. The annual meeting was just a small section within a much, MUCH larger site.
I had to look well beyond the top 100 keywords and phrases to find terms that were most likely to be associated with the annual meeting. It was like finding a needle in a haystack, but I had no way of knowing if it was because of the time of year, or something else.
I decided to look through the top 1,000 keywords and phrases every month. Was that a bad decision?
The annual meeting terms ended up being a tiny percentage of the total queries in the sample each month. They were part of the “long tail” that Gerry McGovern might suggest you ignore. But I argued we should stick with it for two reasons:
- We knew the annual meeting content served a very important audience for the organization.
- They asked us to make recommendations on how they could to improve THIS stuff, not the “long neck” stuff.
So I plowed on.
One year later we did end up with a good picture of seasonality. We saw keyword spikes that lined up with annual meeting events (e.g., registration opens, deadline for the call for papers, etc.). I couldn’t wait to show the client, but I needed a deliverable that they could wrap their brains around. As much as I liked the Michigan State example, I thought it was overwhelming and you couldn’t get the gist at a glance.
My solution was to categorize the keywords and phrases into broader buckets so the client could see the themes at a glance. I made the deliverable look like a heat map using Excel’s conditional formatting color scales so the client could see the seasonality. I also included a detailed view that allowed our client to drill down into the juicy bits. Here’s a portion of the deliverable showing the broad themes and a drill down section:
[Click Image to Enlarge] A portion of the deliverable showing seasonal search queries. I grouped terms into broad categories so the client could see the trends at a glance and also included a way to drill down into the specific keywords and phrases.
Based on this work we could have put together an editorial calendar. There was just one problem. It would have been wrong.
We knew it would have been wrong because of all the other research we had been doing for a year.
You see, we got to know the business unit deeply. Jessica conducted extensive audience research at their annual meeting, an intern did a content inventory of their site, we facilitated staff discussions to identify the most frequently asked questions they received, and Jessica monitored a small portion of their support mailboxes to see the questions submitted in the audiences’ own words. We knew the analytics work was good, but we knew it did not tell a complete story. By definition, it couldn’t.
Remember that a Google Analytics report only shows you the keywords and phrases that were successfully driving traffic to your site in the past. Assuming they continue to drive traffic to your site, you can think of those words as “working”.
I put quotes around working because you still have to evaluate all the other stuff that Kristina Halvorson, Gerry, Lou, and many others teach, like:
- Is the content that gets served up what the audience wants and needs?
- Is it accurate, complete, and written in a way the audience can understand?
- And on and on it goes…
So what were we going to do? Give them an incomplete editorial calendar?
We talked to the client and told them the truth: they needed an editorial calendar, but first they needed to focus their limited resources on surfacing content that wasn’t being found and writing new content that answered the audiences’ most common questions.
(Wanted: Talented writers to perform pro bono work.)
They did not have a big chunk of money set aside for someone with “writing for the web” expertise to work on this full-time. No, it’s the dedication of passionate staff who will do their very best to improve the content in addition to performing their day job. To really help our client, they needed to know what content they should focus on improving now, next, and later. That was where we could make an impact.
Earlier I mentioned our staff sessions to discuss the most frequently asked questions and Jessica’s mailbox monitoring efforts. The intention was to find common questions, or themes of questions, between what we heard from staff and what we heard from the audience. If a theme was popular to both staff and the audience, our client should evaluate the content associated with that theme now.
Yet, I didn’t want to lose the value of our search engine analytics work. So I thought, why not throw the findings into the mix and use the data to help us prioritize further?
For a given theme I asked:
- Did staff mention this question or theme?
- Did we see the same question or theme in the email monitoring work?
- Did we see keywords or phrases related to the question or theme throughout the year?
If the answer was yes to all of these questions, I assigned the priority “NEXT”.
If the answer was yes to the first two questions, but I did NOT see keywords or phrases related to the theme, I gave it a “NOW” priority.
Shouldn’t it be the other way around? If you can answer, ‘yes’ to all three questions, shouldn’t that be the highest priority?
My argument is that if I’m NOT seeing keywords and phrases in the Google Analytics report it suggests:
- People might be trying to find content associated with the question or theme, but they aren’t ending up on the website.
- People might NOT be trying to find the content associated with the question or theme, but they are contacting staff for the answers, so the desire for that information exists.
This method allowed us to ruthlessly prioritize the questions and themes they should investigate now, next, and later, but our client still needed to know WHAT they should do during that investigation. We suggested using a simple do-it map to help them make decisions.
Similar to the method before, ask questions:
- Is the answer to the question or theme on the site?
- Should the answer to the question or theme be on the site?
I hope this evaluation process will allow them to quantify the amount of optimization and writing they need to accomplish. Knowing the scope of the work that needs to be done may help them justify additional funding and decide whether to use in-house resources or supplement the work with consultants.
If Lou, Gerry, and Kristina had a baby…
The work of Louis Rosenfeld, Gerry McGovern, and Kristina Halvorson influenced how I thought through this, but I’m not convinced this method is worth repeating. I’d like your thoughts.
What I like about this method is that it’s mostly repeatable. I loved discovering the seasonality of the keywords and phrases. I think my heat map is pretty cool. This journey reaffirmed my strong feeling that analytics in isolation and user research in isolation don’t give you the fullest picture. We got a fuller picture because we took the time to blend both kinds of research, but I know we don’t always have that luxury.
What concerns me about this method is how long it took to arrive here. Maybe it’s because I was figuring this out as I went along, but this method doesn’t feel “good”. You could combine any number of other methods to arrive at a similar conclusion and they may be faster and cheaper. I’m concerned this method falls into the insidious category, “Just because you CAN, doesn’t mean you SHOULD.” Do you agree?
So if Lou, Gerry, and Kristina had a baby, I think it might look really ugly in the beginning. But perhaps with your help and some more experimentation, this ugly duckling can turn into a beautiful swan?