Join AJ Davis, founder of Experiment Zone, on this engaging podcast as she breaks down results from an extremely effective (and simple!) category listing page A/B test. Discover practical tips and strategies for maximizing the impact of your narrative.

AJ Davis goes on to outline how to think about the role of each page on your site, where to add messaging about what makes your brand unique, and how to get that messaging right. Tune in now to gain valuable knowledge about where and when to tell your story.

Podcast Highlights

0:20 Intro
1:22 About the brand, and the category page experiment
3:53 Your default template might not be ideal
6:17 Different ways to explain your price point
7:39 Think of your ecommerce website like a physical store
8:58 Remember humans have limited working memory when explaining your value
12:11 How to test your assumptions about what your value is
13:17 Where on the site to test value messaging
15:45 The surprising part of this experiment
17:45 A 30% increase in orders
20:43 How we knew this was worth testing
22:52 An example of why you shouldn’t just go with intuition or best practices
25:30 What to do if you can’t run experiments
26:43 Where to find AJ && a special offer

Transcript

AJ Davis 0:00

Ultimately, at the end of the day, we increased orders for people who visited that page by over 30%. So people were not there to read, they were there to shop, they were there to compare. And so if they get to that page, we need to keep them focused on finding the products, rather than teaching them a broader message.

Brian 0:18

Hey, oh, that’s AJ Davis from Experimentzone.com. In this interview, she breaks down a specific experiment on a shop’s category page that I definitely think is going to be relevant to a lot of people out there, especially if you’re still on the default theme that your site started with. But beyond that, she gets into at a high level, What’s the role of your category page in the first place? How should you think about that versus your homepage? Product pages? And furthermore, to the extent that you do talk about yourself on your website, where does that messaging go? How do you get that messaging? What tests do you run to find this out? There are answers to all this inside the interview. So I’m going to stop talking and let you enjoy AJ, Welcome to the show!

[Thanks for having me on again, Brian.]

Yeah, no pleasure, I’m very excited, and intrigued as well, because actually, I don’t really know what we’re about to talk about. This will be a surprise experiment, which may be as it should be. So let’s just jump right in and tell me you’re going to tell us a story about an e-commerce experiment.

AJ Davis 1:21

We work with a company that sells specialized exercise clothing for women. And so there’s kind of a hypothesis out there and from the business that people need to know more about what the brand is, why you want us to buy the specialized product instead of something at your local store, and how they’re different. There’s this natural give and take that we end up with of how much we need to teach people. And where should we teach them? Versus how much we let them just see the product and get into the shopping. Generally, a thing I see in a lot of clients is that they want to talk about the business, they want to educate, and they want people to feel confident, and that is really important but this also can get in the way of a purchase.

The experiment I’m thinking about today is for this client, they had this block of introductory text at the top of their category page. It’s just a product listing page with lots of images, people can filter and choose which products are best for them and they had an image and a block of text that was descriptive to provide some educational content. We ran a test where we collapsed that, but we gave people the option to click “learn more.” So if people were there with that learning mindset, they still had access to all that content. But it really focused the page on the specific product thumbnails and people didn’t really care about “learn more.” Ultimately, at the end of the day, we increase orders for people who visited that page by over 30%. People were not there to read, they were there to shop, they were there to compare. If they got to that page, we really need to keep them focused on how to find the products, rather than teaching them a broader message.

Brian 3:06

Okay, all right, there’s so much here that I want to talk about. One thing is, I always love these stories, removing stuff, hiding stuff, shrinking stuff, which actually leads to more revenue for the store. I think that’s such an important story to tell. Obviously, not always, you can have a blank website, but I hear this so much that just taking stuff away can be a win. And it’s like an easy test to implement just a little CSS to hide the thing, in this case, it sounds like he did a little more. But I guess I want to pull on that a little bit, going into this really thinking about that as a crucial component, just the sheer visual impact of nudging stuff up the page. Like is this been a thing? Have you seen this elsewhere, too?

AJ Davis 3:51

Yeah, we have seen it elsewhere. Oftentimes, a lot of these Shopify templates have huge blocks of imagery at the very top of many pages, the homepage, and category pages. And so we weren’t sure if this text was actually providing value, or if it was doing what the images often do, which is just make it a little harder, add some friction to people seeing the purpose of the page generally. We just wrapped up a test last week on a client, where we shrunk down the top of the page image on the category listing pages to be about a third of the size and we saw similar lists.

This is a little different than the test that we’re often running because there’s a different design, different purpose, and a stronger underlying hypothesis of the purpose this content was providing. But in general, when people get to a listing page, they’re there to see your products. They want to understand what you have, and seeing does so much more than telling. And oftentimes, those lifestyle images that show up on the top aren’t really assisting them in making a choice.

Brian 4:55

So the theme I’m hearing is that Shopify stores are typical or whatever template you’ve started with. I think there’s a tendency to think well, what do we put here? What do we put there? We have this block, what should go in it, we have this text, what text should we insert? But stepping back a pace, Is this even necessary? What purpose does this serve? And does that match with visitor intention? It leads you to a non-standard template.

AJ Davis 5:22

A challenge is sort of the status quo, right? A lot of the companies that are producing the templates are saying these are tested and true, these are best for e-commerce, and they’re good starting places, they certainly provide a good foundation. But a lot of the time there are things that need to be simplified, repurposed, or relocated so that it’s at the right moment in the customer journey. Because at the end of the day, the customers coming to your site need different things than everybody, right? There are specific needs and specific things that they’re doing, because of the design you have, because of the products you have, because of the qualities those people have.

Brian 6:02

Yeah, and actually, you mentioned, this particular site was a specialized workout clothing. Did I remember that? Right? Yeah. Can you say a little bit more about “specialized?” How was it different?

AJ Davis 6:16

Yeah, it’s a high-end, very specialized fabric. So it’s for women and it’s designed to be comfortable, functional, and stylish while you’re exercising. And there’s education that you need to have to understand why this price point. Why is this more of a luxury good? What is it about the material? What is it about the design that sets this apart? The product itself can show that the product pages can be where we teach that, the homepage is a good place to introduce it. Stepping away from this particular test, badges, and a very quick bulleted list, here are the three things that you reiterate across the site that can be really helpful. But generally, people aren’t interested in reading paragraphs about your business. [Sure] If they are, they’ll go to your About page which can be a home for it if people want to know more of a story.

Brian 7:08

Yeah, that makes sense. I like how you framed it as the category pages in between places where you’re on a mission, like the homepage. You might just bounce, you might be like, “Not today,” or “Not this product, not for me.” Product pages, maybe a little more intensive research. I’m setting this up because I want to hear your take but on the category page, nobody’s gonna read it. Nobody needs all that background when they’ve made it this far. Is that a fair restatement of how you’re looking at this?

AJ Davis 7:36

Yeah, I see it as I think of a good analogy, I love to just take it to a physical store. When you are driving along in your car, you might look at a storefront, like a homepage. And if you’ve never heard of the store before, you’re not likely to stop. So you do need sort of nudges, things that are reassuring on the outside of the store. It’s a clean storefront, the signage is clear, and I understand what they’re selling. But then as soon as you walk into the store, you don’t want someone standing next to each product and saying, “Here’s a bag of toilet paper, that can be purchased here” Because of this reason, people want to just see it, feel it and understand the purpose. And they in their own minds are already understanding what the store offers. So to me, it’s analogous to you asking for help and digging into more information about the product, which is the product detail page. But when you’re walking up and down the aisles, figuring out which things you want to put in your cart, that’s the listing page.

Brian 8:32

That’s a great comparison, I’m going to use that, I appreciate it. You mentioned the homepage in the product page detail page as being two places to maybe go more into that. And I think I heard you say bullet points, just the few bullet points, condense it down, don’t write paragraphs, and repeat them throughout the site. I wonder if you could say more about that approach, where do you put this stuff? And how much of it is too much?

AJ Davis 8:57

The underlying philosophy is that we can only retain five to seven bits of information, right, each person has only so much working memory. And on the low end, it’s really three pieces of information on the high end ten. So we don’t ever want to go outside of that band. But we want to push ourselves to adhere to the three pieces of information, I usually recommend for my clients that if we can narrow it to three to five, we’re going to be in better shape than being at like seven to ten because some people are going to lose that and it’ll be too much to process. So we’re taking advantage of working memory.

We also want to know that people are going to get distracted or be thinking about different things as they move through the site. So maybe a huge value for your business. Let’s take an example like the Amazon Today shipping, right? Everyone knows about it. But if you go to their site, it’s everywhere on the site, you feel confident on the product page and your cart and checkout that your products are going to show up, even though that’s such a huge part of their brand.

So if you’re a brand that someone’s less familiar with, and you have a value proposition, like free returns, you’ve got to tell them initially. So they get excited about it and say, Hey, this is a cool brand, and they have my best interest at heart. But then you also want to put it in the position of influencing their decisions. So you go to the product page, and they’ve forgotten about your really great returns policy, so you can reiterate it again. And oftentimes, you want to put it near the action, so that just under the add to cart buttons, a good place for it, and then you can leave it in the cart and checkout. But at the end of the day, all of this is just theoretical until you really run an A/B test on it.

You can test where the placement is, and which messages really resonate with your customers if the way you describe your value propositions is resonating with people. You can even test out “Should I be offering a really great returns policy,” let’s put it on the site. See if people get excited about it, give it to that cohort, or have it run for a month or two for everyone, but only show it to half of the people and just see how different it is. So you can test these bigger business strategic questions as A/B tests as well.

Brian 11:06

Okay, this is what we’re getting outside the scope of the one experiment you came to talk about. But I think this is so important. I think this is really hard. We’ve got this whole story, this whole saga, this brand, epic drama of how we got here and what we do and why it’s special. And what I’m taking away so far from what you’ve said, is really on the category page, it most likely nobody wants to hear that. At that moment. It won’t stop me when I’m browsing the aisles, I’m just trying to look around. Please, you can put it all on the About page. If people really want it, they can find it there. And that’s fair game. But this challenge is to distill it down to a handful, three to five optimized bullet points. So first, we have to distill it down. And then we have to find ways to experiment to decide which ones actually matter and which ones actually change people’s behavior. Like how do you go about this? From the top? How do you get to the bullet points? Do you find clients typically know? Or do you supplement their own perspective with other research?

AJ Davis 12:07

I think a lot of times clients have assumptions about what they are. So you can start by testing what assumptions exist for the business in an ideal world, you would go out and conduct user interviews. So you would talk to people who’ve recently purchased on the site, people who are dedicated to this business, and talk to them about things like, Why do you care about this business? What does it mean to you? What are the most important things they offer? And then you can reflect back on that language in the copy on the site. So you might hear things about the return policy that we just talked about or maybe they don’t even mention it. And so that indicates that’s probably not the place to start?

Brian 12:45

Yeah, okay. I’ve just put you on the spot here. Actually, suppose we have gotten down to where we’re pretty sure we got a list of five. And maybe we’d like to know which three are the ones to really push. Do you have a cross-sight? Is there the first place you would go to try and test that? I guess it depends on where people are entering the site. But do go to pages and elements where it’s a good fit for trying to learn that kind of information from the experiment. Which value proposition? Which bullet points?

AJ Davis 13:16

Yeah, I think that are two places I like to test it: Initially would be on the homepage under the hero, “Hey, you’ve already told them what you sell,” so if they stick around, they’re hooked to it. And then having a very short banner. So it’s not, it’s just one icon tall so that it’s not blocking them from scrolling the page or seeing other elements. But that’s a good place to say, “We offer free shipping, and it’s made in the US and you get free returns.” Those are all very customer-centric things too. I would actually back up a little bit and say, one of the mistakes I see is that brands like to talk about”we do this, we have this” and customers may care about that. But if you can switch it into words that matter to them, you’re going to be more successful, because they care about the impact on them. They don’t care that you’re going to ship it out in two days, they’re going to care that they get it in five days. So thinking about it from that perspective of what they need to hear it’s important to them.

The second place that I would put it in addition to the homepage, if I were just running off the cuff, a test for this would be under the Add to Cart button on the product page. Because those types of values are going to help nudge them to go, Okay, I like this product, but do I feel competent about this business? Okay, I see that they do have some of these values that matter to me. And so that can help nudge people to make that commitment to add it to the cart.

Brian 14:39

Okay, that’s good. I think that’s really great, like helpful tactical advice. If anyone is listening, watching, and they realize, “Oh!, This is me,” I’ve got this long bloated story about how amazing my brand is on my category page. That’s something to look at, and maybe kind of company-centric versus value to the customer-centric Language. These are places you can go, and you can try to distill your bullet points. That’s the hard part but then these are places where typically you’re going to have enough impact on the experience, I guess that you’ll be able to measure the effect of swapping out and landing, whatever is optimal. So super helpful, I guess I want to ask, I know you do this stuff all the time, and you’ve run tons of experiments. And so it sounds like this maybe wasn’t a shocking result to you at this point. But I still want to ask it, this particular experiment on the category page with shrinking the hero, and learning that nobody cared. And if anything, it helps to take that stuff away. Did this change or influence how you approach what you do moving forward?

AJ Davis 15:44

I think the surprise for me in this was that people didn’t really interact with the Learn More. So we had a hypothesis that some portion of people would maybe not be certain and want to read about the specific features of the “leggings” that they were looking at, for example, and at the end of the day, like the fact that I think it was less than one and a half percent of people interacted with it. And then that segment didn’t have any differences in their order outcome just indicating that this wasn’t the right place to have this type of content.

So that was surprising to me because this product line was so specialized and I also thought that there was someplace that would be more important, or some content that might be more important on this page to just give them confidence. So yeah, you’re seeing the prices for the first time, they may be more expensive than you might have expected but here’s why. And if people didn’t need that, as they had already come to terms with, “Hey, this is a more premium, more luxury product.”

Brian 16:44

That’s yeah, that’s a really good point. That’s another unique aspect of the category page, what you said about seeing the prices for the first time, again, similar to browsing the aisles, right, you didn’t know the prices until you walked into the store and started walking around. So I guess what I’m getting is, as a general rule, I don’t want to put words in your mouth. But maybe you would say that a category page doesn’t need a whole bunch of context and that it’s better to let people focus on the products. But in this case, given the specialized nature, given maybe the price point, maybe for this one, maybe for this kind of product, [it is necessary,

AJ Davis 17:20

or at least for a segment of customers, right?] So we expected there would be some portion of customers who would mean that even if most people it was in their way. So that’s the hypothesis we walked in with. And it just turned out overall, it didn’t have an impact on anyone. It was just not a very good place for it.

Brian 17:36

Yeah, let’s dig in for a second on the analytics. I think I heard you say, a 30% increase in budget and orders.

AJ Davis 17:45

Yeah, we saw a 30% increase in orders for people who saw this page. That is

Brian 17:49

huge. I guess, if you’re listening, and watching at home, don’t expect to get a 30% increase in orders on your first experiment at the gates, please. And don’t expect that of your CRO consultants either. But that’s massive. And it’s not one of these up-funnel metrics click through to the PDP or something that may or may not result like the orders. So that’s massive. I think I heard you mentioned you also segmented out the results to look at the handful of people who did interact with this little learn more element that you put in place of all that text. Can you say just a little bit more about how you approach that?

AJ Davis 18:27

Yeah, I guess the one thing I stepped back on is the fact that it’s a 30% increase shows how problematic this was. So like most of the time, we’re just making small improvements, because there’s just a little bit of friction in the templates guiding the website to convert pretty well. And then there’s an opportunity to get 5% lifts, and 15% lifts here and there, we’re always excited to see a 1% lift too. So the range of the lifts is more of a reflection of how important this issue was for your customers. And so for this company, it really was like a huge problem that people came to the page and no products were above the fold. The result of having all that content there was people would get to a listing page, and there would be no products on the listing page. If we think about why that might have happened. It’s likely that people saw that page and thought they got to the wrong place, they thought they were going to see a list of leggings, but they just didn’t see any leggings. So they got confused. They got a bunch of friction in their experience. And then they gave up and well it’s a huge win. And we’re really excited about the win. It actually points to what a pain this experience had been. And we wouldn’t have known that without testing it. So the amount of lift was also a big surprise.

You asked a secondary question about segmenting and sometimes we’ll look at segments based on what was the impact on mobile versus desktop. Does it really matter? So we often look at multiple segments and a test to see if there is one for this one, perhaps like new versus returning can be interesting, right? You might expect new customers and new visitors to your site. Really need that added context. Sure, they also were very consistent in the results. So we didn’t see anything significantly different in any of the segments we looked at, including just that small segment of people that did interact with more and more.

Brian 20:12

Now we know after the fact with the data all in the experiment conclude, we know that this is actually a huge issue for this site that this like a 30% lift is not to be found just anywhere you go. What made you decide to focus on this page in the first place? How did you decide this was an experiment to run versus any other thing you might have done?

AJ Davis 20:34

Yeah, so across all of our testing, right, we have limited resources, we have limited time, we have limited traffic. We always have to prioritize really carefully. So I would love to tell you that we ran a user study and saw this issue, but it was so glaring in the analytics data that we knew we had to start here. And what I mean by that is, this is the page that had the biggest drop off on their site. And it was just extraordinarily high compared to other kinds of heuristically how our other clients behave and how their sites are organized. So we just saw a huge amount of drop off, we knew something was wrong and took a look at it from a heuristics perspective. And knowing from those past test feeds we’re on where we move products higher up on the page, just seemed like the right hypothesis to start with. And then that little bit of nuance with adding the “Learn More” was really driven by the assumption that customers needed more information at every stage along the way for this type of product, which turned out not to be the case.

Brian 21:30

And so we’re talking like in Google Analytics, like exit percent, right, like exit rate for the page. So you look at the homepage, you look at the category pages, as a whole as a collection of pages. You can look at product pages and the rest of the funnel too. And so you did this and just broadly speaking really high, like what’s going on here? Exit rate on the category pages?

AJ Davis 21:53

Yeah, exactly. It’s really like the enhanced ecommerce view, really, it just was glaring, that there was an opportunity here.

Brian 22:02

Cool, obviously, great fix and kind of aligns with my priors that say, where you can remove stuff that isn’t absolutely necessary. It’s usually a good idea. I guess, I want to ask, what’s your read? What’s your comfort level with somebody just saying, Oh, my gosh, this is me, I have this huge hero section on my category pages. It’s just a bunch of useless information. I should just kill it. Is that okay to have the official AJ stamp of approval for that strategy?

AJ Davis 22:28

This is why I love testing, I came into this world having worked in user research before, so I got to observe a lot of people and the issues they come across, and we see a lot of the same kinds of issues. I have a pretty good intuition for what’s going to be needed or what’s going to be a challenge based on seeing so many different people, over hundreds of thousands, if not hundreds of thousands of studies where I feel like I have a good intuition for this stuff. So I’ll actually give you an example because the reason I love testing is you can build up that great intuition, it can work a lot of the time. But if it doesn’t work, it really can hurt your revenue at the end of the day. If you make the wrong choice, and you haven’t measured it, you don’t know it’s a problem, you don’t know that it’s causing you to lose customers. So an example of something that I used to tell everyone was like, “Hey, if you have free shipping, let people know, put it in a banner at the top,” Every test we’ve ever run that has been helpful has a persistent message about what your shipping policy is.

Very recently, we had a customer for whom this was the first test we ran, and the reason we ran it is that we wanted to make sure their conversion rate got hurt by having this on the site. It was like a 20% drop which is very similar, just a big surprise, because we normally will see it help. And with that, and another test that we’ve run around their shipping policy, we have some indication that is actually just like the price point doesn’t make sense. So the price to get free shipping doesn’t work for their customer set. And by increasing the prominence of it, we frustrated more people, and they were like, “I’m never gonna spend that much on the site, I’m taking off,” it seems on face value, like a really good thing to do. But at the end of the day, by testing it, we A; know that it worked or didn’t and B; if it didn’t work. We’re gonna have a much more valuable insight, which there’s something underlying why it didn’t work for this audience for this business when it worked for everyone else. So let’s dig into that.

Brian 24:27

So back to your decision to test on the site that we started off talking about to make a point of testing and add the learn more on the category page specifically because “this is a very specialized product, and it is more expensive than people might expect” that decision. You still stand by the reasoning that says that the site may be out there that actually needs a pretty big text, heavy hero section on the category pages as it could exist. And I guess it’s a big enough change. That’s a risk that you don’t want to just flip a coin on, you want to be careful to gather the data.

AJ Davis 25:05

Yeah, at the end of the day your customers are going to respond to changes you make, we see huge changes from tiny changes, huge changes in behavior from very small iterations on your site. If you don’t have the data of what’s gonna happen when you make that tweak, you don’t know what the impact is. And if you do that over and over again, you may not be in a better place at the end of it. So if you see that this is a problem, I would dig into the data, If you don’t have enough traffic to actually run an A/B test, which is like the best-case scenario, run the A/B test, you can look at other signals. So you can use scroll mapping to see if people are getting to the products on your page? Are they clicking the products on the product listing page? Are they interacting with that hero if there’s some sort of interactive element, so you may be able to triangulate different data sources or different signals in the data to say, All right, I should do this for my business, too? But at the end of the day, the only way you can be confident is if you run an A/B test on it.

Brian 26:01

Thanks so much. I really appreciate you sharing this, I think we got a lot of takeaways and a little bit of a word of caution on how to proceed. But just to recap, distilling your story down to a handful of bullet points, I think there are a lot of stores out there that need to do that hard work, and then test to figure out to make sure you got the wording right. And the placement of them and the prioritization of them to fuse, and that’s something somebody can go start on, then go start thinking about that right now. If you’re listening, and watching, please go start thinking about it right now, when it comes down to the actual test implementation and strategy and stuff. Maybe you need a little help. Maybe you just have some questions. Can you tell us where people can find you on the interwebs? AJ

AJ Davis 26:43

You can find me at experimentzone.com. I’ll also provide you, Brian, with a link to the free consult we’re offering right now, a 30-minute consult, you’ll really get a chance to get some of that feedback I can provide from that more intuitive “hey, what we’ve seen in the 1000s of tests we’ve run and research we’ve done” but also we might be able to direct you on “is this a good time to start testing?” “Where should you start?” We can cover a lot of ground in 30 minutes, and I would love to offer that to your audience.

Brian 27:10

Yeah. Love it. That link will be featured prominently in the description show notes. We’ve covered a lot of ground here in about 30 minutes and you’ve alluded to so much more. I’ve known AJ for a while there’s a lot that she can say. You have all these other studies, experiments, and benchmarks that we mentioned, like “What is the troublesome drop-off rate on my category page,” if you want to know you can talk to AJ about that. So I really appreciate it. Thanks for coming on.

AJ Davis 27:37

It was great chatting with you again, Brian. Thanks for having me on.