Experimenting with the landing page’s showcase design at SVT Play
If you re-design a product due to time constraints and a new flashy feature you want to showcase, how do you know your users actually enjoy the redesign? More importantly, are they getting the most use out of it compared to the old design, or compared to any of the others designs that weren’t developed?
This was the problem background presented to me when I worked with the SVT Play TV-app team for 8 weeks in October and November of 2021. I was working on the module Experimentation in my UX-design course and the Apple-TV team at SVT were in a period of development where they wanted to focus on experimentation for the first time as well; a perfect match!
Background
Swedish Television (SVT) is Sweden’s public service broadcasting company and SVT Play is their streaming platform, which just so happened to turn fifteen years old at the end of 2021. The company is striving to be a data driven company and during my almost six years at SVT’s analysis department they have come a long way. I’ve been working there as a digital analyst for the children’s department since May 2016, but got this opportunity to try out working with UX for SVT Play on the Apple-TV platform.
The Apple-TV team
The challenge
At the beginning of 2021 the Apple-TV platform for SVT Play had undergone a large code transformation. Around the same time a project had just wrapped up in which SVT Play implemented a process for designing key art that was adapted to the streaming environment.
The Apple-TV team decided that upon launching the new version of the app, they would redesign the top of the landing page, what’s known as the showcase, in order to put the new, visually enticing key art into focus. A design was produced and the new version was launched. But what did the users think? How did this design compare to the old one? And was it really optimal? Since no A/B test had been performed, we couldn’t really know. Around nine months had since passed since the launch, and now it was time to get an answer.
Method
User research
The SVT Play team had during the spring of 2021 done an extensive survey study on the goals, pains and gains of our users. Through this study, as well as an exploration of user sessions in our web analytics data, three user, or rather use segments were identified, of which one user could fall into any of the segments at different times. They were:
- The fetcher: I know what I want to watch right now.
- The explorer: I know that I want to watch something right now, but I’m not sure what.
- The planner: I want to find something that I can watch later.
An overall goal for SVT Play has been to convert seldom users to loyal users. By combining frequency groups (ie seldom, now and then, weekly or daily users) with the above named segments, as well as the main pain points and user intentions discovered in the survey together with prioritized metrics covering effectiveness, efficiency and/or satisfaction, a framework was designed which could be used when setting up experiments.
The hypothesis
Using this framework the following hypothesis was produced for our experiment with the SVT Play showcase redesign:
We believe that if we use visually enticing key art and descriptive introductory texts about the content
then users who explore and users who want to see specific, current content
will experience that it is easier to find a good program
which will reduce the time to video start, increase the proportion of video sessions and generate more video starts from the showcase window.
Background research
I found inspiration for the new designs in obvious places: from sleek and uncluttered streaming service landing pages like Hulu on the right, to YouTube with a little bit of everything strewn about on the page. There were also surprising sources like maps with different variations in a grid formation.
A prerequisite however was that the variants would preferably only differ from the original version in one main way in order to make sure that the results of the A/B test could be assumed to derive from that one design change. That meant that we needed to try to show:
- the same amount of key art images as in the original design
- the same amount of descriptive text as in the original design
I ended up with an A/B/C/D test: 3 new variations of the showcase design which were tested against the original version.
Wireframes
Hi-Fi wireframe variants
The original version showed four and a half out of seven title cards. One thing that I thought might be an interesting design change was to avoid the redundancy in the original design which repeated the card in focus ie. both in the list view but also in the large format version in the highlighted view. We can also see the top of the next list under the showcase window.
Description: A carousel which you can scroll left or right in in a continuous loop.
- Large, luxurious image for the selected item.
- Part of all cards in the shop window below the showcase are visible on the screen.
Description: The showcase window as a grid. When you scroll from the third to fourth card you will jump down to the beginning of the row with 4 cards.
- Approximately the same look as a list (which is used in the rest of the product) but with the entire showcase window visible at the same time.
- Nothing is shown of the list below the showcase as it is under the fold, except for the list title.
Description: a list which shows three and a half of seven cards in which you can scroll right to the end, and then left again to the beginning.
- This is the way the showcase is designed in the Smart-TV app of SVT Play.
- No bylines.
Results
The test was run for 21 days with approximately 370,000 user sessions for each variant.
Sessions with videostarts
One key metric for validation was an increase in the number of overall sessions with a video start. We did not find a significant difference here for any of the variants. We also did not see a change in the number of overall sessions with video completions.
Sessions: % with clicks on the showcase window
The grid variant has a much higher amount of clicks in the showcase window, a significant increase of 33 %.
The carousel performed much worse than the original (-30%) and other variants. The list variant is a bit worse than original. We see similar results for videostarts resulting from clicks in the showcase window.
Sessions: % with clicks on the start page
Here we see the opposite effect; the grid variant has a negative effect on the start page as a whole in terms of clicks and video starts. The carousel and list variants have a positive effect, with the start page performing better than original and the list being the best.
Time to first video start (% with time > 30 seconds)
Another key metric was decreasing the time it takes for a user to start a video. The grid variant yields an overall larger share of sessions with longer time to first videostart for users than the original. The list has the opposite effect and reduces the share of sessions with a long time to first videostart. The share of sessions with quick first videostart increases in list variant as well (below).
Percentage of sessions with visits on other views
Our mindset is that if users needs to leave the start page more that indicates that it does not satisfy their need as well. The grid variant triggers the highest share of users to use other parts of the app (search, categories, channels etc). The other two variants have a lower share of users leaving the start page.
Percentage of sessions with a bounce
All new variants have a lower bounce rate than the original. Could this be tied to the novelty of the new designs when landing on a new interface? The test would need to be performed for a longer period in order to be sure.
Analysis
By looking at the visual indicator below, the lines indicate where the next list below the showcase begins. The color is connected to the performance in terms the percentage of sessions with clicks (and videostarts) on the start page.
This indicates that the vertical real estate of the showcase could be connected to the performance of the start page.
Insights
- Our current showcase (original) has the highest percentage of clicks leading to video start, however it is neither best nor worst in the performance of showcase or start page (clicks and videostarts)
- the grid performs the best in terms of getting clicks and starts in the showcase but is the worst for the rest of the start page, with lower clicks and videostarts on start and higher percentage of users leaving the start page to other parts of the app. It also has a longer time to videostart.
- The carousel is much worse in terms of getting users to click and start titles in the showcase BUT helps the start page do its job with more overall clicks and videostarts.
- The list variant is the best performer out of the three variants and the original, based on the metrics and data that we’ve collected. There were slightly less showcase clicks compared to the original but it has the highest share of sessions with clicks and videostarts from the start page. It also lowers the time to first videostart, lowers the bounce rate and reduces the number of visits on other parts of the app.
Our theory is that the vertical real estate of the showcase has a significant impact on the performance of the start page. One hypothesis is that more different types of content/lists displayed above the fold gives the user more of a sense of overview and confidence that they can find what they are looking for here.
We don’t know if displaying the showcase has other long-term effects, for example if exposing these titles to users before they find what they want to watch now, results in them returning to watch them later.
Next steps
Explore with other SVT Play teams:
- Change the position of Fortsätt titta (the list below the showcase) with other types of lists.
- Swap the positions of Fortsätt titta and showcase.
- Remove the showcase altogether.
Currently our Content Discovery team is working with displaying different types of content in the showcase. Originally it is 100% curated by our editorial staff but we are currently testing if we can incorporate a mix of personalized recommendations with curated content.