Under the Radar

Under the Radar 62: App Store Reviews


  welcome to under the radar a show but

  independent iOS app development I'm

  Marco Arment and I'm David Smith under

  the radar is never longer than 30

  minutes so let's get started one star

  useless but a terrible review so today

  we're going to talk about the review

  system in the App Store

  mr. star I think we're gonna start off

  by talking about a little bit of way

  it's structured and it's maybe some of

  its origins and some of the implications

  of like the way our customers actually

  see that and then towards the end I

  think we're going to dive into a little

  bit of the way as a developer we should

  approach using you know you asking for

  reviews and dealing with reviews that

  are given to us and so on so the review

  system in the App Store is very much you

  can kind of tell that it is based on the

  review system for something like a movie

  or music or something you know the

  origins of the App Store's I've

  certainly call from iTunes and so you

  can kind of see that the trickiness of

  that is that it was definitely not

  designed for something like software

  that changes all the time that already

  changes often you know a movie or

  something you publish it and it's the

  same forever a software changes on a

  regular basis and so it was definitely a

  early on especially very awkward because

  reviews would just stay around forever

  and you have reviews bad reviews you

  know for you really said busted version

  you get a bunch of one-star reviews you

  submit the fix those one-star reviews

  were still there and so the vendor shape

  will went through and changed it so that

  now your reviews that are visible

  initially to the user are always based

  on your most recent version which is

  sometimes problematic when you first

  release a new version because that means

  that if you don't have I think it's five

  or so reviews for your application you

  don't just doesn't show any of them I

  think it's just essentially to prevent

  for the situation where if your first

  review was a one-star review or your

  first review is a five-star review

  they're not gonna say that that's their

  the rating for your your app just

  because you only have one so you need a

  certain number of them so it's a little

  awkward when you submit a new update and

  someone comes to look at your app and it

  says hey there's

  no you know there's no right ranking for

  this app which thank you sir it's an

  implication at least if it's like that's

  not good well it's not necessarily bad

  directly but anyway that's sort of the

  way it goes but I think star ratings

  especially are very powerful indicator

  to our customers about the quality of

  our apps because they're one of the few

  things that are shown to the user when

  they first you know and then they're

  first looking at the list of apps so you

  know that isn't from me maybe it's the

  right way to say it you know so they you

  if you search for one of my apps in the

  App Store it pops up you'll see the icon

  I made the name I gave it the first few

  lines of the description it's

  screenshots and then the stars and the

  stars are the only part of that that are

  all right isn't marketing isn't just me

  making stuff up because obviously

  everyone's gonna say my app is great my

  app is awesome the stars have this vague

  sense of objectivity to them and so I

  think they have probably a fairly

  significant impact on what customers

  think about an app and with how you know

  how likely they are to download it and I

  think I've seen that definitely in terms

  of my own downloads where I know that if

  my star rating starts to go down or I do

  a new update and the star rating

  disappears for a couple of days that it

  can have a noticeable immuno reduction

  in downloads that if it's not like they

  go to zero but it's you know there's 20

  30 percent of my downloads will start

  will go away for a period until I'm able

  to build that up and so I think it's

  something that we have to deal with as

  developers to make sure that we're like

  aware of that the importance of this and

  then also it's like really awkward

  because you get into all the tricky

  things that you can get into where

  people start you know paying people to

  review their apps paying people to give

  it five star reviews which you know it

  eventually if Apple you know catches you

  doing that they'll pull you out of the

  App Store for that kind of thing but

  it's this really strange tension of

  where is the appropriateness there you

  know how how is asking for a review fine

  is asking for a review in a in the app

  okay is it fine out somewhere outside of

  the app etc but it's all a big mess but

  ultimately and probably the best place

  to start is just the importance of that

  star rating and

  trying to keep it up yeah I mean it

  looks like a lot of the AppStore it's

  the kind of thing where like the system

  is not perfect we as developers have to

  basically grin and bear it we have to

  work around its flaws or tolerate its

  flaws or you know somehow just

  accommodate them but also like it's hard

  to imagine a lot of better systems like

  I have tons of ideas and how they could

  improve user reviews on the App Store

  but it's not going to be like a massive

  improvement like it's if they're gonna

  have user reviews at all they're doing

  an okay job of it again not not a

  wonderful job but an okay job of it you

  know and and again like changing it

  would be incremental not revolutionary

  and and I do think it is it serves a

  useful purpose to have user reviews like

  you know I've often thought there's a

  lot of contexts in which there are no

  user reviews you know on your own sales

  page like that's how the web works like

  you know if you go to buy software and

  you go to the company's website they're

  not gonna have a set well they usually

  aren't going to have a section of random

  user submitted unfiltered reviews you

  know they're gonna handpick you know

  they're gonna cherry-pick what good

  things have been said about them and

  that's gonna be all you see and it's

  weird to have like on the page that

  everybody will go to even if somebody

  goes to your site for an iOS app fully

  intends to get it you send them through

  to click through to the App Store page

  for them to actually download it and

  they're seeing that those star reviews

  so you can't escape your customers will

  always see those star reviews so they

  are probably important from that point

  of view however I really don't know how

  important the nitty gritty details are

  for customers so for instance I think

  the average star rating that people see

  is probably fairly important you know

  you said you've seen a dip in downloads

  when that when it's been too early for a

  version to show a rating can you tell

  have you did have you like isolated

  enough variables to really have any idea

  whether it matters whether it's like two

  star 4 star average um not not quite to

  that level I don't like I think there's

  I've seen

  measurable impacts when you have a bad

  version that is a really bad rating

  where the you know the number it's like

  I don't know if there's a difference

  between being four four and a half or

  five you're like if you're on the high

  end like the difference between those I

  haven't been at quite been able to

  narrow it down but I think there's

  definitely an impact like if you have a

  really bet really low rating is really

  bad and probably even in some ways worse

  than having no rating at all because

  like the people it's like the active and

  the active statement of like this app is

  bad rather than like I don't know what

  is this app that's true but but there is

  definitely something funny there too I

  think it's also interesting that in the

  app store they show the number of

  reviews that have been left like in the

  little parentheses after the star rating

  and I feel like that number is also

  probably somewhat indicative as well

  because think if the efficient you know

  it's like five stars four views or

  whatever you know a very small number

  I probably carries a lot less weight

  than if you have you know 1,600 reviews

  and the average rating is five stars

  like that also care has more weight to

  it probably that but yeah it's a really

  awkward thing to try and like actually

  isolate and work out the impact because

  it's such a fluid thing and it's not you

  know as a devout as a developer we don't

  have any direct control over and so it's

  like you can like run an experiment of

  like let me try running the App Store if

  I have a five star review let me see if

  I have a four star review so Joe is just

  kind of more anecdotal and kind of just

  gut feeling about how it how it works

  yeah I mean you know as somebody like I

  buy things and Amazon all the time I

  occasionally will look for new apps on

  either iOS or even the Mac App Store and

  as a buyer like that what I'm looking at

  reviews I hardly ever read any actual

  post posted review text right but I

  almost always will look at the average

  star rating and the and the approximate

  number of ratings and that's the one

  thing like when I'm buying something on

  Amazon you know if I'm buying like

  something where there's a lot of fairly

  commoditized similar competition like a

  cable you know something like that like

  if there's one that has 900 reviews and

  a four and a half star average and

  there's one that has five reviews on

  five star Everidge you know that that

  stands out to me and I will pick the

  more popular one those figure I will

  figure there's probably some reason why

  they have so many more that that's

  probably a good sign it's just a larger

  sample set so to achieve a decent

  average with that big of a sample set is

  a pretty good sign compared to something

  with five reviews right and I have to

  imagine that most people will do the

  same thing with App Store reviews

  because it's it you know whatever we

  think of our apps and and our position

  of the market the way the way I think

  the market has has shown us over and

  over again that it actually works is

  very much like people buying basic

  cable's on ahem azan where it's like

  they're looking at a list of a thousand

  apps about looked the same and they're

  gonna just pick the one that has like

  you know that shows up near the top

  probably the one that's free if there is

  one that's free which is they're almost

  always is and they're gonna pick the one

  that has a decent looking you know name

  and icon and screenshot maybe and

  they're gonna and with decent ratings so

  in the aggregate I do think these are

  very important to to kind of just keep

  your average up and to try to get a good

  number of reviews that being said I

  think the specifics of like if you have

  an app that has mostly good reviews and

  there's one angry person who leaves a 1

  star and trashes you in their review

  even if even with stuff that isn't even

  true I mean look I my attitude both in

  you know personally and in my software

  development style my attitude inspires

  extreme feelings in some people and I

  get a lot of of those extremes and and

  some of that is great if most of it is

  great you know most of my reviews tend

  to be pretty positive but there's always

  with anything I do I could write a blog

  post saying the sky is blue and there

  will be one person raging against it on

  Twitter you know it's like anything I do

  because of my attitude and style there

  it okay it will anger a certain number

  of people right and so everything I do

  always has a small percentage of really

  negative response or reviews on it so

  you know even the best versions of

  overcast the best version of Instapaper

  like even like the most uncontroversial

  updates there's always a one-star review

  or two or three or four you know they're

  always there

  some of them get really rude and some of

  them are really personal but whatever it

  doesn't really so

  basically I've stopped reading them like

  because it's just I found it mostly

  unhelpful to read the reviews I find it

  much more helpful to pay attention to

  what people are saying via email and

  Twitter and even though I don't usually

  even respond to those and sometimes

  that's gets overwhelming I can't even

  read them all honestly but but usually I

  am able to keep up with reading them and

  you know the more engaged somebody is

  with trying to contact you and trying to

  use your app I think the more useful

  their feedback tends to be and so App

  Store reviews are kind of the bottom of

  the barrel like somebody can look at

  your app for three seconds and leave a

  review and they don't even like they

  didn't get to your contact link they

  didn't they didn't find your Twitter

  username like it's like there there's a

  lower barrier to entry there and to some

  degree that means you're gonna get more

  feedback that way which is often a good

  thing but it also means that the level

  of engagement these users and level of

  effort these users is pretty pretty low

  on average and so you're gonna get more

  of the crap that you should mentally

  filter out coming in through that

  channel than through other channels and

  so the way I look at like individual

  reviews the ones that are like onestart

  angry and everything is you know if I

  see one

  which again I've been seeing fewer and

  fewer because I stopped reading the

  reviews for the most part but if I if I

  see one making a reasonable point I will

  take mental note you know like okay this

  person really really hates you know my

  decision to remove this feature six

  months ago whatever okay you know that

  that's that's like kind of like an

  upvote in my head of like maybe I should

  reconsider that or or whatever else or

  maybe people want a way to address this

  problem that I removed forever ago but

  for the most part I don't think you have

  to worry that much about any individual

  review both score wise or its content

  because what matters way more to the

  people who are actually browsing for

  your app is the the number of reviews in

  the average star rating and I would be

  very surprised if a lot of people

  actually went through and read a

  meaningful number of review

  now this obviously is a bigger problem

  if you don't have very many reviews and

  so we'll get into this but like you know

  all the the the problem of like the

  occasional one-star angry people is

  magnified the fewer reviews you have

  because the you know the few reviews to

  kind of bury it or average it out or

  whatever else and so I've been lucky

  that overcast has almost always had a

  very large number of reviews and I think

  we'll talk after the after sponsor break

  about kind of how and why and and when

  and when not to ask for reviews because

  I think I think that does have an impact

  but first let's talk about our sponsor

  this week it is a new sponsor it is

  hired you can see hired calm slash under

  the radar for more sign up today to

  connect with opportunity we all know

  that searching for a job can feel

  stressful scary and time-consuming

  there's pushy recruiters oh man the

  recruiters make job hunting the worst

  they try to sell you one roles that you

  don't actually want and all this crap

  and then job boards make you feel like

  you're throwing your resume into a black

  hole never to be seen again I mean job

  boards are very often you know literally

  of the black hole of nothingness this is

  where hired comes in to solve this

  problem hired makes your job search

  faster focused and stress-free by

  offering access to 4,000 innovative

  employers including big brand names like

  Facebook and small emerging startups if

  you want to work that way I'm hired you

  receive personalized interview request

  and upfront salary information so you

  can make informed decisions about which

  opportunities to pursue and today

  listeners can earn double the normal

  $1000 hiring bonus by using our link to

  sign up hire comm slash under the radar

  so that's right you can earn $2,000 as a

  hiring bonus by finding your next

  chapter and hired using our link hired

  calm slash under the radar sign up today

  to connect with opportunity thank you

  very much to hired for sponsoring our

  show yeah because I think what the thing

  that's so tricky with all of these

  things with reviews is this planning

  this balance between like it's um it's

  it's a really it's an odd part of the

  development process because we don't

  have any direct control over it but it

  has a direct impact on us and there are

  very few things I think in the

  development process like one of the

  things that I love about being a

  developer is that like I'm telling my

  app what

  like it's a little bit control freaky

  like I enjoy that I can control all the

  behaviors all the aesthetics everything

  and everything in my app I can control

  because it exists you know as a result

  of like sitting on Xcode and making it

  but the reviews are completely out of my

  control like I have no direct way of

  doing that and instead and I think like

  we're just saying it's it's like any

  individual reviews probably not all that

  important but it's more important though

  is that we are able to encourage our

  customers to go there and a high enough

  volume of customers to go there such

  that you have a representative sample

  and obviously I prefer to be

  representative of customers who love my

  app and want to give it five stars and

  talk about it lovely but at the very

  least I want that number to be

  significant so that it isn't just

  drowned out by the one person who's

  angry or even just given that bad

  impression of like why does this app

  only have seven reviews like you kind of

  least in my mind I start to like I don't

  know what but exactly what the

  multiplier is but I'm like if the app

  you know it's like there's a certain

  like the audience and the customer base

  of this app is probably some multiplier

  of that and so if it's only like seven

  maybe the app is kind of is only used by

  a few handful of people and isn't that

  good and so there's a strange tension

  though and there was this that's

  probably three or four year years now

  but there was a big hoo-ha a couple

  years ago where around the like little

  pop-up dialogue said a lot of apps use

  where it you know you're using the app

  using the app then all of a sudden it

  pops up and says hey we're gonna review

  my app and there was an app there was a

  library called he was operator that was

  used in just over hundreds and hundreds

  of apps and got you know had this very

  aggressive version of that and I think

  it's that's it's something that I try

  and always avoid in my apps like I've

  never really felt great about doing this

  because the reality is like the reviews

  are primarily beneficial to me and

  they're beneficial to future potential

  customers the actual current customer

  who's using it the review doesn't really

  benefit them in any way directly other

  than like

  the vague sense that it makes my it

  helps me which helps them because it

  helps me to be wanting to make the app

  better for them but it's mostly just

  they're doing me a favor and so I need

  to look at it from that perspective and

  I think you know automats I've avoided

  putting in things to kind of ask for

  them for that because it is asking them

  a favor and it's like you don't want to

  you want to be really thoughtful if

  anytime you ask your audience for

  something but then I also look in the

  App Store and next reason this is top of

  mind for me is I just went through and

  added a little section in the settings

  area of pedometer plus plus where it

  says you know would you like please

  leave a review leave me a review really

  helps thank you essentially I can have a

  little button to go to the App Store to

  leave a review and I hadn't had one in

  there for a long time yep usually was

  reviewed fairly well but I was starting

  to look at look at it from the

  perspective of a new customer and I went

  into the App Store and I started you

  know searching for the terms that you

  would search for if you were going to

  try and find that app and what I was

  noticing is that I had various massively

  disproportionately lower number of

  reviews then my cus customers and based

  on like where we rank together our

  downloads are fairly similar which makes

  me think that a lot of it is coming just

  from the fact that I'm pretty sure a lot

  of those competitive competing apps have

  the like in-your-face pop-up hey go

  download the app and enough people go do

  that that their numbers look good but

  from a fresh customers perspective that

  means that my app looks like nobody uses

  it or being a very few or

  disproportionately fewer that you know

  our start of the actual average star

  rating is very similar but I have a

  tenth of the number of users or reviews

  left and so it's like now I've added

  this I don't feel great about it but I

  think putting it in AB you know it's in

  the bottom of the settings area it's the

  kind of thing that you'd probably only

  find if you are you know you've used the

  app a little bit and it's also you know

  it's never in it's never end going to be

  in the course of the core functionality

  of the app so in this case like showing

  you your steps it's not like you open it

  up to see your steps and you know your

  steps are hidden away behind something

  asking you to go and do it but I think

  the reality is not having one of those


  kind of disadvantaged disadvantaged we

  was putting my putting myself at a

  disadvantage that it probably didn't

  need to be yeah I mean I I kind of came

  around to that as well and I came up

  with a a similar solution for overcast

  and an overcast 1.0 was released in was

  2014 it was it was kind of right in the

  middle of a lot of this debate about

  about uh the the irate dialog box and

  and I mean these and these these rape my

  app boxes are now so ubiquitous like you

  know big apps like Instagram does it it

  bothered me all the time my bank chase

  the Chase Bank app does it I'm like

  really like I'm trying to posit a check

  and like this is like business stuff and

  you're interrupting my business stuff

  with my business bank for your stupid

  rape my app dialog for a bank app like

  that but yes it they're so ubiquitous

  and I I think you know what you said is

  is perfect about like you know you're

  asking the app is interrupting good

  usage of your app to do themselves an

  advertising favor and that's that's it

  kind of rubs me the wrong way as a

  customer and user of these apps and so

  in my apps I've always kind of been you

  know not only refusing to do that kind

  of interruption but for a while I would

  like with Instapaper and the magazine I

  never had a link even in settings or

  anything to rate cuz I just thought like

  kind of like the the pure angle kind of

  like back before I would do colon

  keywords after a title of an app you

  know I used to think well this is I want

  to do things like the pure proper way

  and this is just how things should work

  and sometimes how things should work

  isn't how things do work and and so you

  have to find some kind of compromise and

  so you know with all these little you

  know app store realities or shortcomings

  you know like like the keyword title

  thing I found a compromise of with

  overcast I actually couldn't name it

  overcast there was there was another app

  in the store with an exact name was a

  cloud file really dirty app so it wasn't

  even like the same kind of thing but I

  had to call it overcast something so I

  called her an overcast : podcast player

  like the the most basic honest short key

  word thing I could prove I could really

  come up with and and so with the with

  the user ratings thing I had a a similar

  kind of compromise with overcast 1.0

  it was it was clear this kind of thing

  was important so I wanted I wanted to

  address it somehow and so I just put in

  the settings area and in the settings

  screen it's still there now it's been

  unchanged since version 1.0 I have on

  the server side a when when overcast

  syncs it the the part of the sync

  response is the cached ratings count

  from iTunes because you can get it

  through the iTunes API this is the

  iTunes Search API so you can get your

  own ratings count from your server or

  directly from your app that way so the

  app keeps a record of the ratings count

  for the current version and so in the

  settings screen again it's been in there

  since 1.0 I have a simple item right

  next to the feedback item that says

  please rate overcast and below it it

  says 2014 people have rated this version

  and if that number is less than I think

  50 it says only X people have rated this

  version and right below that it says

  overcast will never interrupt you for

  ratings and I think enough piece has

  worked remarkably so my thinking here is

  it's a simple polite relatively compact

  thing that encourages people who like

  the app to go support it in this way

  but it's not pushy about it and

  clarifies right there that they will

  never be interrupted so this is telling

  them that you know I thought about this

  very carefully like this is telling them

  that I respect them it's tell them right

  there I will never interrupt them for

  ratings so it's telling them I respect

  them I won't do this thing that everyone

  does that you hate and also please

  please rate it and if the number seems

  low especially when it's low enough to

  be prefixed with only that's kind of an

  additional push like here's how much I

  can actually use your support right now

  and I've never had to call attention to

  it in like in on Twitter or anything

  else like or I've never called attention

  to it in the app like it's just there in

  settings and it has always gotten enough

  reviews with remarkable speed that and

  part of that so you know I've been

  fortunate with overcast the user base is

  pretty sizeable now but even back like

  in the early days it was always that way

  that kind of approach like when you just

  respect people and and you you let

  people who like the app

  supported in this way in a reasonable

  way and without getting in their face

  and annoying in them that's great now

  that being said that's not gonna apply

  to every app and that's unfortunate you

  know it's one of the defenses back in

  the day when when these boxes were being

  debated was well they work and we need

  them and so I I think what you have to

  just keep in mind is that that's not the

  only approach that can work and while

  you do probably need some kind of easy

  way for people to rate your app doing it

  in a more respectful way like what we

  are doing and in our apps not you know

  not to toot our own horn too much but

  like doing this in a way that doesn't

  interrupt your user and annoy them is I

  think a far superior way to do it

  because it works well enough so like you

  don't need to be more pushy about it

  obviously these kind of ways do work

  well enough and it doesn't compromise

  the user experience and it doesn't annoy

  your your legitimate good customers so

  to me it's a win-win yeah and I think

  too there's like these kind of things I

  always do do also take the other I

  always try and take the other

  perspective to Oh finally I find the a

  rating style log very offensive or at

  least very annoying but I sometimes do

  wonder if my how many of my customers

  actually have that same a that same

  experience in the same perspective and

  it reminds me in many ways have the same

  way that like I don't like using a apps

  that have banner ads in them but I make

  apps that have Henner head's in them

  that I people use on a regular basis and

  doesn't seem to be a problem for people

  like fievel's thresholds for what they

  can support if they're annoyed by or

  what they consider you know undesirable

  they're certainly different I think then

  what I have and which is fine but it's

  also something that I was trying to keep

  in the back of my mind as I'm working on

  through these kinds of things is that

  well I may think that these things are

  really annoying maybe some people don't

  don't think they're annoying maybe some

  people think that they're actually

  useful they're like oh yeah I'd love to

  support the SAP I didn't know I could do

  that or you know this is something that

  I enjoy doing and they like being

  prompted to do it in the feeling of

  empowerment of that or whatever but I

  think whenever I was trying to wrestle

  with this a little bit is I think the

  approach that it probably makes sense

  for most developers

  is to like to keep keep an open mind in

  terms of like what I was just saying

  with myself of like well I may find

  these annoying my customers may not but

  not to take that to mean a as like a

  blank check if I can just like do

  whatever I want because someone might

  like it but to view it from perspective

  of like this is important this is

  something that I think I've been

  negligent in for a lot of my apps of

  making sure that I'm to giving them

  every opportunity to succeed and having

  a good rich set of reviews in the App

  Store is probably an important part of

  that but the way to approach it is to

  not just like jump all the way to 11 and

  be like yeah like just throw dialog

  boxes up every time they launch the app

  like no that's probably not right but

  instead when this is the approach thing

  I'm kind of planning to take with my

  apps going forward it's like I'm gonna

  I'm gonna do this incrementally like I

  used to have no dialog boxes at all I

  would just have the like a sentence or

  two in my release notes was the place

  that I typically did it where like in my

  changelog at the bottom I'd be like hey

  if you have a minute right or maybe

  that'd be great

  not only does nobody read those but

  people who do see them there never gets

  the bottom yeah so like that was you

  know by far the least of the least

  obtrusive version that hasn't been

  working perfectly so now I'm gonna put

  it in the settings area right I'm gonna

  I'm gonna I'm gonna put it there and I'm

  gonna engage the response then I'll see

  how that goes and I'm gonna if that's

  not enough if that's still not quite

  getting the outcome that I'd like I can

  you know put it in a different part of

  the settings area right now it's in the

  bottom maybe I'll move it up to the top

  and if that doesn't work maybe I'll

  periodically show it on the main screen

  but not in a modal way like this is even

  the mindset I'm trying to take though of

  like keeping my options open being

  respectful always and not wanting to do

  things that are complete you know like

  super user hostile but like keeping an

  open mind and doing this in

  incrementally and seeing like where is

  that point because my suspicion is and I

  think your experience certainly is a

  good example of this is like you can

  find a place where you're kind of

  hitting that balance between getting the

  outcome that you need because you don't

  do not that probably much of a

  difference between having you know one

  thousand and two thousand user reviews

  in the App Store for example like

  my guess is once you hit forward four

  digits like people are they get you got

  a sense of like wow that's a lot of

  reviews and so just gotta get whatever

  we have a mechanism such that you can

  hit that number consistently and once

  you can do that I think you're probably

  gonna be fine and so I'm just gonna

  start as basic as I can and I'm just

  gonna keep dialing it up slowly until I

  hit that point and see you know see

  where that actually is I give this

  approach four stars that seems

  reasonable I withhold the fifth until

  you do my favorite pet feature all right

  we're out of time this week thank you

  very much everybody for listening and we

  will talk to you next week bye