Under the Radar

Under the Radar 5: Managing Feedback


  welcome to under the radar show about

  independent iOS development I'm Marco

  Arment and I'm David Smith under the

  radar is never longer than thirty

  minutes so let's get started so today

  what we wanted to talk about is the way

  in which we manage and deal with getting

  feedback on our products on our apps

  both in terms of from customers but I

  think one of the things that I really

  wanted to unpack a little bit is the way

  that we do we handle feedback when

  products are in the earlier stages so

  things like beta testing or early

  interactions with customers because some

  of the things that I found really

  complicated when I'm developing an app

  or developing a major update is I will

  have in my mind a vision for what I want

  that update to do what though what I

  want the app to do and I kind of am

  working towards that I'm working towards

  that but at in this early sort of like

  embryonic state it's only in my head and

  in the little prototypes and the little

  bits of code that I'm building and

  something significant though happens the

  first time you ever show it to someone

  else even beyond just telling someone

  the idea but when you actually show them

  something and you give them something to

  react to react to the way that you think

  about your the app seems to change for

  myself because suddenly it has their

  feedback their suggestions their ideas

  start to be kind of mixed in and muddled

  along with mine and so something that

  I've learned is that I have to be really

  really careful about how I get that

  feedback at what point I get that

  feedback and the way in which I sort of

  manage it because what I don't want to

  do is take too much feedback too quickly

  and kind of lose that vision for what I

  want to do where I want the app to be or

  what I want the update to be and so it's

  something that I it's a weird discipline

  that I think you kind of have to learn

  and I was curious though if you've had a

  similar set of experiences how do you go

  about kind of pulling in that feedback

  during early development I will usually

  you know ask close friends for input on

  certain basic decisions about it basic

  feature decisions how things might be

  structured what features might need to

  be necessary or unnecessary and then I

  I've recently in recent years gotten

  really a big fan

  of medium-sized private betas for me

  that the private beta is extremely

  useful and I've done a number of sizes

  of betas I first you know in the early

  days of overcast I was doing a beta of

  like I think like 20 people

  I've also since then done betas with 800

  people because test flight allowed you

  to do 1,000 now it's actually 2,000 but

  so that I did like 800 people on a beta

  that was like I basically had a public

  signup form and just without that almost

  anybody in more recently I've done much

  smaller babies going back to that

  original size so if like 20 to 40 people

  again and what I found is that the the

  quality of the feedback doesn't actually

  seem to change significantly with the

  size of the beta and which is

  counterintuitive but it seems like you

  can show the app to a relatively small

  number of people and you'll pretty much

  get all the feedback you need from a

  small group and and you know maybe it

  maybe not a small group of like two but

  10 or 20 people if they're actually

  gonna use it and give you feedback which

  is a separate point but if they're

  actually gonna use and give you feedback

  you don't need that many people involved

  in the group now the feedback you get it

  tends to be really focused or really

  heavily weighted towards upfront

  feedback right the very first time

  somebody uses a beta because and I don't

  know if other people do this but I do

  this where if somebody gives me a beta I

  will usually take a lot of time going

  through the app that very first time and

  giving pretty detailed feedback to them

  but after that first time as more

  updates come in I very rarely ever take

  that amount of time again unless I have

  you know spot an obvious but then I can

  you know screenshot a tenant's them

  whatever but general feedback about

  about a beta or about an app I usually

  it's usually front loaded and I found

  that to be the case with most of my

  testers as well so what I will usually

  do is if I can I will have a few new

  testers in every group and test flight

  allows you to see who among the group

  didn't install the latest few version

  like it'll tell you the last version

  each person installed so if I see some

  of my testers losing interest you know

  like if they if they just don't install

  the updates or if the last IP they

  installed was like five builds ago I'll

  just quietly remove them from the

  list because and it's no hard feelings

  you know his I do that office I always

  thought about this all the time because

  I lose interest or I don't have time in

  that in that month or whatever so I

  usually will cut people who who kind of

  aren't using it and you know no hard

  feelings lots a lot of times are my

  friends I don't I don't even you know

  doesn't doesn't matter no hard feelings

  and then I will add a few new people

  every time because it allows me to get

  that that initial impression like

  somebody's first impression of this app

  it allows me to have that ongoing

  throughout the process rather than

  showing it all to the same people right

  up front and never bring any new people

  in along the way so I've only recently

  started doing beta testing properly at

  all I would say like the most recent

  update I did for pedometer plus plus was

  the first time I'd ever done a big

  update our big beta test before I

  launched something big and you know it's

  the before that it had only really been

  I would Institute - I would have it

  running on my my own phone maybe my

  wife's maybe a few like a one or two

  other people and honestly and then on a

  lot of testing devices and I'd never had

  that experience of having this big kind

  of like wide like I did a similar type

  of thing like I just said anybody who

  wants to beta test pedometer plus plus

  let me know and it went to test flight

  and I put them all in and I had this an

  interesting thing that was sort of like

  similar to your experience where you get

  this big wave of comments in this big

  wave of feedback early on and then it

  sort of dies down and then actually I

  found is you ended up with another wave

  like a week later with sort of like the

  huh now that I've been using it for a

  while here's like some other thoughts

  and I would say that I had a very

  similar kind of experience to you

  insofar as there's only a certain number

  amount of feedback was actually helpful

  and like beyond that it you start to get

  a lot more repeats you start to get a

  lot more tangential things but like at

  its core what I found helpful in the

  beta test was validating a that the app

  works and is functional in the way that

  I need it to be and then to its

  validating that it's a good idea that

  people are like I'm not getting to like

  what this doesn't make any sense or

  people who have used the app before and

  going to this version are like this

  doesn't make you know you're totally

  breaking the thing that made the app

  good before because I think those are

  the things that I find out

  struggle with as myself because I have

  been working on if you see I've been

  working on an update for for several

  months my experience with the app at

  that point becomes almost exclusively

  the new version of the app and not what

  the existing version in the App Store

  is and I can tend to lose sight of the

  distance that I'm creating between those

  two things because I never go back to

  all right other than for like my final

  acceptance testing like I won't go back

  to the App Store download the version

  and run that for a few days like that's

  just not typically the way I'd do it

  because obviously I'm constantly running

  development builds on my own machine and

  the feedback that I got was kind of that

  was most helpful is when I have someone

  who's like so I used the old version

  yesterday and I used the new version

  today and this felt weird by going from

  there to there and like those types of

  feedback are things that are really hard

  for me to do because like I said I have

  this whole new vision for the app I've

  imagined and built features around this

  whole new thing and that's what I'm used

  to and months later like sometimes the

  weirdest one is when you go back to the

  App Store version you're like what is

  this like I can't believe nothing and

  you've not even necessary like oh I

  can't believe this shipped but like whoa

  this is different like this functions in

  a different way this has lots of other

  issues and so that was something that I

  learned from this feedbacks like look at

  these little like huh this was weird

  coming from that and that's helpful both

  in terms of from a development

  perspective but then also from in terms

  of what questions are my beta testers

  going to have about the app

  what's weird to them what's new to them

  and then when I take that feedback I can

  structure my documentation or my

  remember like release notes or things to

  be like if you're coming from the old

  version to the new version you might

  find this weird if you find this weird

  this is the answer to your question but

  that's something that I was never able

  to do before I was able to find test

  flight like I have a broader audience

  and something that I definitely now I

  think will use going forward

  whereas before I always view test light

  it's like oh it's beta testing it's just

  like making sure that there's no

  Crashers this making sure there's not a

  lot of things but understanding customer

  perspective changes from the old to the

  new was really helpful

  but even when starting out new I mean

  the the initial overcast beta I did 41.0

  before the app is out was incredibly

  valuable in

  because when you're developing an app

  and you're only using it yourself you

  know like like you mentioned I do the

  same thing where like I'll use it myself

  for like you know months and and as soon

  as I can get it running on my phone

  that'll be I lose it myself full time

  and when you're developing the app

  yourself you know how to use it and you

  know what you did so everything makes

  sense to you and when you first show it

  to beta testers especially for a 1.0

  where they are not familiar with the app

  before the the most valuable perspective

  I got was what doesn't make sense or

  what is being misunderstood from what I

  intended and I made tons of interface

  improvements and tons of changes during

  that initial test you know I thought

  that a beta test would just be like oh

  just let me know if anything crashes or

  breaks and then I'll ship it in a few

  weeks it turned out to be months long

  because everybody gave really good

  feedback about you know this doesn't

  really make sense or I don't understand

  what this is and and the app I

  dramatically improved the app it was

  almost like shipping at version 2.0 you

  know we're headed than shipping 1.0

  because I had such great feedback just

  by asking people and again you don't

  need this was only a group like 20

  people it doesn't need to be a big group

  it just needs to be people who aren't

  you because you you get everything you

  did because you did it and you designed

  it and as soon as you show anybody else

  you immediately will see the flaws in

  what you did because you will see them

  either struggling to to understand what

  you did or misinterpreting things or

  missing entire big features because they

  just don't see them or they don't they

  don't see why they would need them or

  something and it's great like my effects

  panel in overcast look totally different

  when when I sipped the 1.0 beta the very

  first beta voice boost didn't even exist

  by that name it was actually a boost

  slider and it had four different modes

  and and and smart speed was you know it

  was there but everything was kind of

  rearranged differently and and over time

  and the the wording that the labels were

  all different and over that beta I was

  able to really refine that panel reduce

  voice boost down to either an honor and

  off which was actually it's it was at

  strongest setting I

  just I realized happy to development you

  know actually I always just leave it the

  strong a setting because that's the best

  so obviously let me the other ones and

  just make it the strongest setting and

  and it you know stuff like that would

  reduce user confusion and the beta I

  rearranged major parts of the interface

  during the beta the directory was was

  dramatically changed and added to during

  the beta I mean so many things in the

  interface so many changes so many you

  know boarding of microcopy in the

  interface and everything so much of that

  was improved simply by asking people

  like hey you know what do you think of

  this and and watching watching what they

  said and seeing what they got and what

  they didn't get and what they

  misunderstood under the radar this week

  is sponsored by image X IMG I XCOM /ut

  our image X is basically it's hosted

  image processing and image resizing so

  so you kind of give them a Quebec end

  source of images whether it's your web

  server an s3 bucket or even just

  arbitrary URLs that you sign and have

  them process then you can just serve

  their URLs for that for those images in

  your apps on your websites anywhere that

  you could that you need an image served

  over HTTP and you can do real-time

  processing on that image just by URL

  parameters to their service so I use

  image X I used it for a while for

  overcast thumbnails it is a fantastic

  service for just basic things like

  resizing you know like all I need to do

  with overcast thumbnails most of the

  time is resize them and serve them over

  a fast CDN and it does that flawlessly

  there I have no complaints about that at

  all but also what it does a little more

  fancy stuff like for instance you can

  serve you know automatically serve

  different dpi to different screens so

  like if you have a device that it has a

  Retina screen versus one that doesn't it

  can automatically serve the right dpi to

  that to not waste bandwidth or not look

  bad you could automatically serve things

  like like the WebP e format to devices

  that support WebP II things like that

  there's all sorts of great stuff you can

  do automatically and then also you can

  actually adjust the images by using

  those URL parameters so you can do

  things like you know change the colors

  crop them in different ways

  rotate them add annotations I mean

  there's so much you can do with this

  pretty much any kind of like image

  filtering tasks that you can do so much

  editing technique you can do here just

  by changing URL parameters and it all


  their proprietary system that is all at

  gpu-accelerated it's it is so fast I've

  never seen an image processing CDN that

  works this quickly and that's why I use

  them in matrixes API is very easy to use

  I can hurt my own thing myself to do it

  in like one function but if you want any

  other kind of library support they have

  tons of libraries for different

  languages including one for Swift from

  the developers over at ho dinky so check

  this out

  image six comm that's IMG I XCOM slash u

  TR for under the radar thank you very

  much to image chicks for sponsoring our

  show once you're out of this beta phase

  like you've gotten all this good

  feedback from your beta then you have

  the interesting thing of how do you

  transition to feedback from customers

  more generally so you're kind of you're

  past this point where fair enough you've

  got all this great feedback from a small

  group of people you open something up

  you put it out as a broader update to

  lots of people and then things get

  really interesting at least or at least

  complicated because suddenly you have

  lots and lots of customers who all have

  different visions for your app all have

  different goals for your app and just by

  the volume of like the virtue of having

  so many numbers like hopefully your app

  is gonna have thousands tens of

  thousands hundreds of thousands millions

  like you're gonna have such a wide and

  varied user base at a certain point that

  the feedback you get and the way you

  collect and manage that becomes like an

  engineering problem in and of itself

  it's something that I've over time had

  to kind of wrap maybe become more

  comfortable with the understanding that

  I'm not a I'm never going to make

  everybody happy like that's just a given

  like right off the bat if you ever are

  kind of trying to build an application

  catering to everybody you're gonna end

  up building like a terrible application

  because you just can't do it you're

  never gonna make something that makes

  everybody happy but you're still are

  gonna get a lot of feedback and that's a

  good thing like when you get feedback

  from a customer you know they send you

  an email and say hey I love the app I

  wish it did X like that's a good sign

  that means they like the app enough to

  open up an email their email client then

  write down something thoughtful and it's

  a big enough part of their life that

  they see it as something that could be

  better could more fully meet their needs

  and that's a good thing

  but I've definitely gone down the rabbit

  hole many times where I look at that

  feedback and be like oh yeah I should do

  that I should do that I should do that

  and you can end up with a product if you

  kept if you kept saying yes to every bit

  of that feedback like by version 2 or 3

  or what you know several updates down

  the road your app is gonna look nothing

  like what it really was

  especially for fulfilling or like a

  smaller shop like what makes my apps

  good typically is they're like

  simplicity and the design around being

  really accessible because I can't build

  big monolithic massive structures

  because I'm just one guy I have to

  always be fighting that tension and so

  typically what I'll end up doing is I'll

  say hey thank you for the feedback I

  appreciate it it helps like inform my

  future decisions but I never say or very

  rarely will say like yes absolutely

  that's a great idea I will definitely do

  that it'll be in the next update unless

  it's something of course that I'm

  already working on but being able to say

  no to people was something that I

  definitely had to learn it was something

  that I had really struggled with early

  on how do you deal with feedback from

  like the more general public to your

  apps you have to consider when you get

  feedback you know when somebody is

  asking for a feature or asking for a

  change it's easy to look at that and say

  oh well if I can add this feature or if

  I can change the app to work in this way

  then I will get X number of people that

  will be happier with the app or that

  might buy it who wouldn't buy it before

  if you keep following the try but you

  said like it's very easy to basically

  have like the the microsoft office 97

  toolbar of app features you know to have

  like this giant wall of features really

  massive settings areas and and tons of

  tons of just complexity and and burden

  in the app and I can very easily

  overwhelm not just an independent

  developer but also the users who have to

  see all those options and don't

  understand them or might or might

  configure them wrong and not understand

  how they got there and and so what you

  have to consider when when you have

  people saying oh if only it had this

  feature I would be happier there are

  many people who are already happy with

  the app the way it is and if you add

  things to it or if you change the way

  things work they might become less happy

  with the app and so that's always been a

  very tricky balance for me with overcast

  because I when I made overcast it was in

  part in response to what I was seeing

  from so many podcast apps at the time

  which was just so much complexity in the

  interface and the settings and

  everything and I wanted to make

  something simpler a lot of people use

  overcast because it was simpler at the

  time you know and now you know that now

  there's there's a lot of great podcast

  after now many of which are that simple

  but but at the time that was like a

  unique selling point for overcast and I

  I never want to ruin that and you know

  not only for for myself and for my

  customers but also just like for the

  quality of the app like if I add too

  many features the app gets worse because

  it's harder to navigate it's more

  complex there's more edge cases that I

  might not be able to test as reliably as

  I test the core functionality it's a

  very tricky balance like in in 2.0 I

  added streaming and I thought it would

  be amazing if you could just tap any

  episode and before tapping an episode

  would just download it if you didn't

  already have it downloaded and you'd

  have to wait for it to download before

  it would play and in 2.0 I think well

  now I have a streaming engine so if you

  tap on an episode now I'm just going to

  start playing the episode just start

  streaming it and that'll be amazing and

  to me it was amazing

  but the the number one feedback I've

  gotten for 2.0 has been how do I turn

  that off and just go back to the old way

  of doing it because I want to just go

  download a bunch of stuff I don't want

  to start playback immediately now

  downloading things requires two two

  steps you know it's half the input

  button and tap downloads look now I've

  made the way that people used to do it a

  little bit harder because I thought the

  new way was better but not everybody

  thinks so and so now and so now I have

  to weigh this decision of like well do I

  add another option for that do I change

  the interface to be more like kind I

  like the way when you tap a tweet and

  Tweetbot it rather than going directly

  to something it expands a little like

  button panel below it I thought about

  maybe doing something like that there's

  now I have I have this property to solve

  and no matter how I solve it if I leave

  it the same as it is now or if I add a

  setting to just toggle the old behavior

  back on which i think is kind of the

  worst way of doing things or leave the

  way it is now in which case the you know

  somebody's angry or the app is slightly

  worse or I can do the Tweetbot button

  row thing when you tap an episode cell

  which will anger the people who like the

  streaming immediately approach

  now I'm adding I'm making that two taps

  for that instead of one so none of these

  are perfect and that's the thing when

  you're facing app design decisions like

  this and when you're dealing with

  people's feedback nothing you do will be

  perfect nothing you do will satisfy

  everybody and so you basically just have

  to fit you kind of have to develop a

  sense yourself and you can you can see

  the results - you kind of have to

  develop a sense yourself of what will

  probably please the most people and

  result in the best app I feel like it's

  it's relatively counterproductive to

  think too much about what were people

  accustomed to before because all your

  new customers just see the new app and

  the app at the app as a thing the app is

  a concept is what it is today not what

  it was two years ago and so even as

  you're bringing your customer base

  forward through updates I always think

  it's way more important to have a better

  app today and a better app for tomorrow

  than it is to please everybody who

  wanted everything done the old way so as

  I'm weighing this feedback about

  overcast streaming versus downloading

  when you tap on the thing I'm not really

  considering you know do I do I give that

  option to people to just toggle it back

  the way it was before because that that

  to me is like losing the quality fight

  because that is just adding another

  option that new new people will go into

  the app not knowing or caring what that

  does if they accidentally turn it on the

  app will start behaving differently and

  they'll think it's weird and if they

  didn't remember how to turn it on so I'm

  not really considering that option I'm

  more considering like probably the doing

  the tweet button row thing instead

  because I think overall if I was

  starting from scratch now that I have

  many different things you can do on an

  episode rather than just tapping it to

  download it that is probably the best

  approach so that's probably what I'm

  going to do because I'm thinking about

  what what what will make the best app if

  somebody looks at it for the very first

  time not how do I please everybody

  throughout history because not only is

  that not good for app quality but it's

  impossible and I think it also speaks a

  little bit to one of the other traps

  that I know I've run into a lot with

  feedback is the trap that it's very easy

  to over weight the feedback you get

  proportional to the size of your user


  so like the only who you hear from are

  the people who are I typically very

  upset or very happy and the entire like

  middle of your user base which is

  probably in many ways the more important

  group of your user base you'd hear

  nothing from its I've definitely fallen

  into the trap several times where like

  I'll get a feature request like once a

  week for like six weeks and from

  uniforms from six different people and

  because it's happening at the sort of

  interval in my mind it's like oh wow

  people are could keep asking for this

  this must be really important but then

  you take a step back it's like six

  people have asked for this feature like

  the app has 1.4 million users and six

  people have asked for it like it's very

  hard to not conflate those things and

  make it feel like wow this is really

  important because the only people you

  hear from like there's the self

  selection for people who want things

  changed whereas all the people who want

  status quo or at least who are happy

  with what it is you never hear from and

  that's a really funny dynamic that you

  then have to have to like balance with

  and get comfortable with to say like

  okay like is this when I'm like if I get

  what care what it actually is a lot of

  feedback like it is has like 10% of my

  user base reached out to me to ask me

  for a feature like if they do like

  that's probably worth paying attention


  I probably never gotten feedback beyond

  like every - every now and then where

  I'll you'll accidentally like ship a

  horribly horrible crashing bug in which

  case fair enough you hear from a lot of

  people saying fix the crashing bug and

  you know that's like good feedback I

  will definitely fix but in general like

  you just you just kind of have to go

  with it and I think we're to wind this

  down is to focus on in order to build

  software I think you have to be able to

  have a vision for what you want the app

  to do like you're trying to build this

  vision it's been a bit abstract but it's

  like you want have someone to like they

  know it's yours they know what the app

  is like it has a personality maybe

  that's the right way to say it's like

  you're trying to create this persona

  that your app

  and doing things that ultimately will be

  counter to that persona are probably

  going to be counterproductive when you

  get feedback that are people who are

  like it's like I like your app but I

  want it to be something else it's like

  they really want your app to be a

  different person to have a different

  personality and you have to then build

  this you know this whatever the like the

  confidence even to say no this app is

  about this like it's about simplicity

  and focus and ease of use or it's about

  you know maybe it is a really strong

  like complicated advanced thing in which

  case is like yeah let's we every feature

  I can think of all the settings all the

  options the more the better but you need

  as a developer to have that confidence

  to be able to say like this is what my

  app is this is the way it should work

  and to then just be able to go with that

  and make that work sounds good alright

  that's it for this week thank you very

  much for listening please rate us and /

  or please recommend us an overcast and

  tell your friends about our show help us

  grow and we'll talk to you next week bye