Under the Radar

Under the Radar 58: Prerelease Testing


  welcome to under the radar a show about

  independent iOS app development I'm mark

  Ormond and I'm David Smith clear the

  radar is never longer than 30 minutes so

  let's get started so today we wanted to

  talk a little bit about testing about

  that process we go through towards the

  end of developing something where we try

  and take it from mostly working to

  actually working or fully working and

  this is particularly relevant and

  interesting for me at this point because

  I am in that very final of phases for my

  next step when we thought about whole

  bit about last week that I'm in the

  phase now where functionally and you

  know from a feature perspective I've

  kind of drawn the line and said this is

  the app this is as far as I'm gonna go

  with it any other features or any other

  things that come up are gonna be in the

  next version you know version 1.1 or

  I'll push it down the road all um now

  now going to be doing is fixing bugs and

  testing and finding all the little

  issues and problems and it seemed like

  something that would be an interesting

  topic to kind of walk through because

  ultimately the goal is to make sure that

  we ship things that are you know quality

  are not you know riddled with bugs or

  having all kinds of problems and it's a

  tricky thing I think especially at from

  an indie perspective and I think we're

  definitely fair to say but I'm

  definitely talking about this from the

  the more of the indie perspective rather

  than you know if you are in a large

  corporate environment where your app is

  being worked on by lots of different

  people you may actually have an entire

  like QA department whose job it is to

  test your application like I used to

  when a job I used to work at you know

  they had there's an a room full of par

  like 20 people who all day just sat

  there using the product testing is

  trying all the crazy head you know edges

  and they had these big test plans that

  they had to go through before we did a

  build and like that is a very different

  world then I think either of us work in

  and so we're not talking about that kind

  of testing like that is a a discipline

  unto itself it is useful and important

  but for an independent I think that's

  just not the reality both from like a

  resources perspective like I don't have

  the time

  - realistically do that you know to do

  20 people times 40 hours or a week of

  testing like that's just impossible no

  so I'm not sure if it's necessary and so

  this is coming into like a lot of

  there's a certain category of bug that

  I'm less worried about this is sort of

  the in more like the integration bug

  rather than the like functional bug

  where sometimes you just have bugs that

  come up because you had five people

  working on a product and they're you

  know one person changed something here

  someone else changed it over there and

  obviously myself in different time

  domains is different like I can be you

  know like two weeks ago me and my

  current me can be conflicting but that

  happens less often but either way like

  being independent it's still something

  that I have to do and it's a process

  that I go through now and it seems like

  kind of a good thing to walk through but

  I don't do it in a formal way

  marco do you do it any kind of formal

  testing or processes that you have

  around putting something out or do you

  just kind of try it out a bunch and see

  if it works I mostly just try out a

  bunch I mean and so so we should clarify

  also that test-driven development is not

  what we're really talking about neither

  of us really do that we've talked about

  it briefly before and I don't think

  either of us really has a strong opinion

  on it except just to say that we don't

  do it mostly as a choice because of our

  resources and our style of coding

  you know just being single person shops

  it makes it harder to justify the the

  additional amount of time it takes to

  write all the tests and for test-driven

  development so that's not what we're

  talking about here today at least so

  yeah so when it comes to whatever

  testing people would call what we used

  to call testing just like you or other

  people like actually just using your app

  and trying out all the different things

  you know trying to turn you know testing

  out edge cases you know the things that

  a good tester a QA person would do that

  is that's what we're talking about today

  and so my answer to that is generally I

  I mostly rely on my own usage and their

  and reports from beta testers in order

  to to do this and

  this has pluses and minuses I mean

  obviously it this is a less disciplined

  approach than having like a formal unit

  testing process a formal UI testing

  process having dedicated testing people

  or services I should look more into

  those things I haven't to date and and

  it mostly has not caused problems

  there's and and part of this is because

  of the way I developed the app in the

  first place I mean step number one is I

  only make apps that I use if you are

  making an app that you don't use

  yourself on a regular basis and there's

  lots of reasons why you might want to do

  this you know if you're a consultant

  obviously that's a big one

  but you know a lot of people have these

  needs if that's the kind of need that

  that you're working with or you don't

  use the app I really don't know what to

  tell you I think things like unit

  testing make more sense there or having

  having like dedicated QA people or

  services that you can hire to do that

  but because I use the app myself and and

  I also tend to only implement features

  that I will use and and this is not I I

  don't do that 100% of the time like

  there are certain features like in my

  nitpicky details feature about the

  showing the number of unplayed unplayed

  episode as a red badge on the icon of

  the app I think you can tell by the

  description text in that interface that

  I don't want to use this feature but

  enough people requested it and it was a

  small enough feature I mean that

  features something like four lines of

  code it's it's it's a very very very

  tiny little feature and it satisfied

  large user demand a lot of people

  requested that feature always before I

  added it and so it satisfies a large

  user demand it was not that much work to

  do and it's very easy for me to test

  every so often even though I don't use

  it myself so that's something I added

  anyway despite not using it myself but

  otherwise I try not to add features that

  I won't use myself very often if at all

  in part because I don't care and in part

  because it makes it harder for me to

  maintain the quality of the app because

  I won't see problems in day-to-day

  testing like one of the one of the

  issues that I faced recently is I have a

  pretty severe carplay bug in the most

  recent build and I'm working on fixing

  that now I should have it for the next

  one but

  I don't have a car play vehicle we don't

  own a car play vehicle in our household

  and we actually are about to get one in

  about a month and that'll probably help

  although it won't be my car which is

  unfortunate but it's at least my wife

  will have one so I will have hardware to

  test on sometimes but like car plays

  this whole area of the app that I never

  see because I don't drive a car that's

  car play enabled and so there are going

  to be all these different areas where my

  own day-to-day usage is not going to

  cover it or is not going to reveal

  subtle problems at least and so for

  those kind of uses I really just rely on

  beta testers and reports from users

  hopefully you know hopefully before I

  release these versions that have these

  that have problems but you know

  sometimes sometimes they get out there

  and so in order to in order to minimize

  the damage I do try to at least like at

  least cover the the biggest most common

  uses of the app in my own formal or

  informal testing so for example what I'm

  releasing a big version I will almost

  always if I remember to which isn't

  always the case which is a problem but

  actually probably this is again this is

  like one of the reasons why it might be

  useful to have like a checklist or

  something but or you know a more formal

  structure of doing these things or an

  automated way of doing these things but

  one of the things that I do in my app is

  to to basically start fresh start from a

  brand new user every time I do a release

  so like take a device that doesn't have

  the app on an install a fresh version of

  the app and then try creating a new

  account and starting all over again and

  this helps for a number of reasons

  obviously that's a very important thing

  for your app to make sure it works at

  all and then secondly it shows me the

  onboarding experience it shows me the

  first run experience a lot like I see

  that whenever I whenever I do we release

  I I will see the first run experience

  and at least once during that process so

  that kind of helps remind me of any

  shortcomings in it or it helps you know

  helps show me if anything's broken about

  it or if if things aren't quite right

  you know it's so that's nice you know in

  that way as well but this is all a very

  long way of saying basically the way I

  do testing is I just use the app a lot

  myself before I let anybody else use it

  and then when it gets close to release I

  do beta tests

  and I can talk a little bit more about

  that maybe a little bit later but the

  beta test I use tend to find almost any

  other reigning problems yeah I think

  it's to is it's important to say like I

  mean I think my testing strategy is very

  similar to yours it's the I use the app

  a lot and I try and sort of use it like

  as I get closer I tend to kind of use it

  slightly and anger like I use it in a

  way that it's kind of trying to break it

  like I'm bashing on all the buttons and

  like what if I change this setting

  really quickly back and forth back and

  forth like is anything weird gonna

  happen like you can use it in that way

  and then yeah like you said it's that

  kind of testing where and the honestly

  what I usually do is for the like trying

  out a new user testing it's the you know

  I - I currently use an iPhone 7 that's

  the phone that I use like during

  development like 98% of my development

  happens on that phone and so before I

  launch I need to you know very least run

  it unlike a 5s and on a 7 plus I'm just

  in terms of to make sure that there's

  not any weird UI bugs or issues or

  sometimes even just weird there's

  performance things obviously going if

  you go back to like a 5 or a 5s they're

  much slower and so that's a I find a

  great place to do the kind of on board a

  2 on board testing because you know I'm

  starting fresh on those devices half the

  time anyway that will also say one thing

  that has bitten me many many times in

  the past and so now I always try and

  make sure I do too is I do the fresh

  user onboarding experience process but I

  do that running the old version of the

  app like the existing I could go to the

  App Store and download whatever this is

  whatever the current version of the app

  is from the App Store on board a fresh

  user and then install the version that

  I'm trying to verify and test with nice

  because often what I found is there's

  issues that you can run into that like

  the you know so usually it's a data

  issue or a settings issue or something

  that isn't being translated over

  correctly and obviously the majority of

  your users are going to be are like are

  coming from that previous version and

  then running your new version and so if

  there's any issues there you want to

  catch them but that kind of a this kind

  of approach of just taking a lot of

  devices like on my desk

  I have a bunch of different you know

  there's all the different iPhones like I

  never sell an iPhone whenever I get a

  new one and so I have all that all the

  different testing devices available to

  me that way

  you know similarly like right now I'm

  wearing two Apple watches I have a 38

  millimeter Apple watch on my right hand

  and a 42 millimeter repla watch on my

  left hand and they're paired two

  separate phones many seven sizes and I'm

  just running the app that I'm testing

  constantly now like I just he keep this

  process of kind of banging up against it

  and if in general I think that it's it's

  you'll it's amazing how good much how

  good the coverage you can get from it

  that kind of a informal but like

  constant high-volume testing like maybe

  it's not as rigorous and so it loses

  something there but it definitely

  benefits from just doing it all the time

  and if you build apps that you use you

  certainly have that benefit at the very

  least of exposure to it yeah I mean one

  other note on the devices before I


  app review does not seem to test much if

  at all on a variety of devices as far as

  I can tell they test on one device only

  and usually it's a recent one so as far

  as I can tell so like a lot of times

  people people will have issues where

  like they'll get an update into the

  store approved that like crashes on

  launched on the iPhone 5s or something

  like that right because it's like an

  older device that is still supported and

  it still runs on and that your customers

  might still use but it doesn't work for

  you know it and I purview didn't catch

  it you know and so there's you can't

  depend on app review to catch like the

  big obvious Crashers because if it

  happens on anything any conditions that

  they're not trying including any other

  device besides whatever one they happen

  to be using you're not going to have a

  good time so that's that's a very very

  important to note and it's also probably

  worth saying that app review is not QA

  like app review is this very like course

  sieve that every now and then will catch

  problems and I love it when they do this

  but like that is not at all what they're

  doing they're not really looking to test

  your app to find problems they're too

  kind of running it through at a very

  different level and so I would not as

  certainly isn't one of those like well

  I'll just

  if a perv you can find any bugs and if

  they don't I'm sure it's fine like that

  is a tear there'll be a terrible

  approach to take to you with your app

  yeah it's kind it's like the seat belt

  on a plane it's like that's that is not

  the safety mechanism you should be

  relying on this you should definitely

  like it basically if a preview rejects

  your app for a crash you have a serious

  problem in your QA process it's like

  whatever you submit to ever view should

  not be in that kind of state where it

  would crash in like the first four

  minutes of somebody using it once you

  know but you know overall like and I

  should also you know suggest to that if

  you are going to rely so heavily as we

  do on self testing basically and again I

  I do I apologize greatly to people who

  enjoy formal testing and automated

  testing and unit testing and then 100%

  test coverage I apologize because this

  this entire discussion must be driving

  you totally crazy hi Casey

  but ultimately it's hard for single

  person developers to really justify the

  additional time and/or expense that more

  rigorous testing methods bring on for

  things like this so now that's a lot of

  qualifiers in that sentence obviously if

  you have a larger staff if you have a

  lot of money if your app is more

  important and then like you know then

  then you know like a sleep tracker for

  your wrist or for the podcast player

  like if you're really bad things happen

  if you like lose people's important data

  or you know if somebody will be

  seriously put out or hurt if your app

  breaks obviously that's a place where

  you need a more formal testing procedure

  and and framework in place but most

  people who make iOS apps aren't working

  in that kind of capacity and and not

  only don't have those resources but it

  would actually be be kind of a bad way

  to spend resources basically like

  tripling all of your work and this is a

  huge debate in programmer circles of

  whether test-driven development

  is really more work or whether it ends

  up saving you more work in the long run

  I honestly don't care which one of those

  it is I think it's probably unknowable

  because it's probably like the answer

  like everything else is probably it

  depends but but ultimately

  this isn't the way we do things and and

  I think there's a lot of developers out

  there who who do things more like what

  we do anyway that being said what more

  thing before I go into the sponsor for

  the week and that is if you're relying

  so much on self testing as we do if your

  app has like you know different modes or

  settings or ways you can do things

  change it up like in your in your day to

  day usage change it up every so often

  and spend a lot of time in each kind of

  condition so for instance overcast has

  dark mode or light mode and during

  development I usually switch about once

  a week between which one of those I'm

  using because I want to kind of get used

  to it and live in it and make sure that

  I'm that I'm designing and making the

  best app I can for both of those things

  same thing with streaming versus

  downloading like you know I wrote this

  whole streaming engine I hardly ever use

  it and that's kind of a problem so

  basically I force myself to change my

  own preference and and spend a good

  amount of time like a week or a month in

  streaming mode in the middle of

  development cycles where I would

  otherwise be in downloading mode you

  know just just to make sure I'm getting

  a really good idea of like these vastly

  different modes that my app can be in so

  that my own testing is more effective

  anyway we were sponsored this week by

  Pingdom Pingdom can help remove the need

  for you to have somebody who

  automatically logs in your website every

  minute and refreshes it to make sure

  it's up because Pingdom does that

  automatically with their testing

  framework see for yourself at Pingdom

  comm slash radar Pingdom is an awesome

  monitoring service for websites and web

  services and when you use offer code

  radar at checkout you get a holiday

  special of 50% off your first invoice

  this is an awesome deal usually 20% now

  it's 50 percent for the holidays so

  check it out at Pingdom comm slash radar

  Pingdom offers powerful easy to use

  monitoring tools and services for your

  website and web servers web sites are

  very sophisticated these days and you

  need you know there's all sorts of

  things that can go wrong and they know

  this they see it because they monitor it

  for lots of people

  you need Pingdom to be monitoring your

  site they can monitor as often as every

  minute by requesting pages by running

  through checkouts by logging in by doing

  you know transactions that they can do

  all sorts of things all

  simulated from there more than 70 global

  test servers and they can do it as often

  as once a minute and if anything goes

  down they can alert you via so many

  different means whatever you want you

  can have email text push notification or

  all an income any combination of these

  things and you can set the granularity

  it's incredibly customizable I've been

  using Pingdom now for oh geez I think

  eight years nine years something crazy

  it's a very long time because I've used

  it way before they were sponsor way

  before I was even a podcaster I was

  using Pingdom to monitor tumbler back in

  the day and and I just love them I used

  them constantly highly recommended that

  you use Pingdom for your all your

  monitoring needs it is really the best

  and if you run any kind of website or

  web service the last thing you want is

  for strangers on Twitter to be telling

  you when your site is down when you

  didn't already know that like you do not

  want to be learning that from Twitter

  you want to know when your site is down

  you want to be the first to know and

  then you want to be able to go in and

  fix it before all the people on Twitter

  see it and start bugging you about it

  and with Pingdom you can because Pingdom

  will alert you the moment things go down

  with that up to one minute granularity

  that's really incredible and they are so

  fast and so reliable

  I love Pingdom check them out yourself

  go to Pingdom comm slash of radar you

  will get a 14-day free trial and when

  you enter offer code radar at checkout

  you will get a holiday special of 50%

  off your first invoice check it out

  today no be the first to know when your

  site is down thank you very much to

  Pingdom for sponsoring our show so I

  think for the last part of the show it

  seemed like some that may actually make

  sense for us to focusing on is a bit of

  the actual like the details of what this

  kind of user testing approach that we've

  just been talking about it looks like

  and for me what that usually means is so

  I get into this phase now where I'm

  doing this final level of testing and

  I'm just constantly trying to think of

  things to try I'm trying them out I'm

  using the app in different circumstances

  in different places like I'll be using

  it on different networks like hey I'm

  out on a cell network I'm at home now

  and so whenever I find an issue you're

  inevitably you'll find a bug you'll see

  something that you're like huh that's a

  little interesting what I found is a

  very important thing is to have some

  kind of reliable way of collecting

  all these little issues and bugs and

  problems and for me usually that is I

  have I'll have a list in OmniFocus that

  I am just adding items to and anytime I

  see something usually I'll grab a

  screenshot of it and if it's obvious

  I'll just attach the screenshot to the

  task in OmniFocus or if not I'll take it

  and run it through what used to be an

  app that you made Marco which used to be

  called buckshot what I think is now

  called pin point mmm which is a great

  little tool we have a link in the show

  notes to it but it's just a little thing

  for just drawing arrows basically or

  circling things um it's great for screen

  shot sort of when you're like huh

  you know there's this weird graphics

  glitch or something so you can mark it

  up add it to it to get and then you know

  I go through all of like and when I'm in

  the face now like this morning I sat

  down I had the head a list about five or

  six little bugs I've noticed yesterday I

  went through and I'm just knocking them

  out and it's kind of like this I work

  all the way down till I have no bugs and

  then I make a build and then I go

  through the process again and you kind

  of just keep cycling through collecting

  as you go and for me that kind of just

  that kind of process of just keeping

  this running log tea seems to work well

  it I like doing it that way rather than

  kind of you know having like sitting

  down and just testing for hours never

  really works for me I find that I get

  kind of bored or I start doing the same

  patterns over and over again and so it's

  better I think for me I'd try and do it

  spread it out throughout the day and

  obviously if it's an application that

  you would use normally and naturally

  throughout the day then this makes it

  easier but otherwise you just kind of

  have to sort of remind yourself to do it

  and this is also a little bit tricky if

  you have an app that requires a certain

  amount of time for testing so like for

  example I have a sleep tracker like in

  order for it to really like like I

  cannot compressed beyond just running it

  on multiple devices the testing of that

  because it takes about eight hours to

  have eight hours of data collection and

  like I have simulated stuff in the app

  for you know when I'm doing like debug

  testing that I can kind of like create a

  night's data and run my algorithms on it

  but for actual testing there's nothing

  you can really do other than wearing

  local watches which is what I do like

  when I'm doing extensive sleep you know

  testing like I'll have all my watches on

  my wrist during the night and I'm

  running it you know three or four times

  but you know sort of in the same way I

  imagine for you with overcast like

  there's a certain amount if you just

  have to play a lots of audio because if

  you don't you're never going to find all

  the weird audio issues or glitches or

  problems there's nothing that you can

  really do to speed that up or compress

  that testing yeah basically I mean I

  like I just find ways to shove podcasts

  into more of my life so so like you know

  like like ivories and and also to test

  out things like syncing you know like

  this my syncing engine took forever to

  really figure it mean took me almost a

  year to really nail syncing and one of

  the ways I started finally getting

  better at that was I started needing it

  more because I started using an iPad to

  play podcast in the kitchen and I used

  my iphone to play them everywhere else

  so I will I was very frequently

  switching between multiple devices that

  would use the sink back-end and just by

  doing that myself but by needing that

  myself for a few months I really fixed a

  ton of sink problems and got it to a

  pretty good place now and and so yeah I

  mean it really is just a matter of just

  you know again it's like if you only

  make apps that you will use a lot this

  becomes a lot easier if if you don't

  make apps that use a lot this you know

  you're gonna have to find other ways to

  do this effectively but it really works

  that great for me and and I I will again

  once again go in briefly about beta

  testing also and and we pretty sure we

  did a whole episode on it if not we will

  and we should but I think we already did

  we have so many episodes now I've kind

  of lost track

  but I do rely a lot on beta testers to

  to really do like a final a final check

  on the app they aren't my number right

  there are my first testers I am my first

  tester and I I will usually use changes

  to the app for weeks or even months

  before I give it to anybody else to test

  just because I really want to make sure

  that I know I have feel for it make sure

  it's good make sure it's the right

  choice and then I will finally share

  with testers like right before the

  public gets it but basically beta

  testers can can help a little bit not a

  ton but but a little bit I will also add

  that the what you mentioned earlier

  about your workflow about taking taking

  screenshots and then adding arrows and

  then adding them to OmniFocus and things

  like that there if you want to go off

  the deep end there are lots of crazy

  tools and frameworks that automate that

  process to varying degrees and have all

  sorts of cool integrations so there are

  some like you know like I made bug shot

  kit which is basically shoved bug shot

  into an app as you could do things like

  it will detect screenshots being taken

  and then automatically bring up a modal

  controller with the screenshot setting

  here mark this up and send it in to

  report a bug you know and you can you

  can give that to all your testers and

  you can use that yourself during your

  test process and so you can basically

  have like easy or automated bug

  reporting right right embedded into your

  app and there's again there's lots of

  ways to do this I'm not gonna recommend

  any particular one because I haven't

  looked but there are many diseases exist

  now and you can you can really go off

  the deep end with like the levels of

  integrations you can have it like post

  to slack and then it make an entry in

  your bug tracker like all sorts of crazy

  things you can do now if you really want

  to get into all that stuff so the you

  know the options are pretty good for now

  for like tools and frameworks and things

  like that to automate like the the

  tedious stuff but for the for the real

  just using the app really just again

  just using it a lot yourself is a very

  very effectively to do it obviously as I

  know automated things exist sorry Casey

  again but just using your app yourself

  will show you so much and will tell you

  so much and will tell you will show you

  things like you know this this workflow

  is kind of clunky or this animation

  doesn't really look right or this I I

  really need this text to be lighter when

  it's really sunny outside or something

  like that like there's a large category

  of either problems or or good and bad

  design decisions for which just using

  yourself is really the only way to know

  and the really the only way to test and

  that isn't true for everything obviously

  sorry again Casey but that's true for

  quite a lot that we do in our apps and I

  think with other thing I would say with

  beta testing that's an important thing

  too it's another use for beta testing is

  the importance of not overly preparing

  your beta testers for what you know for

  the app so in terms of trying to give

  him giving them an experience that is

  similar to the experience that your app

  store users are going to have

  and so when they download the app and

  they're like what do I do or this is

  really confusing like some of the best

  feedback I get from beta testers is sort

  of accidental where they're not actually

  they don't know it but they're telling

  me something is fundamentally flawed or

  is really confusing where they're asking

  a question that like it implies missing

  a fundamental misunderstanding about

  something and that kind of feedback is

  always really good and so I try whenever

  I'm beta testing to just almost like

  blindly send out the betas to people

  where I'm not giving them too much

  preparation or tell are like giving them

  step-by-step walkthroughs or details

  it's very much like as though you just

  downloaded this from the App Store you

  know maybe you've seen a screenshot or

  two maybe you've had a high-level

  discussion but you're working it out

  just as you would as a regular user and

  that's a great test because have you as

  the developer of the app of ously I know

  how everything works everything is

  intuitive to me because that's the way I

  made it but it is often not that not the

  case for your users and so one of the

  best uses for beta testing isn't it's

  great when they find you know actual

  bugs and issues and problems but it's

  also it's really useful for capturing

  those kind of like it's like if you're

  asking this question you clearly have

  not communicated the purpose of this

  feature correctly or I've hidden it in a

  weird way or you think it should do

  something and it does the opposite and

  those are the only thing that is almost

  exclusively possible to find in beta

  testing because no matter how hard you

  try to kind of like reset your brain to

  what if I was a fresh user opening the

  application it's kind of impossible

  where you know I the current app I'm

  working on I have been working on it

  essentially every day for months at this

  point so I'm so in it that I just don't

  see it's like I'm so far into the woods

  that I have no idea that there is a

  forest like all I have is trees like it

  is deep down into it I mean and maybe

  when I come back to it and a couple of

  you know like I'll put the app to the

  side for a little bit and I'll come back

  to it and maybe a month maybe I'll I

  will have that kind of fresh experience

  but until that time the only way I can

  get that is with beta testing and so it

  is definitely a valuable and important

  aspect of that

  all right and I guess we're gonna see

  your new app pretty soon right yeah it

  should be hopefully launching next week

  so should be kind of exciting and I

  expect to go home sure we'll talk about

  on the show how the lunch went and kind

  of some of the details of it as the app

  goes but it is very exciting to get to

  this point and I will say as you know

  it's like stalking all the other

  independent developers out there it's

  like like I just put in the work because

  there's definitely been times in this

  project where I was like oh gosh what

  have I gotten myself into maybe I should

  just like can this project move on but

  there is hope eventually we'll get there

  eventually we'll ship and it is


  when you get there can't tell you a

  secret to the audience you can I know

  what this is and I've used it and it's

  really good so look forward to it anyway

  thanks a lot for listening everybody

  we're out of time this week and we'll

  talk to you next week bye