58: Prerelease Testing
00:00:00
◼
►
Welcome to Under the Radar, a show about independent iOS app development.
00:00:04
◼
►
I'm Mark Orment.
00:00:05
◼
►
And I'm David Smith.
00:00:06
◼
►
Under the Radar is never longer than 30 minutes, so let's get started.
00:00:10
◼
►
So today we wanted to talk a little bit about testing, about that process we go through
00:00:16
◼
►
towards the end of developing something where we try and take it from mostly working to
00:00:22
◼
►
actually working or fully working.
00:00:25
◼
►
And this is particularly relevant and interesting for me at this point because I am in that
00:00:29
◼
►
very final of phases for my next app, the one we talked a little bit about last week,
00:00:36
◼
►
that I'm in this phase now where functionally and from a feature perspective, I've kind
00:00:41
◼
►
of drawn the line and said, this is the app.
00:00:43
◼
►
This is as far as I'm going to go with it.
00:00:45
◼
►
Any other features or any other things that come up are going to be in the next version,
00:00:49
◼
►
version 1.1, or I'll push it down the road.
00:00:52
◼
►
All I'm now going to be doing is fixing bugs and testing and finding all the little issues
00:00:57
◼
►
and problems.
00:00:59
◼
►
And it seemed like something that would be an interesting topic to kind of walk through
00:01:03
◼
►
because ultimately the goal is to make sure that we ship things that are quality, are
00:01:10
◼
►
not riddled with bugs, they're having all kinds of problems.
00:01:14
◼
►
And it's a tricky thing, I think, especially from an indie perspective.
00:01:18
◼
►
And I think it's really fair to say, I'm definitely talking about this from the more
00:01:23
◼
►
of the indie perspective rather than if you are in a large corporate environment where
00:01:28
◼
►
your app is being worked on by lots of different people.
00:01:31
◼
►
You may actually have an entire QA department whose job it is to test your application.
00:01:36
◼
►
I used to work at a job I used to work at.
00:01:40
◼
►
There was a room full of probably 20 people who all day just sat there using the product,
00:01:46
◼
►
testing it, trying all the crazy edges, and they had these big test plans that they had
00:01:50
◼
►
to go through before we did a build.
00:01:52
◼
►
That is a very different world than I think either of us work in.
00:01:55
◼
►
And so we're not talking about that kind of testing.
00:01:58
◼
►
That is a discipline unto itself.
00:02:01
◼
►
It is useful and important.
00:02:02
◼
►
But for an independent, I think that's just not the reality, both from a resources perspective.
00:02:09
◼
►
I don't have the time to realistically do that, to do 20 people times 40 hours a week
00:02:18
◼
►
That's just impossible.
00:02:19
◼
►
And also, I'm not sure if it's necessary.
00:02:22
◼
►
And so this is coming into a certain category of bug that I'm less worried about.
00:02:29
◼
►
This is sort of the integration bug rather than the functional bug, where sometimes you
00:02:35
◼
►
just have bugs that come up because you had five people working on a product, and one
00:02:40
◼
►
person changed something here, someone else changed it over there.
00:02:42
◼
►
And obviously, myself, in different time domains, is different.
00:02:46
◼
►
I can be, like two weeks ago, me and my current me can be conflicting, but that happens less
00:02:52
◼
►
But either way, being independent, it's still something that I have to do.
00:02:56
◼
►
And it's a process that I go through now.
00:02:59
◼
►
And it seems like kind of a good thing to walk through.
00:03:01
◼
►
But I don't do it in a formal way.
00:03:03
◼
►
Marco, do you do any kind of formal testing or processes that you have around putting
00:03:08
◼
►
something out, or do you just kind of try it out a bunch and see if it works?
00:03:12
◼
►
I mostly just try out a bunch.
00:03:15
◼
►
And so we should clarify also that test-driven development is not what we're really talking
00:03:21
◼
►
Neither of us really do that.
00:03:23
◼
►
We've talked about it briefly before.
00:03:25
◼
►
And I don't think either of us really has a strong opinion on it, except just to say
00:03:28
◼
►
that we don't do it mostly as a choice because of our resources and our style of coding,
00:03:35
◼
►
of just being single person shops.
00:03:38
◼
►
It makes it harder to justify the additional amount of time it takes to write all the tests
00:03:43
◼
►
for test-driven development.
00:03:45
◼
►
So that's not what we're talking about here today, at least.
00:03:48
◼
►
So when it comes to whatever testing people would call what we used to call testing.
00:03:55
◼
►
So yeah, just like you or other people actually just using your app and trying out all the
00:04:02
◼
►
different things, testing out edge cases, the things that a good tester or QA person
00:04:09
◼
►
That's what we're talking about today.
00:04:12
◼
►
And so my answer to that is generally I mostly rely on my own usage and reports from beta
00:04:19
◼
►
testers in order to do this.
00:04:23
◼
►
And this has pluses and minuses.
00:04:25
◼
►
I mean obviously this is a less disciplined approach than having like a formal unit testing
00:04:32
◼
►
process, a formal UI testing process, having dedicated testing people or services.
00:04:39
◼
►
I should look more into those things.
00:04:42
◼
►
I haven't to date and it mostly has not caused problems.
00:04:47
◼
►
And part of this is because of the way I developed the app in the first place.
00:04:50
◼
►
I mean step number one is I only make apps that I use.
00:04:55
◼
►
If you are making an app that you don't use yourself on a regular basis, and there's lots
00:05:00
◼
►
of reasons why you might want to do this.
00:05:02
◼
►
If you're a consultant, obviously that's a big one.
00:05:04
◼
►
But lots of people have these needs.
00:05:06
◼
►
If that's the kind of need that you're working with where you don't use the app, I really
00:05:10
◼
►
don't know what to tell you.
00:05:11
◼
►
I think things like unit testing make more sense there or having like dedicated QA people
00:05:16
◼
►
or services that you can hire to do that.
00:05:18
◼
►
But because I use the app myself, and I also tend to only implement features that I will
00:05:27
◼
►
And this is not, I don't do that 100% of the time.
00:05:30
◼
►
There are certain features like my nitpicky details feature about showing the number of
00:05:35
◼
►
unplayed episodes as a red badge on the icon of the app.
00:05:40
◼
►
I think you can tell by the description text in that interface that I don't want to use
00:05:44
◼
►
this feature.
00:05:45
◼
►
But enough people requested it and it was a small enough feature.
00:05:48
◼
►
I mean that feature is something like four lines of code.
00:05:50
◼
►
It's a very, very, very tiny little feature.
00:05:54
◼
►
And it satisfied large user demand.
00:05:57
◼
►
A lot of people requested that feature always before I added it.
00:06:01
◼
►
And so it satisfied a large user demand.
00:06:03
◼
►
It was not that much work to do.
00:06:05
◼
►
And it's very easy for me to test every so often, even though I don't use it myself.
00:06:08
◼
►
So that's something I added anyway, despite not using it myself.
00:06:12
◼
►
But otherwise, I try not to add features that I won't use myself very often, if at all.
00:06:18
◼
►
In part because I don't care, and in part because it makes it harder for me to maintain
00:06:23
◼
►
the quality of the app because I won't see problems in day to day testing.
00:06:28
◼
►
One of the issues that I faced recently is I have a pretty severe carplay bug in the
00:06:32
◼
►
most recent build.
00:06:34
◼
►
And I'm working on fixing that.
00:06:35
◼
►
I should have it for the next one, but I don't have a carplay vehicle.
00:06:38
◼
►
We don't own a carplay vehicle in our household.
00:06:40
◼
►
And we actually are about to get one in about a month.
00:06:43
◼
►
And that'll probably help.
00:06:44
◼
►
Although it won't be my car, which is unfortunate.
00:06:46
◼
►
But at least my wife will have one, so I will have hardware to test on sometimes.
00:06:51
◼
►
But carplay is this whole area of the app that I never see because I don't drive a
00:06:56
◼
►
car that's carplay enabled.
00:06:57
◼
►
And so there are gonna be all these different areas where my own day to day usage is not
00:07:02
◼
►
going to cover it or is not gonna reveal subtle problems at least.
00:07:08
◼
►
And so for those kind of uses I really just rely on beta testers and reports from users.
00:07:13
◼
►
Hopefully it's before I release these versions that have problems, but sometimes they get
00:07:21
◼
►
So in order to minimize the damage I do try to at least cover the biggest most common
00:07:29
◼
►
uses of the app in my own formal or informal testing.
00:07:33
◼
►
So for example, when I'm releasing a big version I will almost always, if I remember
00:07:38
◼
►
to which isn't always the case, which is a problem, but I should probably, again this
00:07:42
◼
►
is one of the reasons why it might be useful to have a checklist or something, or a more
00:07:47
◼
►
formal structure of doing these things or an automated way of doing these things.
00:07:50
◼
►
But one of the things that I do in my app is to basically start fresh, start from a
00:07:56
◼
►
brand new user every time I do a release.
00:08:00
◼
►
Take a device that doesn't have the app on it, install a fresh version of the app, and
00:08:04
◼
►
then try creating a new account and starting all over again.
00:08:08
◼
►
And this helps for a number of reasons.
00:08:10
◼
►
Obviously that's a very important thing for your app to make sure it works at all.
00:08:15
◼
►
And then secondly it shows me the onboarding experience, it shows me the first run experience
00:08:21
◼
►
Like I see that whenever I do a release.
00:08:23
◼
►
I will see the first run experience at least once during that process.
00:08:27
◼
►
So that kind of helps remind me of any shortcomings in it, or it helps show me if anything's
00:08:33
◼
►
broken about it, or if things aren't quite right.
00:08:36
◼
►
So that's nice in that way as well.
00:08:38
◼
►
But this is all a very long way of saying basically the way I do testing is I just use
00:08:44
◼
►
the app a lot myself before I let anybody else use it.
00:08:47
◼
►
And then when it gets close to release I do beta tests.
00:08:52
◼
►
And I can talk a little bit more about that maybe a little bit later, but the beta tests
00:08:56
◼
►
I use tend to find almost any other ruining problems.
00:09:00
◼
►
I think it's important to say like, I mean I think my testing strategy is very similar
00:09:05
◼
►
It's the, I use the app a lot and I try and sort of use it, like as I get closer I
00:09:13
◼
►
tend to kind of use it slightly in anger.
00:09:15
◼
►
Like I use it in a way that is kind of trying to break it.
00:09:18
◼
►
Like I'm bashing on all the buttons and like what if I change this setting really
00:09:21
◼
►
quickly back and forth, back and forth.
00:09:23
◼
►
Like is anything weird going to happen?
00:09:24
◼
►
Like you can use it in that way.
00:09:27
◼
►
And then yeah like you said it's that kind of testing where, and honestly what I usually
00:09:30
◼
►
do is for the like trying out a new user testing it's the, you know I currently use an iPhone
00:09:38
◼
►
That's the phone that I use like during development like 98% of my development happens
00:09:42
◼
►
on that phone.
00:09:43
◼
►
And so before I launch I need to at the very least run it on like a 5s and on a 7 plus.
00:09:51
◼
►
Just in terms of to make sure that there's not any weird UI bugs or issues or sometimes
00:09:55
◼
►
even just weird, there's performance things obviously going if you go back to like a 5
00:09:59
◼
►
or a 5s they're much slower.
00:10:01
◼
►
And so that's I find a great place to do the kind of onboarded, onboard testing because
00:10:07
◼
►
you know I'm starting fresh on those devices half the time anyway.
00:10:10
◼
►
I will also say one thing that has bitten me many, many times in the past and so now
00:10:15
◼
►
I always try and make sure I do too is I do the fresh user onboarding experience process
00:10:22
◼
►
but I do that running the old version of the app like the existing, like I go to the app
00:10:28
◼
►
store and download whatever this is, whatever the current version of the app is from the
00:10:32
◼
►
app store, onboard a fresh user and then install the version that I'm trying to verify and
00:10:41
◼
►
Because often what I've found is there's issues that you can run into that like the,
00:10:43
◼
►
you know, usually it's a data issue or a settings issue or something that isn't being
00:10:48
◼
►
translated over correctly and obviously the majority of your users are going to be, are
00:10:54
◼
►
coming from that previous version and then running your new version and so if there's
00:10:58
◼
►
any issues there you want to catch them.
00:11:00
◼
►
But that kind of approach of just taking a lot of devices like on my desk I have a bunch
00:11:05
◼
►
of different, all the different iPhones, like I never sell an iPhone whenever I get a new
00:11:10
◼
►
one and so I have all the different testing devices available to me that way.
00:11:15
◼
►
You know, similarly like right now I'm wearing two Apple watches.
00:11:18
◼
►
I have a 38 millimeter Apple watch on my right hand and a 42 millimeter Apple watch on my
00:11:23
◼
►
left hand and they're paired to separate phones, running seven sizes and I'm just running the
00:11:28
◼
►
app that I'm testing constantly now.
00:11:30
◼
►
Like I just keep this process of kind of banging up against it.
00:11:35
◼
►
And if in general I think like you said, it's amazing how good the coverage you can get
00:11:41
◼
►
from that kind of informal but like constant high volume testing.
00:11:48
◼
►
Like maybe it's not as rigorous and so it loses something there but it definitely benefits
00:11:53
◼
►
from just doing it all the time and if you build apps that you use, you know, you certainly
00:11:57
◼
►
have that benefit at the very least of exposure to it.
00:12:00
◼
►
- Yeah, I mean one other note on the devices before I forget, app review does not seem
00:12:07
◼
►
to test much if at all on a variety of devices.
00:12:11
◼
►
As far as I can tell they test on one device only and usually it's a recent one.
00:12:15
◼
►
So as far as I can tell, so like a lot of times people will have issues where like they'll
00:12:20
◼
►
get an update into the store approved that like crashes on launch on the iPhone 5S or
00:12:25
◼
►
something like that, right?
00:12:26
◼
►
Because it's like an older device that is still supported and then it still runs on
00:12:29
◼
►
that your customers might still use but it doesn't work for, you know, and app review
00:12:35
◼
►
didn't catch it, you know.
00:12:36
◼
►
And so there's, you can't depend on app review to catch like the big obvious crashers because
00:12:42
◼
►
if it happens on anything, any conditions that they're not trying including any other
00:12:46
◼
►
device besides whatever one they happen to be using, you're not gonna have a good time.
00:12:50
◼
►
So that's very, very important to note.
00:12:52
◼
►
- And it's also probably worth saying that app review is not QA.
00:12:56
◼
►
Like app review is this very like course sieve that every now and then will catch problems
00:13:03
◼
►
and I love it when they do this but like that is not at all what they're doing.
00:13:07
◼
►
They're not really looking to test your app to find problems.
00:13:11
◼
►
They're kind of running it through at a very different level.
00:13:14
◼
►
And so I would not, it certainly isn't one of those like, well, I'll just see if app
00:13:18
◼
►
review can find any bugs and if they don't, then I'm sure it's fine.
00:13:21
◼
►
Like that is a terrible, that would be a terrible approach to take to with your app.
00:13:25
◼
►
- Yeah, it's like the seatbelt on a plane.
00:13:27
◼
►
It's like that is not the safety mechanism you should be relying on.
00:13:31
◼
►
You should definitely like, basically if app review rejects your app for a crash, you have
00:13:37
◼
►
a serious problem in your QA process.
00:13:39
◼
►
Like whatever you submit to app review should not be in that kind of state where it would
00:13:42
◼
►
crash in like the first four minutes of somebody using it once, you know.
00:13:47
◼
►
But you know, overall like, and I should also suggest too that if you are going to rely
00:13:52
◼
►
so heavily as we do on self-testing basically.
00:13:57
◼
►
And again, I do, I apologize greatly to people who enjoy formal testing and automated testing
00:14:04
◼
►
and unit testing and 100% test coverage.
00:14:06
◼
►
I apologize because this entire discussion must be driving you totally crazy.
00:14:12
◼
►
But ultimately it's hard for single person developers to really justify the additional
00:14:20
◼
►
time and or expense that more rigorous testing methods bring on for things like this.
00:14:28
◼
►
So now, that's a lot of qualifiers in that sentence.
00:14:31
◼
►
Obviously if you have a larger staff, if you have a lot of money, if your app is more important
00:14:37
◼
►
than like, you know, than like a sleep tracker for your wrist or for a podcast player.
00:14:45
◼
►
Like if really bad things happen, if you like lose people's important data or you know,
00:14:51
◼
►
somebody will be seriously put out or hurt if your app breaks, obviously that's a place
00:14:57
◼
►
where you need a more formal testing procedure and framework in place.
00:15:01
◼
►
But most people who make iOS apps aren't working in that kind of capacity.
00:15:06
◼
►
And not only don't have those resources, but it would actually be kind of a bad way
00:15:10
◼
►
to spend resources basically like tripling all of your work.
00:15:14
◼
►
And this is a huge debate in programmer circles of whether test driven development is really
00:15:18
◼
►
more work or whether it ends up saving you more work in the long run.
00:15:21
◼
►
I honestly don't care which one of those it is.
00:15:24
◼
►
I think it's probably unknowable because it's probably like the answer like everything
00:15:28
◼
►
else is probably it depends.
00:15:31
◼
►
But ultimately this isn't the way we do things and I think there's a lot of developers
00:15:35
◼
►
out there who do things more like what we do.
00:15:37
◼
►
Anyway, that being said, one more thing before I go into the sponsor for the week and that
00:15:41
◼
►
is if you're relying so much on self-testing as we do, if your app has different modes
00:15:47
◼
►
or settings or ways you can do things, change it up.
00:15:52
◼
►
In your day to day usage, change it up every so often and spend a lot of time in each kind
00:15:57
◼
►
of condition.
00:15:58
◼
►
So for instance, Overcast has dark mode or light mode and during development I usually
00:16:03
◼
►
switch about once a week between which one of those I'm using because I want to kind
00:16:07
◼
►
of get used to it and live in it and make sure that I'm designing and making the best
00:16:11
◼
►
app I can for both of those things.
00:16:12
◼
►
Same thing with streaming versus downloading.
00:16:14
◼
►
Like I wrote this whole streaming engine and I hardly ever use it and that's kind of
00:16:19
◼
►
So basically I force myself to change my own preference and spend a good amount of time
00:16:26
◼
►
like a week or a month in streaming mode in the middle of development cycles where I would
00:16:31
◼
►
otherwise be in downloading mode.
00:16:33
◼
►
Just to make sure I'm getting a really good idea of these vastly different modes that
00:16:39
◼
►
my app can be in so that my own testing is more effective.
00:16:42
◼
►
Anyway, we're sponsored this week by Pingdom.
00:16:46
◼
►
Pingdom can help remove the need for you to have somebody who automatically logs into
00:16:50
◼
►
your website every minute and refreshes it to make sure it's up because Pingdom does
00:16:53
◼
►
that automatically with their testing framework.
00:16:56
◼
►
See for yourself at Pingdom.com/radar.
00:17:00
◼
►
Pingdom is an awesome monitoring service for websites and web services and when you use
00:17:05
◼
►
offer code radar at checkout you get a holiday special of 50% off your first invoice.
00:17:12
◼
►
This is an awesome deal, usually 20% now it's 50% for the holidays.
00:17:15
◼
►
So check it out at Pingdom.com/radar.
00:17:19
◼
►
Pingdom offers powerful easy to use monitoring tools and services for your website and web
00:17:26
◼
►
Websites are very sophisticated these days and you need to, you know, there's all sorts
00:17:30
◼
►
of things that can go wrong and they know this, they see it because they monitor it
00:17:33
◼
►
for lots of people.
00:17:34
◼
►
You need Pingdom to be monitoring your site.
00:17:37
◼
►
They can monitor as often as every minute by requesting pages, by running through checkouts,
00:17:41
◼
►
by logging in, by doing, you know, transactions.
00:17:44
◼
►
They can do all sorts of things, all simulated from their more than 70 global test servers
00:17:49
◼
►
and they can do it as often as once a minute.
00:17:51
◼
►
And if anything goes down they can alert you via so many different means, whatever you
00:17:56
◼
►
You can have email, text, push notification or any combination of these things and you
00:18:00
◼
►
can set the granularity.
00:18:01
◼
►
It's incredibly customizable.
00:18:03
◼
►
I've been using Pingdom now for, oh geez, I think eight years, nine years, something
00:18:07
◼
►
crazy, it's a very long time because I've used it way before they were a sponsor, way
00:18:11
◼
►
before I was even a podcaster.
00:18:13
◼
►
I was using Pingdom to monitor Tumblr back in the day and I just love them.
00:18:17
◼
►
I use them constantly.
00:18:19
◼
►
Highly recommended that you use Pingdom for all your monitoring needs.
00:18:23
◼
►
It is really the best and if you run any kind of website or web service, the last thing
00:18:28
◼
►
you want is for strangers on Twitter to be telling you when your site is down when you
00:18:32
◼
►
didn't already know that.
00:18:33
◼
►
You do not want to be learning that from Twitter.
00:18:36
◼
►
You want to know when your site is down.
00:18:38
◼
►
You want to be the first to know and then you want to be able to go in and fix it before
00:18:42
◼
►
all the people on Twitter see it and start bugging you about it.
00:18:45
◼
►
And with Pingdom you can because Pingdom will alert you the moment things go down with that
00:18:49
◼
►
up to one minute granularity.
00:18:51
◼
►
That's really incredible and they are so fast and so reliable.
00:18:54
◼
►
I love Pingdom.
00:18:55
◼
►
Check them out yourself.
00:18:57
◼
►
Go to Pingdom.com/Radar.
00:18:58
◼
►
You will get a 14-day free trial and when you enter offer code radar at checkout, you
00:19:03
◼
►
will get a holiday special of 50% off your first invoice.
00:19:07
◼
►
Check it out today and you'll be the first to know when your site is down.
00:19:10
◼
►
Thank you very much to Pingdom for sponsoring our show.
00:19:13
◼
►
So I think for the last part of the show it seemed like something that may actually make
00:19:16
◼
►
sense for us to focus in on is a bit of the actual like the details of what this kind
00:19:23
◼
►
of user testing approach that we've just been talking about looks like.
00:19:28
◼
►
And for me what that usually means is so I get into this phase now where I'm doing this
00:19:32
◼
►
final level of testing and I'm just constantly trying to think of things to try.
00:19:36
◼
►
I'm trying them out.
00:19:37
◼
►
I'm using the app in different circumstances in different places like I'll be using it
00:19:41
◼
►
on different networks like hey I'm out on a cell network.
00:19:43
◼
►
I'm at home.
00:19:45
◼
►
And so whenever I find an issue, you know inevitably you'll find a bug.
00:19:47
◼
►
You'll see something that you're like huh that's a little interesting.
00:19:51
◼
►
What I found is a very important thing is to have some kind of reliable way of collecting
00:19:58
◼
►
all these little issues and bugs and problems.
00:20:00
◼
►
For me usually that is I have a list in OmniFocus that I am just adding items to.
00:20:08
◼
►
And so anytime I see something usually I'll grab a screenshot of it and if it's obvious
00:20:13
◼
►
I'll just attach the screenshot to the task in OmniFocus or if not I'll take it and run
00:20:19
◼
►
it through what used to be an app that you made Marco which used to be called Bugshot
00:20:24
◼
►
what I think is now called Pinpoint which is a great little tool.
00:20:27
◼
►
We have a link in the show notes to it but it's just a little thing for just drawing
00:20:29
◼
►
arrows basically or circling things.
00:20:32
◼
►
It's great for screenshot sort of when you're like huh you know there's this weird graphics
00:20:36
◼
►
glitch or something and so you can mark it up add it to a ticket.
00:20:39
◼
►
And then you know I go through all of like when I'm in the phase now like this morning
00:20:45
◼
►
I sat down I had the head of list about five or six little bugs I've noticed yesterday.
00:20:50
◼
►
I went through and I'm just knocking them out and it's kind of like this I work all the
00:20:55
◼
►
way down till I have no bugs and then I make a build and then I go through the process
00:20:59
◼
►
again and you kind of just keep cycling through collecting as you go.
00:21:04
◼
►
And for me that kind of this that kind of process of just keeping this running log seems
00:21:10
◼
►
to work well.
00:21:11
◼
►
It I like doing it that way rather than kind of having like sitting down and just testing
00:21:17
◼
►
for hours never really works for me.
00:21:19
◼
►
I find that I get kind of bored or I start doing the same patterns over and over again
00:21:24
◼
►
and so it's better I think for me I try and do it spread it out throughout the day and
00:21:28
◼
►
obviously if it's an application that you would use normally and naturally throughout
00:21:32
◼
►
the day then this makes it easier.
00:21:35
◼
►
But otherwise you just kind of have to remind yourself to do it and this is also a little
00:21:41
◼
►
bit tricky if you have an app that requires a certain amount of time for testing.
00:21:46
◼
►
So like for example I have a sleep tracker like in order for it to really like I cannot
00:21:51
◼
►
compressed beyond just running it on multiple devices the testing of that because it takes
00:21:57
◼
►
about eight hours to have eight hours of data collection and like I have simulated stuff
00:22:02
◼
►
in the app for you know when I'm doing like debug testing that I can kind of like create
00:22:06
◼
►
a night's data and run my algorithms on it.
00:22:10
◼
►
But for actual testing there's nothing you can really do other than wearing multiple
00:22:13
◼
►
watches which is what I do like when I'm doing extensive sleep testing like I'll have all
00:22:18
◼
►
my watches on my wrist during the night and I'm running it three or four times.
00:22:22
◼
►
But you know sort of in the same way I imagine for you with overcast like there's a certain
00:22:26
◼
►
amount if you just have to play lots of audio because if you don't you're never going to
00:22:30
◼
►
find all the weird audio issues or glitches or problems.
00:22:33
◼
►
There's nothing that you can really do to speed that up or compress that testing.
00:22:39
◼
►
Yeah basically I mean I like I just find ways to shove podcasts into more of my life so
00:22:46
◼
►
like you know like I've reached and also to test out things like syncing you know like
00:22:50
◼
►
my syncing engine took forever to really figure out I mean it took me almost a year to really
00:22:54
◼
►
nail syncing and one of the ways I started finally getting better at that was I started
00:22:59
◼
►
needing it more because I started using an iPad to play podcasts in the kitchen and I
00:23:04
◼
►
use my iPhone to play them everywhere else.
00:23:06
◼
►
So I will I was very frequently switching between multiple devices that would use the
00:23:10
◼
►
sync backend and just by doing that myself by needing that myself for a few months I
00:23:16
◼
►
really fixed a ton of sync problems and got it to a pretty good place now.
00:23:20
◼
►
And so yeah I mean it really is just a matter of just you know again it's like if you
00:23:24
◼
►
only make apps that you will use a lot this becomes a lot easier.
00:23:28
◼
►
If you don't make apps that use a lot this you know you're gonna have to find other
00:23:33
◼
►
ways to do this effectively but it really works out great for me and I will again once
00:23:38
◼
►
again go in briefly about beta testing also and we pretty sure we did a whole episode
00:23:43
◼
►
on it if not we will and we should but I think we already did.
00:23:47
◼
►
We have so many episodes now I've kind of lost track but I do rely a lot on beta testers
00:23:53
◼
►
to really do like a final check on the app.
00:23:57
◼
►
They aren't my first testers.
00:23:59
◼
►
I am my first tester and I will usually use changes to the app for weeks or even months
00:24:06
◼
►
before I give it to anybody else to test just because I really want to make sure that I
00:24:09
◼
►
know I have a feel for it, make sure it's good, make sure it's the right choice and
00:24:12
◼
►
then I will finally share with testers like right before the public gets it.
00:24:16
◼
►
But basically beta testers can help a little bit not a ton but a little bit.
00:24:22
◼
►
I will also add that the what you mentioned earlier about your workflow about taking taking
00:24:28
◼
►
screenshots and then adding arrows and then adding them to OmniFocus and things like that.
00:24:33
◼
►
If you want to go off the deep end there are lots of crazy tools and frameworks that automate
00:24:37
◼
►
that process to varying degrees and have all sorts of cool integration.
00:24:40
◼
►
So there are some like I made bug shot kit which is basically shoved bug shot into an
00:24:45
◼
►
app as you could do things like it will detect screenshots being taken and then automatically
00:24:51
◼
►
bring up a modal controller with the screenshot saying here mark this up and send it in to
00:24:56
◼
►
report a bug.
00:24:57
◼
►
You can give that to all your testers and you can use that yourself during your test
00:25:00
◼
►
process and so you can basically have like easy or automated bug reporting right embedded
00:25:05
◼
►
into your app.
00:25:06
◼
►
And there's again there's lots of ways to do this.
00:25:08
◼
►
I'm not going to recommend any particular one because I haven't really looked but
00:25:10
◼
►
there are many of these that exist now and you can you can really go off the deep end
00:25:15
◼
►
with like the levels of integrations.
00:25:17
◼
►
You can have it like post to slack and then make an entry in your bug tracker.
00:25:21
◼
►
All sorts of crazy things you can do now if you really want to get into all that stuff.
00:25:25
◼
►
So the options are pretty good for now for like tools and frameworks and things like
00:25:30
◼
►
that to automate like the tedious stuff.
00:25:33
◼
►
But for the real just using the app really just again just using it a lot yourself is
00:25:39
◼
►
a very very effective way to do it.
00:25:41
◼
►
Obviously as I know automated things exist sorry Casey again but just using your app
00:25:45
◼
►
yourself will show you so much and will tell you so much and will show you things like
00:25:49
◼
►
you know this this workflow is kind of clunky or this animation doesn't really look right
00:25:54
◼
►
or this I really need this text to be lighter when it's really sunny outside or something
00:25:58
◼
►
like that like there's a large category of either problems or or good and bad design
00:26:04
◼
►
decisions for which just using it yourself is really the only way to know and really
00:26:09
◼
►
the only way to test and that isn't true for everything obviously sorry again Casey but
00:26:13
◼
►
that's true for quite a lot that we do in our apps.
00:26:15
◼
►
Yeah and I think with other building I would say with beta testing that's an important
00:26:19
◼
►
thing to it or another use for beta testing is the importance of not overly preparing
00:26:26
◼
►
your beta testers for what you know for the app so in terms of trying to give in giving
00:26:32
◼
►
them an experience that is similar to the experience that your app store users are going
00:26:36
◼
►
to have and so when they download the app and they're like what do I do or this is really
00:26:42
◼
►
confusing like some of the best feedback I get from beta testers is sort of accidental
00:26:47
◼
►
where they're not actually they don't know it but they're telling me something is fundamentally
00:26:53
◼
►
flawed or is really confusing where they're asking a question that like implies a fundamental
00:26:59
◼
►
misunderstanding about something and that kind of feedback is always really good and
00:27:03
◼
►
so I try whenever I'm beta testing to just almost like blindly send out the betas to
00:27:08
◼
►
people where I'm not giving them too much preparation or like giving them step by step
00:27:12
◼
►
walkthroughs or details it's very much like as though you just downloaded this from the
00:27:17
◼
►
app store you know maybe you've seen a screenshot or two you've had a high level discussion
00:27:21
◼
►
but you're working it out just as you would as a regular user and that's a great test
00:27:26
◼
►
because as the developer of the app obviously I know how everything works everything is
00:27:32
◼
►
intuitive to me because that's the way I made it but it is often not the case for your users
00:27:37
◼
►
and so one of the best uses for beta testing isn't you know it's great when they find you
00:27:42
◼
►
know actual bugs and issues and problems but it's also it's really useful for capturing
00:27:47
◼
►
those kind of like it's like if you're asking this question you clearly I clearly have not
00:27:51
◼
►
communicated the purpose of this feature correctly or I've hidden it in a weird way or you think
00:27:56
◼
►
it should do something and it does the opposite and those are the only thing that is almost
00:28:01
◼
►
exclusively possible to find in beta testing because no matter how hard you try to kind
00:28:08
◼
►
of like reset your brain to what if I was a fresh user opening the application it's
00:28:13
◼
►
kind of impossible where you know like the current app I'm working on I have been working
00:28:16
◼
►
on it essentially every day for months at this point so I'm so in it that I just don't
00:28:24
◼
►
see it's like I'm so far into the woods that I have no idea that there is a forest
00:28:27
◼
►
like all I have is trees like it is deep down into it I mean and maybe when I come back
00:28:34
◼
►
to it in a couple of you know like I'll put the app to the side for a little bit and
00:28:38
◼
►
I'll come back to it in maybe a month maybe I will have that kind of fresh experience
00:28:43
◼
►
but until that time the only way I can get that is with beta testing and so it is definitely
00:28:47
◼
►
a valuable and important aspect of that.
00:28:49
◼
►
All right and I guess we're going to see your new app pretty soon right?
00:28:53
◼
►
Yeah it should be hopefully launching next week so it should be kind of exciting and
00:29:00
◼
►
I expect to I'm sure we'll talk about it on the show how the launch went and kind of
00:29:03
◼
►
some of the details of it as the app goes but it is very exciting to get to this point
00:29:08
◼
►
and I will say as you know it's like starting all the other independent developers out there
00:29:12
◼
►
it's like I just put in the work because there's definitely been times in this project
00:29:17
◼
►
where I was like oh gosh what have I gotten myself into maybe I should just like can this
00:29:21
◼
►
project and move on but there is hope eventually we'll get there eventually we'll ship and
00:29:25
◼
►
it is delightful when you get there.
00:29:27
◼
►
Can't tell you a secret to the audience?
00:29:30
◼
►
I know what this is and I've used it and it's really good so look forward to it anyway
00:29:34
◼
►
thanks a lot for listening everybody we're out of time this week and we'll talk to
00:29:37
◼
►
you next week.