Developing Perspective

#140: Managing Beta Testing.


00:00:00   Hello and welcome to Developing Perspective,

00:00:02   Developing Perspective is a podcast discussing

00:00:04   News of Note and iOS development, Apple and the like.

00:00:07   I'm your host, David Smith.

00:00:08   I'm an independent iOS and Mac developer

00:00:10   based in Herndon, Virginia.

00:00:11   This is show number 140 and today is Friday, August 30th.

00:00:15   Developing Perspective is never longer than 15 minutes,

00:00:17   so let's get started.

00:00:19   All right, I'm gonna have a couple of quick topics

00:00:21   to talk through today.

00:00:22   The first is I'm gonna announce it here

00:00:24   probably for the first time.

00:00:25   I'm gonna be doing a, I say, short run

00:00:28   of developing perspective t-shirts.

00:00:31   T-shirts are always kind of one of these funny things.

00:00:33   I think it's kind of like everybody does them,

00:00:35   and it becomes a bit of a cliche.

00:00:36   But I kind of wanted to talk into why I'm doing it

00:00:38   and kind of what it's like.

00:00:39   And just sort of if that's the kind of thing

00:00:42   that you'd be interested in doing, by all means.

00:00:45   So first off, the t-shirt itself,

00:00:47   I'm doing it through Teespring because that's

00:00:49   ridiculously easy, and easy is good.

00:00:51   And so they're already just a company

00:00:53   that's kind of one of these things.

00:00:54   You go and buy it, and if you reach a certain threshold,

00:00:57   And it's sort of crowdfunded in that way, though I think I'll be setting my minimum

00:01:02   to the smallest number that's possible, because I just want to make sure that they exist far

00:01:06   more than anything else.

00:01:09   And the reason I'm doing it, which I thought was more of an interesting topic, is I'm a

00:01:13   big fan of these geeky t-shirts.

00:01:15   I have probably more of them than I should.

00:01:19   My closet is sort of half nice clothes and half geeky t-shirts.

00:01:23   And sometimes I think the geeky districts are gradually expanding and overwhelming.

00:01:29   The reason I have them, the reason I like them, is-- it's something that I kind of had

00:01:33   to step back and think about, is how the nature of being independent-- and I've talked about

00:01:37   this before-- is it's sort of intrinsically a little bit lonely in that I don't work with

00:01:43   people on a day-to-day basis, which is something I miss from kind of a more nine-to-five job.

00:01:47   I'm pretty much just on my own for a lot of the time and a lot of the work I do.

00:01:51   And so it's kind of difficult to sometimes have a sense of belonging, to have a sense

00:01:55   of camaraderie or a team or whatever you want to call it.

00:01:59   And there's things you can do around that.

00:02:00   But one of the things I kind of like is when I wear, wearing a t shirt, to me anyway, can

00:02:06   kind of feel like I'm part of something bigger, that it draws a sense of connection.

00:02:10   And you know, when I'm out, it's kind of like, I'm in on an in joke when I'm wearing like,

00:02:16   when I'm, you know, today I'm wearing my hypercritical t shirt, for example.

00:02:19   Like when I'm wearing that, like, most people have no idea what that is.

00:02:22   But if someone does, then I know they're my kind of people.

00:02:25   And even if they don't, like, it's kind of cool that I have, I sort of, and I know the

00:02:29   secret handshake, I know what's going on there.

00:02:31   And that's why I kind of I like that sense of that sense of community that kind of having

00:02:35   a, I guess you could say, uniform for encourages and kind of allows.

00:02:39   And so that's kind of a, it's kind of a fun thing.

00:02:42   And so that's kind of I was like, well, you know, I have all these other ones for all

00:02:46   the various things that I like, the things that I enjoy, different products I've been

00:02:50   associated with, etc. And so I thought, "Well, why don't I make one for developing perspective?"

00:02:54   It's a show that's been around for over two years now. This is episode 140, I think. It's

00:03:02   been around enough that I think it's probably worth the kind of thing to have a shirt for,

00:03:06   have something that I can wear, and that I can, you know, it's just sort of something

00:03:10   that I own, the community that I'm, I guess, directly part of, if not at the center of.

00:03:15   The design was actually done by friend of mine, Jory Raphael, who you may know from

00:03:18   things like symbolicons.

00:03:20   He is just a graphic designer, sort of vector artwork genius.

00:03:25   And when I decided on kind of generally what I wanted to do, I knew he would be the right

00:03:28   person to sort of take that and turn it into an actual design.

00:03:32   And so the design is pretty straightforward.

00:03:34   It's an underscore followed by a coffee mug.

00:03:37   It's actually my favorite coffee mug, which is, I'll have a picture of it in the show

00:03:40   notes.

00:03:41   But this particular coffee mug I got when I was visiting the Cupertino Apple

00:03:45   Campus a couple years ago, and it's a perfect shape

00:03:48   for drinking warm beverages.

00:03:50   If you're not into coffee, imagine it as a teacup, hot chocolate,

00:03:53   whatever floats your boat.

00:03:55   But that's kind of the concept.

00:03:56   And it was just something that I thought was a little bit geeky,

00:03:58   a little bit understated, and a little bit silly,

00:04:01   which is kind of one of the things that I enjoy in a t-shirt,

00:04:05   and something that I think is kind of fun.

00:04:07   The underscore at the beginning is, of course,

00:04:08   a reference to the fact that I now have a nickname, which

00:04:10   is kind of a funny thing.

00:04:11   Because for my entire life, I don't

00:04:13   think I've ever had a nickname.

00:04:14   The closest thing was being called Dave rather than David,

00:04:17   which I don't mind, which I guess is only vaguely

00:04:19   a nickname.

00:04:20   But for better or worse, I've ended up

00:04:21   being known as underscore in this community.

00:04:23   I think that sort of definitely peaked when I was recently--

00:04:26   I think it was Dr. Drang in one of an article

00:04:29   he referenced to me.

00:04:30   He referred to me, and then he abbreviated my name

00:04:33   as UDS, underscore David Smith.

00:04:35   So I have new initials, too.

00:04:36   So anyway, so that's kind of the concept.

00:04:39   It's fairly straightforward, fairly simple.

00:04:41   It's just a basic design.

00:04:42   There'll be a link in the show notes,

00:04:44   or you can go to teespring.com/underscore

00:04:46   to look at it and to buy it.

00:04:49   The price is $1,459, which is kind of a play on words

00:04:52   on the length of a-- my target length for a Developing

00:04:55   Perspective episode, which at that point,

00:04:58   it's really not for the money.

00:05:00   I'm not doing this to make a lot of money.

00:05:01   It's the kind of thing that's basically-- it's

00:05:03   a little bit above cost.

00:05:04   Essentially, my goal is to break even on the design

00:05:06   and the actual shipping of it.

00:05:08   My goal is to get more people to have it.

00:05:13   And so that's kind of it.

00:05:16   So anyway, I'll have a link in the show notes if that's something that you've been interested in, or if that's something that you want.

00:05:17   They should ship in time, I think for most of the fall conferences.

00:05:21   It should probably be arriving somewhere in mid-September when the campaign finishes in about, I think it finishes next week.

00:05:24   It'll just do a short one week kind of campaign for it.

00:05:30   So anyway, if that's your thing, great.

00:05:34   If not, I'm going to get into the rest of the topic for the show.

00:05:35   And that topic today is going to be, I'm going to talk about beta testing.

00:05:37   which is I always try-- this is a really tricky time for me

00:05:39   in terms of the time of year for coming up with new topics,

00:05:42   coming up with things to say, because a lot of what

00:05:44   I'm working on I can't talk about.

00:05:46   A lot of the new exciting things that are coming out

00:05:48   are either under NDA or unannounced.

00:05:50   And so we're in kind of this middle ground.

00:05:51   And so it's always kind of tricky.

00:05:52   But I think whenever that happens, the best thing to do

00:05:55   is just to fall back on what it is that I'm actually

00:05:56   doing it right now.

00:05:57   And right now, that's beta testing.

00:05:59   Right now, that is working on the final phases

00:06:02   of getting my apps all queued up to go out to Apple.

00:06:06   And so I'm going to talk a little bit about beta testing

00:06:08   generally, and then a few things that I've learned,

00:06:10   or situations, or struggles, or problems that you run into,

00:06:12   and what I do with that.

00:06:14   So first, I'm going to talk about what beta testing is

00:06:17   in iOS and how that works.

00:06:19   So generally, beta testing is the process

00:06:21   of taking your app right before it's ready for launch

00:06:25   and seeding it out to a small group of people

00:06:27   to get feedback on to identify bugs, problems, issues,

00:06:30   whatever it may be, and to do it in a way that

00:06:34   is constructive, hopefully.

00:06:35   Because ultimately, the goal, at least in my mind, for beta

00:06:37   testing, is to improve the product every time

00:06:41   you get around out.

00:06:43   There's some amount of it that is perhaps marketing oriented,

00:06:46   that it's certainly some amount of beta testing's value

00:06:49   is that it puts it in the hands of people who will hopefully

00:06:52   talk about it.

00:06:52   Mostly what you're really trying to do

00:06:54   is try to collect feedback from a small group

00:06:56   of trusted, interesting people who can help craft and sort

00:07:02   of direct the app in that final stage.

00:07:06   So typically, you've got the idea, the concept,

00:07:10   and then usually in sort of like 99% of the implementation

00:07:12   built. And now you're going to show it to some people

00:07:14   and get their feedback on it.

00:07:17   And that's always a dangerous thing.

00:07:19   And I'll say that, and mostly just in that,

00:07:22   it's always tricky when you open up an app to somebody else

00:07:25   and you're getting their feedback on it.

00:07:27   It can change your own view on that app.

00:07:30   It can change your direction for that app, for better or worse.

00:07:36   And so I think something I always want to emphasize

00:07:39   is it's important to be thoughtful about who

00:07:41   you show it to.

00:07:42   And this is something that I personally try--

00:07:45   my typical opinion on this is I show it

00:07:48   to a relatively small number of people.

00:07:51   And they're people whose opinions I trust,

00:07:54   who I really want to hear what they have to say,

00:08:00   and whose opinions, I think, are things that I tend to agree with,

00:08:03   are things that I would respect and respond well to.

00:08:07   I'm not the kind of person who tends to do these kind of betas

00:08:10   where you're trying to open it up to like 50 or 100 people,

00:08:13   many of whom you don't actually know.

00:08:15   They're just general customers.

00:08:17   And sometimes that can be useful if you're

00:08:19   solving a specific bug that customers have

00:08:22   and you want to have them reproduce it.

00:08:23   That makes sense to me.

00:08:24   But for a lot of times, what I find that's difficult

00:08:26   is the more and more beta testers you have,

00:08:28   the more time and energy it takes to actually manage

00:08:30   that process.

00:08:32   And so I'm not the kind of person who tends to just like,

00:08:34   every now and then I'll get requests on Twitter,

00:08:36   things like that.

00:08:36   People are like, hey, do you have any space in your beta

00:08:38   group to beta test something?

00:08:40   And I've almost done everything I've ever responded to someone

00:08:44   like that and said, yeah, that's great.

00:08:45   Let's put it in.

00:08:46   Because what I want to do is I want to--

00:08:49   in that critical phase of right before an episode is ready

00:08:52   to launch, what I want to do is to narrow in rather than

00:08:55   broaden out, necessarily.

00:08:56   And this is just the way that I work.

00:08:57   Some people do it differently, so the more the merrier.

00:09:00   Kind of open betas and things where you can get

00:09:02   lots of feedback.

00:09:03   What I'm trying to do is I'm trying to narrow down

00:09:06   my feedback and to be able to manage and really invest

00:09:08   and to have a lot of deep discussion back and forth,

00:09:11   which I just can't do with 50 people.

00:09:13   You can kind of collect crash reports

00:09:14   and those kinds of things, maybe things

00:09:15   that can be aggregated.

00:09:17   But for me, what I want to do is I want to have

00:09:19   a couple of conversations with people who I know

00:09:22   will have interesting feedback and can really help me

00:09:24   improve my product.

00:09:26   So what I do is you collect a group of people,

00:09:28   and you decide who it's going to be, and you send it out.

00:09:31   In iOS, the way you actually manage that is you

00:09:33   have to do what's called an ad hoc build, which

00:09:35   is if you've never done it, you just

00:09:37   go into the provisioning profile area in the developer portal.

00:09:42   And you set up-- it's a different signing certificate

00:09:44   and key.

00:09:45   It's similar to what you do when you're doing an app store build,

00:09:48   but you have to specify that the actual device

00:09:50   IDs for every device that you want that to run in--

00:09:53   and this gets into the whole kind of the way

00:09:55   that an iOS device is very locked down.

00:09:57   And so the only way to run an app is if it's signed.

00:09:59   And the only way to get it signed

00:10:00   is to have the-- you have to create an ad hoc

00:10:03   profile for the specific device that you want it to run on.

00:10:06   So you have to go and collect those IDs from everybody.

00:10:08   And most people in this community,

00:10:10   whether in the press or the developers,

00:10:11   should have no problems collecting that device ID.

00:10:14   If you need to do it from someone else who's

00:10:15   not quite as technical, there's all kinds of different tools

00:10:18   and things you can do to collect that.

00:10:20   Often the best is to use one of the tools, programs,

00:10:24   or services like TestFlight or Hockey,

00:10:28   which are two different services that

00:10:30   do very similar stuff around this kind of beta testing

00:10:32   and sending out builds.

00:10:34   But one of the things they often can do for you is they can collect

00:10:37   devices directly.

00:10:38   So you'll invite someone with an email.

00:10:39   They'll hit a button on the device,

00:10:41   and it'll pull down all the relevant information

00:10:44   that you're going to need.

00:10:45   So you prepare the build.

00:10:47   I definitely recommend that you send the builds

00:10:48   to people using one of those two services, TestFlight or Hockey.

00:10:51   I've used them both.

00:10:52   They're both pretty good.

00:10:53   You kind of go back and forth on which one's better.

00:10:55   I don't really know.

00:10:56   I think hockey is more of the kind of-- it's

00:10:59   a service you pay for.

00:11:00   It's a bit more robust in that way.

00:11:03   TestFlight is kind of more kind of-- it's a free service.

00:11:07   You can do lots of things with it.

00:11:08   So it's kind of sometimes nice for experimentation

00:11:11   or the first time you adopt this or if it's just

00:11:13   something you're doing on a hobby, maybe.

00:11:14   But you get what you pay for in a lot of ways.

00:11:17   And so it's just kind of a decision you have to make.

00:11:20   But whichever service you're going to use,

00:11:21   You put that-- you make your build, do it in Xcode,

00:11:25   upload it up to one of these services,

00:11:26   put in some useful distribution notes,

00:11:28   and then you can just send people links.

00:11:30   They take that link, and they download it onto their device

00:11:32   and they can get you feedback.

00:11:34   The actual way in which you collect feedback is-- a lot

00:11:37   of people do it very differently.

00:11:38   I've been recently on some great betas.

00:11:40   People have glassboard rooms on it

00:11:43   where you are collecting feedback there.

00:11:45   You have a discussion forum for beta testers.

00:11:47   You can have people email you.

00:11:49   They can IM you.

00:11:50   they can do kind of whatever makes sense for you.

00:11:53   And it's kind of hard to be prescriptive about that

00:11:55   because I think some people just react differently

00:11:58   to their feedback.

00:12:00   For me, I just want to, whatever is most convenient

00:12:02   for the person giving the feedback,

00:12:03   a lot of times it'll take emails,

00:12:05   it'll be taking items from people,

00:12:07   high messages, whatever it is.

00:12:08   And my goal is to try and understand the problems

00:12:11   that they're having, the things that are confusing,

00:12:13   number one, the things that are crashing, number two,

00:12:17   and the things that they wish it did.

00:12:20   And I do it in that order because first, things

00:12:23   that are confusing.

00:12:25   No matter what, if it's confusing,

00:12:26   that's very, very problematic for users,

00:12:29   and especially for support.

00:12:30   Where often, the worst thing you can ever have on a support

00:12:33   thing is where someone asks for a feature to-- oh,

00:12:36   I love the app, but I wish it did x.

00:12:38   And it actually already does x, but that

00:12:40   isn't clear in the interface, or it isn't well communicated.

00:12:43   And so making sure that things are clear, very important.

00:12:46   Making sure things are stable and solid.

00:12:48   Definitely in the beta phase, I highly

00:12:50   recommend putting in some kind of crash reporting.

00:12:52   I think both TestFlightHockey does it.

00:12:54   You can use Crashlytics, whatever, PLCrashReporter.

00:12:56   Whatever makes sense for you can incorporate crash reporting.

00:13:01   But it's really important in this phase

00:13:03   to be collecting that kind of stuff.

00:13:04   Because you'll find all the little bugs, all the little edge

00:13:05   cases, all the little things that you didn't expect

00:13:08   would happen, hopefully happen in your beta phase

00:13:10   so it doesn't happen broadly in production.

00:13:13   And then the last thing, which is really interesting,

00:13:15   is when you put an app out into actual people's hand

00:13:18   and you get them actually using it,

00:13:20   you'll get feedback about the things

00:13:21   that they wish it did additionally.

00:13:23   And this is, I've found, especially in Pod Wrangler,

00:13:26   what I wish I'd been doing most heavy beta testing for recently.

00:13:29   You find there are things that you

00:13:31   can add to your application.

00:13:33   And the best are the ones that are

00:13:35   these tiny little small features that have big impacts, things

00:13:39   that you can relatively easily add on,

00:13:41   that you find that your app is doing 99% of what they want.

00:13:44   But they just wish to, oh, I wish I just

00:13:46   did this one extra thing.

00:13:48   And adding that will make the app so much more useful,

00:13:50   so much more functional for that person.

00:13:52   And any time you can find this, it's a huge win and a big

00:13:56   benefit to find during the beta phase, so that you can

00:13:59   add that in.

00:14:00   And when you release it to the community at large, you can

00:14:05   take advantage of that.

00:14:06   And you can have a nice big splashy launch that has a lot

00:14:10   of the features, a lot of the things that you could squeeze

00:14:12   into in that time.

00:14:15   Anyway, so that's sort of beta testing.

00:14:16   That's kind of how the process works, kind of what I do.

00:14:19   I think it's important.

00:14:20   I think you can overemphasize it.

00:14:22   I think it's just something that you

00:14:23   want to do to collect that feedback,

00:14:24   to understand how your app's good, how it's bad,

00:14:28   and what you can do to make it better.

00:14:31   Anyway, that's it for today's show.

00:14:32   As always, if you have questions, comments, concerns,

00:14:34   complaints, I'm on Twitter @_davidsmith,

00:14:37   david@developingperspective.com.

00:14:38   Otherwise, have a great weekend.

00:14:40   If you're in the US, have a great Labor Day.

00:14:42   Long weekend.

00:14:43   And otherwise, have a great week.

00:14:44   and I'll see you next week. Bye.

00:14:48   [ Silence ]

00:15:06   [ Silence ]

00:15:24   [ Silence ]