112: Ideal vs. Pragmatic
00:00:00
◼
►
Welcome to Under the Radar, a show about independent iOS app development.
00:00:03
◼
►
I'm Mark Orment.
00:00:04
◼
►
And I'm David Smith.
00:00:05
◼
►
Under the Radar is never longer than 30 minutes, so let's get started.
00:00:09
◼
►
This week we wanted to talk about when you're forced to do something for outside reasons
00:00:16
◼
►
or market demand or pragmatic reasons that maybe you wish you wouldn't have to do or
00:00:23
◼
►
you think you shouldn't have to do, but you're forced to do it anyway.
00:00:28
◼
►
And this was brought about, the reason I wanted to talk about this initially is that I had
00:00:33
◼
►
a number of things happen recently with Overcast that were of this nature and where I had to
00:00:38
◼
►
basically adopt to the way people actually are or the way things actually work rather
00:00:44
◼
►
than hold onto my long-standing belief that it should be a different way or my wish that
00:00:51
◼
►
it was a different way because it would save me some work, which is often the case.
00:00:58
◼
►
The big one that kind of drove all this was Overcast recently had to implement feed redirect
00:01:05
◼
►
support or permanent feed change support.
00:01:08
◼
►
And I talked about this a little bit on ATP last week, so I'm not going to go too far
00:01:12
◼
►
into it, but the basic idea here is that podcast feeds change over time.
00:01:21
◼
►
Podcasters often will move the feed to a different service or to a different host or to a different
00:01:27
◼
►
ad tracking platform or whatever else.
00:01:30
◼
►
They just move the feed and they don't, if you move a website, you usually set up a redirect
00:01:35
◼
►
from the old URLs to the new URLs and you usually leave that redirect there forever.
00:01:40
◼
►
And even if you give it the permanent redirect code of 301 rather than the temporary one
00:01:45
◼
►
of 302, you still leave up the redirect forever because you know that there's old links that
00:01:51
◼
►
point to it, there's old search engine ranking that might point to it, and so you know that
00:01:56
◼
►
it's a good practice on the web to basically make redirects work forever.
00:02:01
◼
►
So if you're writing something that keeps URLs in a database and crawls them every so
00:02:06
◼
►
often, on the web you can pretty much never process permanent redirect codes and be fine
00:02:14
◼
►
because almost all the redirects will keep working indefinitely into the future.
00:02:20
◼
►
And when I made Overcast, this is how I set up the database.
00:02:23
◼
►
And this was, you know, I made the database in 2013.
00:02:26
◼
►
That's when I made the schema, I started crawling and started building up my database of feeds
00:02:30
◼
►
and the app launched about a year later.
00:02:33
◼
►
And so for me, at the time, I made this decision of how to structure my feeds table and my
00:02:39
◼
►
feed items table and how users interact with feeds and feed items.
00:02:45
◼
►
I designed this whole schema back then and there was this one big flaw in the schema,
00:02:49
◼
►
which was that feeds couldn't have their URLs changed.
00:02:54
◼
►
If a feed was created with a new URL, it would be a whole new feed entry and then there was
00:03:00
◼
►
no mechanism in place to do things like, you know, if a new URL is created and then one
00:03:06
◼
►
of the old ones redirects to that one, can I merge those entries?
00:03:10
◼
►
Do I copy the old one to the new one?
00:03:13
◼
►
Does the old one have some kind of alias list that also points to the new one?
00:03:18
◼
►
I had no setup like that in place.
00:03:20
◼
►
The entire app and everything and the entire database were built up without that consideration
00:03:25
◼
►
or without that possibility.
00:03:28
◼
►
And this started to become a problem pretty soon after launch that a couple of web, a
00:03:33
◼
►
couple of podcasts started switching their feed URLs and for a while I thought, well,
00:03:38
◼
►
you know, people will just subscribe to the new one.
00:03:40
◼
►
The old one will still work forever because it will redirect and it will be fine.
00:03:43
◼
►
There were a number of problems with the setup in reality.
00:03:46
◼
►
One of the big ones is that when I would do things like count subscribers to do things
00:03:51
◼
►
like search ranking, the two different entries would count as independent feeds.
00:03:56
◼
►
I would have to do search filtering, so things like, you know, I filter, I don't show search
00:04:00
◼
►
results that don't have an iTunes ID associated.
00:04:03
◼
►
So when the iTunes ID would move, it would basically lose the search ranking and the
00:04:08
◼
►
entire history of the previous one, et cetera.
00:04:10
◼
►
So there were all sorts of problems with that.
00:04:12
◼
►
But it wasn't a big problem because not a lot of podcasts were moving their feeds in
00:04:19
◼
►
And then over the following years, a lot of podcasts switched from HTTP to HTTPS and because
00:04:28
◼
►
as a lot of websites did, as we all kind of learned that that would, we probably should
00:04:31
◼
►
be doing that.
00:04:33
◼
►
That created a whole lot of duplicate entries.
00:04:37
◼
►
Only during that time, certain podcast networks shut down or moved or changed their names,
00:04:43
◼
►
changed their domains, and that caused even more extra or moved entries.
00:04:49
◼
►
And then the big one was over the last year or two, a lot of major podcast publishers
00:04:56
◼
►
have switched to dynamic ad insertion platforms, which host the feed at their own URL for who
00:05:01
◼
►
knows what reason.
00:05:03
◼
►
And so tons of major podcasts are now switching their feeds.
00:05:08
◼
►
If they didn't already do it with HTTPS like two years ago, in the last year, they're now
00:05:12
◼
►
doing it with ad tracking platforms.
00:05:15
◼
►
So I've had a, it basically went from a small problem in year one to a medium-sized problem
00:05:22
◼
►
in year two and a really big problem in year three.
00:05:25
◼
►
And so I finally had to implement feed redirects.
00:05:29
◼
►
And I had, you know, all this time I've been thinking like, I can just follow redirects.
00:05:33
◼
►
I can, like, they'll be fine.
00:05:34
◼
►
They'll be, they'll be good forever.
00:05:37
◼
►
And the reality was just different.
00:05:38
◼
►
The reality was podcasters don't do things the way web web developers do things.
00:05:42
◼
►
They break redirects.
00:05:44
◼
►
Like they consider iTunes the authoritative source.
00:05:47
◼
►
And if iTunes moves the feed for them, which they can do through multiple different means,
00:05:52
◼
►
either a 301 or putting a special XML tag in the feed that was returned with a 200 response
00:05:58
◼
►
or going through like an iTunes customer support representative running the podcast directory
00:06:03
◼
►
and like just like emailing somebody and saying, sorry, we messed up our feed.
00:06:06
◼
►
Can you fix it?
00:06:08
◼
►
And none of those things would carry over to Overcast.
00:06:11
◼
►
So the reality of the market was very, very different from how I wanted it to be.
00:06:14
◼
►
I really badly wanted it to work in this one way that websites always were, which is like,
00:06:19
◼
►
you know, set up a redirect and keep it up forever.
00:06:22
◼
►
But the reality was very different.
00:06:23
◼
►
The reality was podcasters would move their feeds frequently.
00:06:26
◼
►
They would break the old URLs and redirects most of the time.
00:06:30
◼
►
And even if they didn't, I still had this problem of like split feeds in the database.
00:06:35
◼
►
So that's no good.
00:06:37
◼
►
So finally over the last couple of weeks, I finally wrote redirect support.
00:06:40
◼
►
And because of my database schema being so unfriendly to it, it's a huge hack and it's
00:06:46
◼
►
not perfect.
00:06:47
◼
►
The way I do it is I basically like copy your old entries onto the new feed and delete your
00:06:54
◼
►
And that's not great.
00:06:55
◼
►
Like there's a whole lot of downsides to that.
00:06:57
◼
►
One of the big ones is that the app seeing a new feed item ID for all these episodes,
00:07:03
◼
►
the app redownloads them all.
00:07:05
◼
►
That's no good either.
00:07:07
◼
►
So like it maintains your history, but you have to download, like all of a sudden your
00:07:11
◼
►
phone is downloading like all your saved episodes of a certain show that you hadn't listened
00:07:15
◼
►
And that's kind of weird to happen unexpectedly.
00:07:18
◼
►
So that's not a great solution.
00:07:21
◼
►
But I had to be pragmatic.
00:07:23
◼
►
I had to address this problem somehow because the market was very different than what I
00:07:30
◼
►
wanted it to be.
00:07:33
◼
►
And I think I also hear in what you're saying, it's this thing that is so happens to me.
00:07:38
◼
►
This happened to me so many times is it's as much as we would like.
00:07:42
◼
►
I think we would like to be able to predict the future.
00:07:46
◼
►
And when we're initially building our applications and we're designing them and we're structuring
00:07:50
◼
►
them, we would make choices that our future selves will thank us for.
00:07:58
◼
►
And conceptually, that sounds great.
00:08:00
◼
►
That, okay, yeah, of course, I should structure these things with maximum flexibility and
00:08:07
◼
►
I should be able to, ahead of time, I should consider all these possibilities.
00:08:12
◼
►
The reality is I've done this enough to know that you can't.
00:08:17
◼
►
That is impossible.
00:08:20
◼
►
Things will change and break and become problems in ways that you could never have predicted.
00:08:26
◼
►
Maybe you get slightly better at this.
00:08:28
◼
►
Maybe you don't make the same mistake twice.
00:08:31
◼
►
Those types of things may be improving.
00:08:33
◼
►
But I feel like the situation you find yourself in is something that is universal to app development
00:08:38
◼
►
or just software engineering in general.
00:08:41
◼
►
That you will at some point be forced to either, you either have to make an assumption or you
00:08:47
◼
►
have to explode complexity.
00:08:52
◼
►
One of those two things you have to do at hundreds of points in your application as
00:08:57
◼
►
you're developing it.
00:08:59
◼
►
You can make an assumption.
00:09:00
◼
►
You can say, in this case, you're going to assume that mapping a feed URL to the feed
00:09:06
◼
►
ID directly will be a stable link.
00:09:10
◼
►
Or you have to explode complexity.
00:09:12
◼
►
And you have to do some very complicated things to make that work flexibly.
00:09:19
◼
►
And in this case, you chose to start with the assumption.
00:09:23
◼
►
But that's the same thing that we do all the time.
00:09:26
◼
►
So much of software development is about getting good at making that decision between making
00:09:33
◼
►
an assumption and expanding complexity.
00:09:35
◼
►
And you can apply this to so many things.
00:09:37
◼
►
How complicated is your class hierarchy?
00:09:40
◼
►
What kind of choices do you do for data persistence?
00:09:45
◼
►
So many different things come down to this very simple situation.
00:09:48
◼
►
Are you going to make an assumption or are you going to explode complexity?
00:09:54
◼
►
And that means that you're stuck.
00:09:56
◼
►
Now you have to go and be forced to do this thing because your assumption proved to be
00:10:02
◼
►
wrong or proved to be wrong enough that continuing with that assumption becomes problematic.
00:10:10
◼
►
And it's so much more complex too because like as I've mentioned this on Twitter here
00:10:13
◼
►
and there the last couple of weeks, I've gotten a number of people who are suggesting like
00:10:17
◼
►
oh why didn't you just do this with your database or why didn't you just make this migration
00:10:21
◼
►
And a lot of them, a lot of the ideas or suggestions don't really work or aren't as simple as they
00:10:26
◼
►
sound because Overcast is not just a local app with a local database.
00:10:31
◼
►
It is a service with a server backend that syncs between multiple copies of your app on
00:10:35
◼
►
multiple devices.
00:10:36
◼
►
So any kind of like one way migration either can't happen or is way more complex than you
00:10:42
◼
►
would think.
00:10:43
◼
►
And also because it is a server backed app, the complexity of the schema has a direct
00:10:49
◼
►
relationship to how scalable it is and how efficient it is and how expensive it is to
00:10:57
◼
►
If every look up to the feeds table has to then make another look up to some kind of
00:11:02
◼
►
like URL aliases table or something like that to map the URL to the feeds I'm fetching,
00:11:08
◼
►
that's a pretty significant hit in performance.
00:11:11
◼
►
That's much more complexity and that could be way more load on the database servers and
00:11:15
◼
►
that could really make scaling more difficult.
00:11:17
◼
►
And if you have many of those things in your schema for flexibility and everything, then
00:11:24
◼
►
you start running into pretty big scaling problems and big challenges that either just
00:11:28
◼
►
make your life hard or make things cost way more to host and that could break your business
00:11:33
◼
►
So there's all sorts of other considerations here that make things like this sound like
00:11:38
◼
►
a hard problem.
00:11:39
◼
►
Anyway, so that was one problem of many that I've had and I kind of wanted to talk for
00:11:46
◼
►
the rest of the show about a few more of those.
00:11:48
◼
►
But before we do, speaking of scaling your web servers, we respond to this week by Linode.
00:11:55
◼
►
Linode has fast, powerful web hosting options that you can get set up in just seconds.
00:11:59
◼
►
It's very, very easy to understand all their tools and they can let you choose the resources
00:12:03
◼
►
and the next distribution that you want, giving you the power and flexibility that you need.
00:12:06
◼
►
All this starts at just $5 a month.
00:12:08
◼
►
That gets you a server with one gig of RAM in the Linode cloud.
00:12:12
◼
►
Linode servers are very, very fast.
00:12:14
◼
►
They have industry leading performance.
00:12:16
◼
►
They have native SSD storage on all their servers.
00:12:19
◼
►
They have a lot of
00:12:56
◼
►
web hosting options.