Under the Radar

00:00:00 ◼   ► Welcome to Under the Radar, a show about independent iOS app development.

00:00:04 ◼   ► I'm Marco Arment.

00:00:05 ◼   ► And I'm David Smith.

00:00:06 ◼   ► Under the Radar is never longer than 30 minutes, so let's get started.

00:00:10 ◼   ► So today we wanted to unpack the situation that occurred,

00:00:17 ◼   ► I guess it was the beginning of this week, the end of last week, something like that,

00:00:21 ◼   ► where Parse, a fairly widely used platform for app backends,

00:00:27 ◼   ► announce that they will be shutting down. And while the actual situation and the nuances

00:00:33 ◼   ► of that aren't particularly generally applicable or interesting, the actual sort of -- the

00:00:38 ◼   ► fundamentals of that, of having this big general purpose platform that was used by lots of

00:00:43 ◼   ► apps finally shutting -- or deciding it was going to shut down has a lot of knock-on effects

00:00:48 ◼   ► that are probably worth unpacking.

00:00:49 ◼   ► Before we dive into that, it probably makes sense to just sort of talk about what Parse

00:00:53 ◼   ► And so Parse was this platform that made it relatively easy to make a backend for your

00:00:59 ◼   ► application that would do object persistence, user management, things like that, like a

00:01:04 ◼   ► very basic high-level, like, non-specific to a particular industry backend sort of system

00:01:10 ◼   ► that a lot of apps could use.

00:01:11 ◼   ► And it had fairly attractive pricing, and including a free tier, which is always a bit

00:01:16 ◼   ► of a trap for these kinds of things, where you can say, "Oh, we can use it for free,

00:01:19 ◼   ► and if our app gets really successful, then we -- only then will we have to pay."

00:01:23 ◼   ► And eventually, a couple years ago, it was bought by Facebook, which made everyone who

00:01:26 ◼   ► was using it kind of happy because, oh, now it's not this sort of fly-by-night thing.

00:01:30 ◼   ► It's backed by, you know, this major corporation.

00:01:33 ◼   ► But now it's being shut down.

00:01:36 ◼   ► Facebook has decided that that's not something that they want to invest in and continue to

00:01:39 ◼   ► maintain.

00:01:40 ◼   ► And so a year from now, they are going to be turning it off.

00:01:43 ◼   ► They're doing it pretty well.

00:01:45 ◼   ► Like they're giving a year's notice and a bunch of migration tools.

00:01:48 ◼   ► But at the end of the day, this thing that I've heard varying reports, but at least,

00:01:55 ◼   ► it's fair to say quite a lot of apps make use of, at least in part, is just going to

00:01:59 ◼   ► be turned off.

00:02:00 ◼   ► And as a result, the apps that use it, if they haven't been updated or migrated, are

00:02:04 ◼   ► just going to stop working.

00:02:06 ◼   ► And that's kind of tricky.

00:02:07 ◼   ► >>

00:02:07 ◼   ► - Yeah, I think one of the weird things about this

00:02:09 ◼   ► is like, you know, it kind of ties back to App Economics,

00:02:11 ◼   ► where in order for these apps to continue working,

00:02:15 ◼   ► it has to be worth their developers' time,

00:02:18 ◼   ► and their developers have to have the budget

00:02:20 ◼   ► to now do a noticeable update.

00:02:23 ◼   ► And you know, they've made it relatively easy.

00:02:26 ◼   ► They've released big parts of their service

00:02:28 ◼   ► as open source that you could just install on any server.

00:02:31 ◼   ► Microsoft Azure has started trying to attract people

00:02:33 ◼   ► to migrate to them, and they're making it a little bit easier

00:02:36 ◼   ► So there are migration options here that aren't going to be incredibly work heavy, but it

00:02:41 ◼   ► is still work.

00:02:42 ◼   ► You have to still do an update.

00:02:43 ◼   ► There are going to be things you have to change and rewrite.

00:02:46 ◼   ► And so it has to be worth that happening by the apps developers.

00:02:51 ◼   ► So if you're relying on an app that uses this that hasn't been updated in a long time, that

00:02:55 ◼   ► might never be updated for this.

00:02:57 ◼   ► It might never be worth somebody's time to update it.

00:03:00 ◼   ► And that's unfortunate.

00:03:01 ◼   ► And that is going to cause a lot of problems in the app store as these apps just kind of

00:03:03 ◼   ► slowly collect one-star reviews and stop working and they just kind of live as zombies forever.

00:03:10 ◼   ► >> And so as developers, when I see something like this, like I said, the specifics of the

00:03:16 ◼   ► par situation are sort of vaguely or intellectually interesting but aren't actually practically

00:03:21 ◼   ► that interesting. But what it makes me think about is it makes me evaluate the dependencies

00:03:27 ◼   ► that I have in my own apps and as I build apps, and increasingly, fewer and fewer of

00:03:33 ◼   ► the things that I build have no web component whatsoever.

00:03:37 ◼   ► Because it seems like these days, you're going to need some kind of backend for your application.

00:03:44 ◼   ► Either something like parse or that's sort of like off the shelf or something you build

00:03:49 ◼   ► yourself.

00:03:50 ◼   ► And you're probably going to need these for at least one of three reasons.

00:03:53 ◼   ► You're going to need some kind of, your app's probably going to do something backup-related,

00:03:57 ◼   ► where people are, if people are putting any amount of data into your application, they're

00:04:02 ◼   ► probably going to want to be able to have it backed up.

00:04:05 ◼   ► And this is something that for a long time I used to say, "Oh, I rely on the iTunes

00:04:09 ◼   ► iCloud backup system."

00:04:11 ◼   ► But that is all kinds of problems and issues that you'll run into, where, like, I have

00:04:16 ◼   ► a recipe manager, and I ran into issues where their recipes were fully backed up in the

00:04:22 ◼   ► the latest backup that they did,

00:04:23 ◼   ► but they accidentally deleted the app.

00:04:25 ◼   ► And so now the only way they can get their recipes back

00:04:27 ◼   ► is to do a full restore of an old backup onto their device,

00:04:32 ◼   ► potentially destroying newer data

00:04:36 ◼   ► on other apps and things.

00:04:38 ◼   ► It's a mess.

00:04:39 ◼   ► So you wanna be able to backup your data,

00:04:40 ◼   ► or you wanna be able to sync your data

00:04:42 ◼   ► between different devices,

00:04:44 ◼   ► so you'll need some kind of backend to do that.

00:04:46 ◼   ► Or you just have an app that has a core service.

00:04:49 ◼   ► Obviously, I imagine in Overcast,

00:04:51 ◼   ► you need a backend, or a lot of what you do wouldn't work

00:04:56 ◼   ► if you didn't have some kind of backend to run for it.

00:04:59 ◼   ► - Oh, sure, I mean, there are podcast apps

00:05:02 ◼   ► that don't use server-side backends as intermediaries

00:05:05 ◼   ► and just crawl feeds directly and everything,

00:05:07 ◼   ► but that's not how I built mine,

00:05:09 ◼   ► and it affords me a bunch of advantages

00:05:11 ◼   ► to have done it the way I do it,

00:05:12 ◼   ► but now I have this big dependency on my service.

00:05:15 ◼   ► - Yeah, and I think ultimately, that's the right word.

00:05:18 ◼   ► At the end of it, building these backends

00:05:21 ◼   ► that are maybe, aren't always required,

00:05:23 ◼   ► but are going to be required in a lot of cases,

00:05:26 ◼   ► the biggest thing that I think this situation

00:05:29 ◼   ► is instructive for is making us aware

00:05:32 ◼   ► of the things that we're dependent on

00:05:34 ◼   ► and that our apps are dependent on,

00:05:36 ◼   ► because we're always gonna be dependent

00:05:38 ◼   ► on something, it seems.

00:05:39 ◼   ► There's no way to really say,

00:05:41 ◼   ► okay, I'm gonna be completely independent,

00:05:42 ◼   ► because ultimately, you're gonna be,

00:05:44 ◼   ► I'm very reliant on Apple, for example, and iOS.

00:05:49 ◼   ► If Apple announced tomorrow that, hey, we've decided this iOS thing isn't really working

00:05:55 ◼   ► out, we're just going to turn it off.

00:05:58 ◼   ► We're just going to stop making iPhones.

00:05:59 ◼   ► I'm not saying that's likely, but if they did, my apps would stop working.

00:06:03 ◼   ► In the same way that if Parse going away meant that apps that relied on it go away.

00:06:08 ◼   ► Or maybe a more practical example on the Apple side of things is if they decide, you know

00:06:13 ◼   ► what, CloudKit isn't working out, we're going to turn that off.

00:06:16 ◼   ► Or those types of things.

00:06:18 ◼   ► or I rely on my hosting provider.

00:06:20 ◼   ► I host all my Linux servers on Linode,

00:06:23 ◼   ► and if they decide they're gonna go out of business

00:06:28 ◼   ► or they decide they're not gonna do the kind of hosting

00:06:30 ◼   ► that I need anymore, suddenly I'm in a big bit of a bind.

00:06:34 ◼   ► And so there's no way to avoid being dependent.

00:06:37 ◼   ► You're always dependent on something,

00:06:38 ◼   ► but there's a lot of dependencies

00:06:41 ◼   ► that you kind of have some choice in.

00:06:43 ◼   ► Do I wanna be so wed to a particular platform

00:06:47 ◼   ► back end or system? Or do I want to be a bit more flexible and do, you know, usually it

00:06:52 ◼   ► ends up like the more custom you make it and how much of it you control, you're going

00:06:56 ◼   ► to have more like portability and be able to be like, you know, if this particular host

00:07:01 ◼   ► goes away, I can just get another one. And you're going to be able to look at the trade-offs

00:07:05 ◼   ► and make more choices than if you're just all in on one thing. That's when you start

00:07:10 ◼   ► to get a little bit awkward.

00:07:12 ◼   ► awkward.

00:07:13 ◼   ► Yeah, and that's why the selection of what you depend on, this is why I'm usually very

00:07:18 ◼   ► conservative with these. Obviously I try to minimize how many external services and companies

00:07:23 ◼   ► and things I depend on, but you're always going to depend on something, as you said.

00:07:27 ◼   ► This is why I always try to choose as conservatively as possible. So yeah, Apple could shut down

00:07:33 ◼   ► the entire app store and that would really be disruptive for us, but that's very unlikely.

00:07:39 ◼   ► The App Store's continued success is pretty important

00:07:42 ◼   ► to Apple as well, so I've aligned my incentives

00:07:45 ◼   ► with this now pretty old and pretty important thing

00:07:48 ◼   ► to its parent company that it is very unlikely

00:07:51 ◼   ► that Apple will do that and that will be a problem for me.

00:07:54 ◼   ► Facebook shutting down Parse, this thing they bought,

00:07:58 ◼   ► did not have that kind of luxury.

00:08:00 ◼   ► If you were a Parse customer six months ago

00:08:01 ◼   ► and you're evaluating this dependency,

00:08:04 ◼   ► it isn't that important to Facebook to keep this running.

00:08:08 ◼   ► So that this was foreseeable,

00:08:09 ◼   ► that this was a high risk of happening,

00:08:12 ◼   ► that this company made this service,

00:08:13 ◼   ► it got big, it got bought,

00:08:15 ◼   ► the parent company didn't really depend

00:08:16 ◼   ► on its continued operation for their core strategy.

00:08:19 ◼   ► So this was always gonna be a risk, right?

00:08:22 ◼   ► So if Linode, Linode, however we're supposed

00:08:26 ◼   ► to be pronouncing it, I say Linode, you say Linode,

00:08:27 ◼   ► they say Linode, if that particular Linux VPS host

00:08:32 ◼   ► gets shut down, well, that's unlikely

00:08:35 ◼   ► 'cause they're really big

00:08:36 ◼   ► and they've been around a while,

00:08:38 ◼   ► but even if that happens,

00:08:40 ◼   ► migrating away from that is not that big of a problem

00:08:43 ◼   ► because there are other Linux VPS hosts just like it.

00:08:48 ◼   ► And if every Linux VPS host went away,

00:08:52 ◼   ► you could get a Linux server somewhere

00:08:54 ◼   ► that behaved very similarly.

00:08:55 ◼   ► If every Linux server provider went away,

00:08:59 ◼   ► you could, as a last ditch, run one in your house.

00:09:02 ◼   ► You shouldn't, but you could.

00:09:04 ◼   ► So the transition options away from something

00:09:08 ◼   ► are also very important.

00:09:09 ◼   ► Parse shut down, they did a decent thing here

00:09:12 ◼   ► where they open sourced a big part of their server

00:09:15 ◼   ► and made it installable on your own stuff.

00:09:17 ◼   ► But what if they didn't do that?

00:09:18 ◼   ► Lots of things shut down and never do that

00:09:20 ◼   ► because they just either can't or won't

00:09:22 ◼   ► or don't feel like it.

00:09:24 ◼   ► So suppose you depend on an Amazon web service

00:09:29 ◼   ► for your business and Amazon shuts that down.

00:09:32 ◼   ► Most of the time, that is very hard to replace

00:09:35 ◼   ► 'cause they are so custom and proprietary,

00:09:37 ◼   ► you can't just kinda do your own thing.

00:09:39 ◼   ► If you depend on any kind of high-level service like this,

00:09:43 ◼   ► then it is always a risk that the more custom

00:09:47 ◼   ► and proprietary and high-level something is,

00:09:49 ◼   ► the risk of it being hard to replace

00:09:52 ◼   ► if it ever does go away increases.

00:09:54 ◼   ► - Yeah, and I think that's ultimately probably

00:09:57 ◼   ► like the enticement and why it's this weird tension

00:10:00 ◼   ► that you find yourself in as you're developing a service

00:10:02 ◼   ► as you're thinking about a feature, you're saying, "If I do it with this high-level

00:10:08 ◼   ► construct that this company's providing, I can save myself a lot of time up front,

00:10:15 ◼   ► because I'm not having to build that again."

00:10:19 ◼   ► If there's this solution that they've come up with that means that user authentication

00:10:24 ◼   ► is just a thing that I can just plug into my app, and it handles all the secure password

00:10:31 ◼   ► storing and email resets and all that kind of stuff.

00:10:34 ◼   ► Say there's a service that does that off the shelf.

00:10:36 ◼   ► That's days, weeks, months of time that you weren't spending building that thing, but

00:10:42 ◼   ► instead you're just kind of more integrating directly into your application.

00:10:46 ◼   ► And so it's enticing.

00:10:48 ◼   ► And you're getting this enticement at the benefit of that upfront time, but it's sort

00:10:53 ◼   ► of at the detriment of this risk that you're increasing in your application.

00:10:58 ◼   ► And maybe that makes sense, like if you're just kind of prototyping something and throwing

00:11:02 ◼   ► it out there, or you aren't in a situation that's very time limited, that you have to

00:11:07 ◼   ► get, you know, if you don't ship your app in a month, it's you're going to miss some

00:11:12 ◼   ► kind of market window or opportunity that or like, that's the only amount of like, you

00:11:16 ◼   ► just have that much money to make a run at it, and you just kind of have to, then great,

00:11:21 ◼   ► like it, there's nothing bad about those types of things.

00:11:25 ◼   ► But it's this weird tension that you're finding of because you're so locked in at that point,

00:11:30 ◼   ► you're setting yourself up for difficulty down the road.

00:11:33 ◼   ► Because it's not necessarily your short-term benefit and long-term pain, because developing

00:11:40 ◼   ► it yourself, there's also long-term pain.

00:11:42 ◼   ► It's a different kind of pain, but you have to then be the one who's maintaining it, or

00:11:46 ◼   ► when security issues happen, you're the one going in and patching your web server or the

00:11:54 ◼   ► the Linux distribution you're installing on your servers

00:11:56 ◼   ► or whatever, at some point there's always

00:11:58 ◼   ► long-term challenges with these things.

00:12:01 ◼   ► But the difference is more one of you're totally locked in

00:12:05 ◼   ► and at the whim of whatever that company is.

00:12:08 ◼   ► And unless you're their biggest customer,

00:12:11 ◼   ► which for the kind of people who I imagine

00:12:13 ◼   ► listen to a show like this, you're unlikely to be

00:12:15 ◼   ► a service provider's biggest customer,

00:12:19 ◼   ► you're just gonna be kind of like,

00:12:21 ◼   ► sort of washed back and forth based on

00:12:23 ◼   ► make sense for them.

00:12:24 ◼   ► And that may or may not be a position

00:12:27 ◼   ► that you find yourself in that you'd be comfortable with.

00:12:31 ◼   ► This episode of Under the Radar is brought to you by Hover.

00:12:34 ◼   ► Quite simply, Hover is the best way

00:12:35 ◼   ► to buy a managed domain names.

00:12:37 ◼   ► And when it comes to buying a domain name,

00:12:38 ◼   ► Hover is the first place I check.

00:12:40 ◼   ► Now, when you have an idea for a project,

00:12:42 ◼   ► naming it can be difficult.

00:12:43 ◼   ► When you finally get that name, you

00:12:44 ◼   ► want to be able to quickly and easily get the domains

00:12:46 ◼   ► that you need.

00:12:47 ◼   ► Hover provides a simple, fast, and hassle-free method

00:12:50 ◼   ► of buying domains.

00:12:51 ◼   ► I don't want to be faced with 1,000 screens

00:12:53 ◼   ► and all these add-ons, high prices,

00:12:55 ◼   ► all these custom weird services that seem like scams.

00:12:58 ◼   ► I just want to get in, select what I need,

00:13:00 ◼   ► buy it and get on with my life and building my new idea.

00:13:02 ◼   ► Hover makes this very, very easy.

00:13:04 ◼   ► Their search is very nice.

00:13:06 ◼   ► It suggests things for you if nothing's available.

00:13:08 ◼   ► They can search all the TLDs, all the crazy new ones,

00:13:11 ◼   ► in addition to all the nice old ones.

00:13:13 ◼   ► And they have .com domains starting at just $12.99 a year.

00:13:17 ◼   ► Great prices on all the other ones as well.

00:13:19 ◼   ► All of these include Whois privacy for free

00:13:22 ◼   ► with every hover domain, because they believe

00:13:24 ◼   ► that you shouldn't have to pay extra for something like that.

00:13:27 ◼   ► Obviously, you want to keep your private information private.

00:13:29 ◼   ► They also have fantastic customer support.

00:13:31 ◼   ► If you want to call them, they have a no hold, no wait,

00:13:34 ◼   ► no transfer telephone support policy.

00:13:36 ◼   ► When you call them, you talk to an actual human being,

00:13:38 ◼   ► not a robot, not a menu.

00:13:40 ◼   ► You don't have to say like, operator,

00:13:42 ◼   ► like those stupid speech menus.

00:13:43 ◼   ► It's a real human being you can just talk to directly.

00:13:46 ◼   ► They pick up the phone.

00:13:47 ◼   ► And if you do, of course, prefer the robots,

00:13:49 ◼   ► they also have great support documents

00:13:51 ◼   ► and support guides on their website

00:13:52 ◼   ► for getting everything you need,

00:13:53 ◼   ► and you can email them as well if you'd like.

00:13:56 ◼   ► And they also have a valid transfer service

00:13:58 ◼   ► where they can take all the hassle out of switching

00:13:59 ◼   ► from your current provider,

00:14:00 ◼   ► simply because they do it all for you.

00:14:02 ◼   ► You can just give them your login to your old provider

00:14:04 ◼   ► and they will transfer names for you if you'd like.

00:14:06 ◼   ► All that for free, of course.

00:14:08 ◼   ► They have so much more great stuff.

00:14:09 ◼   ► They have volume discounts,

00:14:11 ◼   ► they have custom email addresses, storage and forwarding,

00:14:13 ◼   ► and so much more stuff.

00:14:15 ◼   ► Check it out today at hover.com.

00:14:18 ◼   ► Use code perspective at checkout.

00:14:20 ◼   ► That is once again, Code Perspective at checkout,

00:14:23 ◼   ► and you will get 10% off your first purchase at hover.com,

00:14:26 ◼   ► and you will show your support for Under the Radar

00:14:28 ◼   ► and all of Relay FM.

00:14:29 ◼   ► Thank you very much to Hover for sponsoring this episode.

00:14:32 ◼   ► - So it seems like we should probably also dive now,

00:14:34 ◼   ► dive into kind of like what we do, how we approach this,

00:14:38 ◼   ► because I think we both have found ourselves

00:14:41 ◼   ► at the end of the thought process

00:14:44 ◼   ► on how we should bake back ends for our service

00:14:46 ◼   ► with the like, well, we're gonna build them ourselves,

00:14:48 ◼   ► and we're gonna build custom applications

00:14:51 ◼   ► running on Linux VPSs that we use.

00:14:56 ◼   ► And probably worth saying why we kind of do that.

00:15:00 ◼   ► - I mean, to me, first of all,

00:15:02 ◼   ► it's all about control for me.

00:15:03 ◼   ► I'm a control freak, and I wanna do everything myself,

00:15:05 ◼   ► and I want everything to be under my control,

00:15:08 ◼   ► because I don't want to have major parts of my roadmap

00:15:12 ◼   ► dictated by a dumb change in my host that,

00:15:16 ◼   ► oh, all of a sudden this entire thing I depend on

00:15:18 ◼   ► is shutting down and I gotta change that.

00:15:21 ◼   ► Apple gives us enough of those things,

00:15:23 ◼   ► with new device releases and everything,

00:15:25 ◼   ► but those are kind of an unavoidable part

00:15:27 ◼   ► of working with Apple.

00:15:28 ◼   ► But when it comes to running your services,

00:15:30 ◼   ► you control a lot more of that

00:15:31 ◼   ► and you can avoid those things.

00:15:33 ◼   ► And so I love that part of it.

00:15:35 ◼   ► And for me, it's also a lot about capability and cost,

00:15:39 ◼   ► low cost and just being able to do a lot.

00:15:43 ◼   ► CloudKit is very appealing in a lot of ways.

00:15:45 ◼   ► And if I was making a new app today,

00:15:48 ◼   ► I would think very hard about whether I could just do it

00:15:50 ◼   ► all on CloudKit and whether that would be

00:15:51 ◼   ► the right move for me.

00:15:53 ◼   ► But it is still limited in what it can do, what it can't do.

00:15:58 ◼   ► And so for me, a website, like a regular Linux backend,

00:16:02 ◼   ► is the default for me.

00:16:03 ◼   ► I know how to do it.

00:16:04 ◼   ► It really isn't that hard,

00:16:06 ◼   ► which we'll get into in a little bit.

00:16:07 ◼   ► It really isn't that hard.

00:16:08 ◼   ► And it is surprisingly capable for surprisingly little cost.

00:16:13 ◼   ► - Yeah, exactly.

00:16:14 ◼   ► I think the reasons are fairly similar for me. I think the thing that I like most is

00:16:20 ◼   ► being able to tailor the backend of my application to not necessarily the application, but it's

00:16:27 ◼   ► tailored to the way that I think and the way that I solve problems and the way that I'm

00:16:31 ◼   ► thinking about the problems that are being solved in my app. So when I'm dealing with

00:16:37 ◼   ► something like sync, like the generic term for one of the hardest problems in computer

00:16:42 ◼   ► science. I like that I can—I'm solving that problem in a way that makes sense to

00:16:50 ◼   ► me, that I'm not having to kind of shoehorn my application and the way I think about it

00:16:55 ◼   ► into the model that a service provider provides. And they say, "Well, we handle conflict

00:17:01 ◼   ► resolution using last updated wins," or something like that. And maybe that works,

00:17:07 ◼   ► maybe it doesn't. And so when you build it yourself, I actually understand it. I have

00:17:12 ◼   ► to go—I've gone through and I've made the decisions at the various levels of like,

00:17:18 ◼   ► "Well, I want this to work this way, I want this to work that way." And so then down

00:17:22 ◼   ► the road, when I'm debugging something, I have a better understanding of how I expect

00:17:28 ◼   ► it to work, and when things go wrong, I have a sense of where they might be going wrong.

00:17:32 ◼   ► Like, is this an app problem, or is this a web service problem? And ultimately, it probably

00:17:38 ◼   ► also just makes my apps better and makes me a better programmer. Like, having this breadth

00:17:42 ◼   ► of experience that at this point, I can build something all the way from the UI in the application,

00:17:50 ◼   ► the business logic inside of the application, and then all the way through to the web service

00:17:57 ◼   ► that's managing that information and a database at the back that's storing that information.

00:18:03 ◼   ► Having been able to do all those things is just good for me from a career and personal

00:18:09 ◼   ► development perspective. Like, I've learned to solve more problems doing it this way that

00:18:15 ◼   ► ultimately I think makes me a better developer. Like, I write less. My apps are probably better

00:18:20 ◼   ► because they're the kind of calls they're making. Like, I know what the server is trying to do with

00:18:25 ◼   ► those calls. And so you don't end up just like, "Well, this is like the naive, obvious solution.

00:18:30 ◼   ► I'll just kind of throw all this data at the server." Or I'll, "Hey, let me just ask for

00:18:35 ◼   ► for all of it, all the time, and because if the servers are overwhelmed, that's not my

00:18:41 ◼   ► problem. Those are things that ultimately probably make my apps better. And like you

00:18:45 ◼   ► were saying, it is kind of crazy how inexpensive it is to do a lot of these things now. A lot

00:18:52 ◼   ► of my things are just backed by two or three VPSs that cost—at a basic one, it's like

00:19:00 ◼   ► $20 a month or something like that, $20, $40 a month. For a lot of my applications, I end

00:19:04 ◼   ► up spending maybe $100 a month in servers, and that's really not too bad for the kind

00:19:13 ◼   ► of capability and the throughput and the number of users that you can support even just at

00:19:18 ◼   ► that level.

00:19:19 ◼   ► Oh yeah, I mean, even the $20 a month server level on a modern host like LineOut or Digital

00:19:25 ◼   ► you can get so much for this money now.

00:19:28 ◼   ► And when you're using boring old fast tools

00:19:31 ◼   ► like MySQL or Postgres and you have like

00:19:35 ◼   ► a modern web language in front of it,

00:19:37 ◼   ► you know you have, even in the old ones,

00:19:38 ◼   ► PHP, Ruby, you know like Python,

00:19:41 ◼   ► or more recently you might have like Go,

00:19:44 ◼   ► these are so fast, you can do so much,

00:19:47 ◼   ► you can support so much usage,

00:19:48 ◼   ► it's way more than you think.

00:19:50 ◼   ► 'Cause now you have these modern processors

00:19:52 ◼   ► doing the virtualization,

00:19:53 ◼   ► You have SSDs on almost all these hosts now.

00:19:56 ◼   ► It is incredibly fast to do.

00:19:58 ◼   ► And so you really can support a lot on very little hardware.

00:20:02 ◼   ► - Yeah, and I think ultimately that makes it a lot easier.

00:20:04 ◼   ► It's, they're the hardest problems I've ever had to solve.

00:20:08 ◼   ► The only time I kind of regretted doing backends myself

00:20:11 ◼   ► is they were the early days of Feed Wrangler,

00:20:14 ◼   ► my RSS syncing system, which I was doing stuff

00:20:18 ◼   ► that in retrospect was really foolish

00:20:21 ◼   ► and was just crushing my database.

00:20:24 ◼   ► My Postgres database was just constantly dying

00:20:28 ◼   ► and falling over.

00:20:29 ◼   ► And in retrospect, it was because I was being,

00:20:34 ◼   ► I'd made a few really bad assumptions upfront,

00:20:36 ◼   ► but even there, that's the only time I've ever really

00:20:41 ◼   ► had to do any low-level performance tuning

00:20:44 ◼   ► of any of my applications.

00:20:46 ◼   ► Otherwise, just out of the box,

00:20:48 ◼   ► things are just fast and work and it's fine in a way that,

00:20:53 ◼   ► it would be problematic if I really needed to be

00:20:56 ◼   ► a database administrator, a serious DBA,

00:21:00 ◼   ► whatever they call them these days,

00:21:02 ◼   ► doing that kind of work, but most of the times

00:21:04 ◼   ► I just install Postgres with the defaults,

00:21:07 ◼   ► tweak a few things the way I like it, and then it's fine.

00:21:10 ◼   ► And it just runs quickly enough for,

00:21:13 ◼   ► that my users don't even really notice

00:21:15 ◼   ► any kind of performance issues or problems.

00:21:17 ◼   ► - Yeah, I mean, you might think,

00:21:19 ◼   ► if you haven't done this before,

00:21:21 ◼   ► or the last time you did this was like 10 years ago,

00:21:23 ◼   ► you might think that running servers requires

00:21:25 ◼   ► lots of low-level tweaking and performance tuning

00:21:28 ◼   ► and getting these right config variables

00:21:29 ◼   ► to exactly the right buffer size and everything,

00:21:31 ◼   ► and you don't really need to do that anymore.

00:21:33 ◼   ► It's very rare for most people to need to get

00:21:37 ◼   ► that down into the nitty-gritty stuff.

00:21:38 ◼   ► It really is, as you said,

00:21:40 ◼   ► you can just install these things with the defaults,

00:21:43 ◼   ► and usually you're fine.

00:21:45 ◼   ► That's usually all you need to do,

00:21:46 ◼   ► because everything is just so good now.

00:21:48 ◼   ► There's so much headroom.

00:21:49 ◼   ► The software is very mature in a lot of these things,

00:21:53 ◼   ► and the hardware is very mature too.

00:21:54 ◼   ► So you really get a lot with just the defaults now.

00:21:58 ◼   ► - And I think one thing that I was kind of looking forward to

00:22:01 ◼   ► when we got into this topic is you said

00:22:03 ◼   ► you had a few little pro tips

00:22:05 ◼   ► for getting into this kind of administration,

00:22:07 ◼   ► because I think it can be a little bit intimidating

00:22:11 ◼   ► to go and install Linux.

00:22:15 ◼   ► Then you start to like, well, what version of Linux?

00:22:17 ◼   ► What should I do?

00:22:18 ◼   ► How to get started?

00:22:19 ◼   ► And I remember it being a little intimidating,

00:22:21 ◼   ► but at least the fun thing is once you get going,

00:22:24 ◼   ► there's tremendous resources,

00:22:26 ◼   ► and you can just kind of get going.

00:22:27 ◼   ► And once you know it, you know it,

00:22:28 ◼   ► because this stuff doesn't really change.

00:22:31 ◼   ► - Yeah, basically, Google is your friend,

00:22:34 ◼   ► not the corporate structure, but the search engine,

00:22:37 ◼   ► Stack Overflow, all these things.

00:22:39 ◼   ► These are your friend, because lots of people

00:22:42 ◼   ► have been running Linux servers for years.

00:22:44 ◼   ► And as you said, the tools and the commands

00:22:47 ◼   ► and what you need to do doesn't change very often.

00:22:50 ◼   ► Typically, you learn this stuff like once

00:22:52 ◼   ► and you have to learn something new maybe every two years.

00:22:55 ◼   ► It's pretty stable, it doesn't change much.

00:22:58 ◼   ► So number one tip I can give is to pick a very popular

00:23:03 ◼   ► but somewhat conservative Linux distribution

00:23:06 ◼   ► to do this with.

00:23:07 ◼   ► For years, I recommended CentOS,

00:23:09 ◼   ► which was based on Red Hat Enterprise Linux.

00:23:11 ◼   ► I think today, I think Ubuntu might have more momentum behind it. So I actually just managed

00:23:17 ◼   ► my first Ubuntu server recently and things are a little bit different but I was able

00:23:23 ◼   ► to figure it out. So between CentOS and Ubuntu, you can't really go wrong. Turn on auto updates

00:23:31 ◼   ► for as much of the system software as it makes sense to do that for. Usually every major

00:23:35 ◼   ► distro has a way to do this. It's very straightforward. That will take care of most security problems

00:23:40 ◼   ► for you. If you basically are not an idiot, which you're not, trust me, if you're not

00:23:46 ◼   ► an idiot and if you leave things mostly at their defaults with the distro and what it

00:23:50 ◼   ► comes with, modern Linux distros are very secure by default because they know that that

00:23:54 ◼   ► matters, like the defaults matter, so they've all adopted pretty conservative and pretty

00:23:59 ◼   ► secure defaults for the most part. Keeping things updated automatically is very easy

00:24:03 ◼   ► and things like that. On other high level stuff, only run the software that you need

00:24:08 ◼   ► to be running. And they're all very good at letting you manage this. So like, if you have

00:24:12 ◼   ► a server that you have your website on, don't also install like, "Oh, let me install FTP

00:24:16 ◼   ► so I can like trade files with my friends." No, just leave that off. That's just a liability.

00:24:20 ◼   ► Just don't do that, you know. Install what you need to install. And if you want to play

00:24:24 ◼   ► around with different things, you can create a second VPS for like five or 10 bucks a month

00:24:27 ◼   ► and play around on that. Don't play around on your main servers. Run only what you need

00:24:31 ◼   ► to be running on them. Take advantage of the built-in isolation in Linux machines, especially

00:24:38 ◼   ► with regard to networking. Almost every service that you'll be running will have some kind

00:24:44 ◼   ► of like listening port where you can say, "All right, this database should listen on

00:24:47 ◼   ► this interface, on this port." If you only have one server, make the internal stuff listen

00:24:52 ◼   ► on local host so that you can't log in to MySQL from outside. Like, you shouldn't need

00:24:58 ◼   ► to do that anyway. You should be doing things on the server if you need, you know, management

00:25:01 ◼   ► stuff. Lock that down. If you have multiple servers, use private networking. Every host

00:25:05 ◼   ► that's worth their salt supports private networking between your own machines. So if you have

00:25:10 ◼   ► multiple servers and you need to talk to each other, have them talk to each other only over

00:25:14 ◼   ► private interfaces. Have things like MySQL or Memcache listen only on private or local

00:25:19 ◼   ► interfaces. That helps a lot. Just make it, don't rely on like keeping your passwords

00:25:25 ◼   ► secure. Make it so that passwords don't even work from the outside. So that also applies

00:25:29 ◼   ► with things like SSH when you're doing login, remote login.

00:25:32 ◼   ► So disable root logins once you have a user set up,

00:25:35 ◼   ► have that user have pseudo access with the password,

00:25:38 ◼   ► and then that user account that you're logging in as,

00:25:40 ◼   ► say you're logging in as David,

00:25:41 ◼   ► make that the only user that can log in via SSH

00:25:43 ◼   ► and make that key authentication only,

00:25:46 ◼   ► disable password authentication in SSH.

00:25:49 ◼   ► This is very simple stuff to do.

00:25:50 ◼   ► You can Google how to do it.

00:25:52 ◼   ► So that, right there, you have no way

00:25:54 ◼   ► to log in with a password.

00:25:56 ◼   ► You have to have the encryption key to log in.

00:25:59 ◼   ► That knocks out massive brute force possibilities

00:26:03 ◼   ► and everything.

00:26:04 ◼   ► That helps so, so much.

00:26:06 ◼   ► Between that and private networking for private services,

00:26:09 ◼   ► you really eliminate a lot of problems.

00:26:11 ◼   ► Now, moving on slightly to user data.

00:26:14 ◼   ► Collect as little user data as possible to get your job done.

00:26:19 ◼   ► Because worst case scenario, somebody hacks into your server.

00:26:22 ◼   ► Worst case scenario, they take your database.

00:26:25 ◼   ► What do they have?

00:26:26 ◼   ► Think about it.

00:26:27 ◼   ► When you're designing your database,

00:26:29 ◼   ► you're designing your service,

00:26:30 ◼   ► what information do you really need from people

00:26:32 ◼   ► and what can you get away with not having?

00:26:34 ◼   ► If you don't need to get people's email addresses,

00:26:36 ◼   ► don't get their email addresses.

00:26:38 ◼   ► If you're taking passwords from people, hash those

00:26:41 ◼   ► so that people aren't getting just the MD5,

00:26:44 ◼   ► for God's sake don't do that.

00:26:45 ◼   ► Use secure password hashing like Bcrypt on strong settings.

00:26:49 ◼   ► There's lots of good practices for this,

00:26:50 ◼   ► lots of things to tell you how to do this.

00:26:52 ◼   ► I've considered even for overcast,

00:26:54 ◼   ► I do have the email addresses for people

00:26:56 ◼   ► because I figure I have email addresses

00:26:58 ◼   ► and I have hashed passwords with strong bcrypt,

00:27:01 ◼   ► but I'm like, I've been thinking recently,

00:27:02 ◼   ► do I even need the email address?

00:27:04 ◼   ► Could I hash that too?

00:27:06 ◼   ► 'Cause then you have, if you steal my database,

00:27:08 ◼   ► you just have no email addresses.

00:27:10 ◼   ► That would be amazing.

00:27:11 ◼   ► And I was thinking, if you hash the email address

00:27:14 ◼   ► so it works just like the password basically,

00:27:16 ◼   ► then you could still have logins,

00:27:18 ◼   ► you could still have password resets.

00:27:20 ◼   ► The only thing you really can't do is,

00:27:22 ◼   ► I can't email people randomly out of my database,

00:27:26 ◼   ► but I've never done that.

00:27:27 ◼   ► I don't send a newsletter, I don't do any of that stuff.

00:27:31 ◼   ► So, you know, stuff like that.

00:27:32 ◼   ► Think about just like what data you have,

00:27:34 ◼   ► what you're collecting, and what you can afford

00:27:36 ◼   ► not to collect.

00:27:37 ◼   ► Simple security measures beyond that, you know.

00:27:41 ◼   ► You should have database backups.

00:27:42 ◼   ► You should also be encrypting those backups.

00:27:44 ◼   ► There's built-in stuff, there's a crypt command

00:27:45 ◼   ► you can pipe tar through and everything like this.

00:27:48 ◼   ► Really simple stuff on Unix to do all this very securely.

00:27:51 ◼   ► Make sure, though, that you are testing these backups.

00:27:54 ◼   ► Make sure you can decrypt them.

00:27:55 ◼   ► So that's very important.

00:27:57 ◼   ► Don't store the encryption key only on the server

00:28:00 ◼   ► because then if that server gets wiped

00:28:02 ◼   ► or gets lost or whatever, you've lost your data

00:28:05 ◼   ► and your backup decryption key, that's no good.

00:28:07 ◼   ► One strategy I employ there is I write my database backups,

00:28:13 ◼   ► I copy them onto a write only S3 account.

00:28:17 ◼   ► So the credentials that are on the machines

00:28:20 ◼   ► can only write to the bucket,

00:28:21 ◼   ► they can't read or delete from it.

00:28:23 ◼   ► So that way if somebody hacks into the machine,

00:28:25 ◼   ► they can't also go and delete all my backups.

00:28:27 ◼   ► So I have separate credentials that I can pull the backups

00:28:32 ◼   ► off of there and restore that never live on my servers.

00:28:34 ◼   ► Those stay with me and my personal documents.

00:28:36 ◼   ► Those never live on the servers.

00:28:38 ◼   ► So keep things as secure and separate as you can

00:28:41 ◼   ► just by design like this.

00:28:43 ◼   ► And that's really about it for basic security stuff.

00:28:46 ◼   ► It really is not as hard as you think.

00:28:49 ◼   ► And you don't have to do very much.

00:28:51 ◼   ► You don't have to constantly keep on top of your servers

00:28:54 ◼   ► and be constantly babysitting, for the most part,

00:28:56 ◼   ► you set it up and it basically runs itself.

00:28:58 ◼   ► And if you set it up with sensible default

00:29:00 ◼   ► using conservative software and some basic security settings

00:29:03 ◼   ► like what I've said here, you can be pretty much fine.

00:29:06 ◼   ► - Yeah, exactly, and I think that it's the kind of thing

00:29:08 ◼   ► that if you can't do this kind of thing,

00:29:12 ◼   ► and if everything that Marco just ran through

00:29:15 ◼   ► is complete gibberish to you,

00:29:17 ◼   ► you should probably do something about that.

00:29:19 ◼   ► It's an important skill to be a developer,

00:29:21 ◼   ► to understand some of these basics,

00:29:23 ◼   ► sort of like the fundamentals that run the internet.

00:29:26 ◼   ► Like, you should understand what this is

00:29:28 ◼   ► and to sort of take control of that.

00:29:30 ◼   ► And like you said, just get a $5 like,

00:29:35 ◼   ► VPS somewhere and start messing around

00:29:36 ◼   ► and start seeing, you know, learning.

00:29:38 ◼   ► 'Cause that's how most people learn this stuff.

00:29:40 ◼   ► You just start doing it and you get better at it.

00:29:43 ◼   ► - All right, we're out of time this week.

00:29:45 ◼   ► Thanks for listening everybody.

00:29:46 ◼   ► And next week we're gonna go into a little more detail

00:29:48 ◼   ► about our server setups, lessons we've learned,

00:29:51 ◼   ► and how to minimize the workload.

00:29:52 ◼   ► We'll see you next week.

00:29:54 ◼   ► - Bye.