00:00:17 ◼ ► And this is sort of broadly the... it's one of these things that... it's not necessarily a novel concept, but I think it can have large implications for the way that you approach development.
00:00:44 ◼ ► And specifically this is the concept that I've been sort of struck by recently of how decisions we make early in the development process are disproportionately impactful on the final product that we're able to create.
00:01:00 ◼ ► And as a result, the thoughtfulness which we need to apply to any decision sort of scales exponentially with how close it is to the start, that the first decisions are massively important.
00:01:12 ◼ ► And obviously the most important one is probably to start with, like, should I build this or should I not build this, which is dramatic. It has a binary impact on the final product, whether the final product exists or not.
00:01:24 ◼ ► But then sort of falling off from there, we initially in any process, and this is obviously like can be a whole product or a new feature, which is where I've been wrestling with it recently.
00:01:33 ◼ ► But every decision we make in that early phase, or as we progress, is creating a foundation that all of the subsequent choices that we make have to build upon and have to adapt to and have to deal with.
00:01:45 ◼ ► And so if we make good decisions in the early stages of a process where we have something that is thoughtful and rich and is setting us, are setting ourselves up for success in the future, like, that's great.
00:01:59 ◼ ► If we go through the early stages in kind of a naive, not really thinking about it approach to things, then we can set ourselves up in the future for lots of problems or issues that we're going to have to sort of dig ourselves out of.
00:02:12 ◼ ► And an example of this that I think is interesting, and I wrote a blog post about this that I'll link to in the show notes, but I recently had an example where I was, where I'm working on a feature that does, gives you hiking directions for a hike that you want to go on.
00:02:26 ◼ ► So you give it essentially a series of waypoints that you'd like to go to, and I use the Mapbox Directions API to gather up the sort of a walking route between those points that you give it.
00:03:14 ◼ ► Like, well, do I want the full resolution or the simplified one? But what I choose here, I think is going to, like this is, my history will tell me that what I choose here will have massive implications on every other aspect of this feature.
00:03:38 ◼ ► And the solutions that I'm able to use for something, you know, can I put this reasonably in memory? Can I put this, can I store this in SQLite? Or do I need to store this in some other kind of custom thing?
00:03:47 ◼ ► Just like, you know, just using basic indices. Like, all of these things will be impacted by this decision and will subsequently also kind of change the things that are possible in the future.
00:03:59 ◼ ► And so in this case, it's like I spent a lot of time really thinking through what I do here. And what I ended up actually doing is I took the full version of it, which is the, you know, has the 20 times more points.
00:04:34 ◼ ► So I'm checking the boxes on the user experience side, but I'm not setting myself for lots of pain and discomfort down the road on the technical side, because I'm not, you know, for a typical, like a four mile hike, I'm only having to store maybe 100 points, which is should be fine, rather than storing 400 points, which would be less good, you know, four times slower, four times faster.
00:04:55 ◼ ► You know, four times slower, four times more difficult, four times running into things. And obviously, you need to be careful with this kind of stuff of being like premature optimization, because you don't.
00:05:11 ◼ ► So you but you have to like still wrestle with this and make good decisions. If for the future, you'd like to set up your future self for success with all the information that you have now and taking the time to be thoughtful in the, you know, in the present.
00:05:36 ◼ ► But then, you know, if they somehow still wouldn't because honestly, usually I'm usually whenever I underestimate the computing power of modern devices, I'm usually very pleasantly surprised.
00:06:05 ◼ ► So I would actually have just tried it. But then, you know, if for some reason it was still bad, I would have then probably made some kind of like dynamic, you know, point provider where you were like at different levels of detail, it will provide you a different number of points.
00:06:24 ◼ ► But all of that would probably be fairly complex and would set me up for like, you know, for instance, you know, if I if I had that kind of dynamic provider thing, first of all, the number of bugs that could create different levels of the stack is truly massive.
00:06:39 ◼ ► If you're if you're dealing with a different level of detail than you think you are with the data that you're getting back from it, but then also like the, you know, for you know, what if that operation of providing the points is asynchronous for some reason, you know, then you are introducing this complexity where you can only ever deal with points from an async context.
00:06:57 ◼ ► So there's, that's, that's the kind of thing that see, like, this is that actually that specific example that that has hit me kind of intentionally with with my new overcast rewrite and the, you know, so when you're when you're setting up, you know, early on in the project, you know, it's very similar, like with, you know, what I'm doing with the rewrite here.
00:07:16 ◼ ► Some of the decisions that you make are mainly meant to prevent you from running into major problems in the future. So for instance, I think starting a brand new project today, that's 100% Objective C and UI kit is probably going to cause you issues in the future sooner than if you had done it with Swift and Swift UI.
00:07:40 ◼ ► And so that's why today, again, if you're starting a brand new project, you should probably be using Swift and Swift UI, you know, for lots of reasons. But, you know, just you don't want to get to a point in the future where where all of a sudden, you have a problem because either the thing you're using just got, you know, deprecated or unsupported some some long way down the line, or, you know, there's some big capability that you can't use, or that will take you way more work than it'll take everyone else to use.
00:08:08 ◼ ► Believe me, I'm an expert in this area. And being on the wrong side of this. So some of the decisions you make early on are kind of about just like give yourself the biggest chances of a smooth future. And so that's things like tech decisions, language decisions, framework decisions.
00:08:27 ◼ ► You know, one of the reasons why I don't like to use third party frameworks is not that I hate everyone else's code. I mean, I do. But that's not the reason. The reason is that I've been burned before.
00:08:39 ◼ ► You know, earlier in my career, I've had issues where I was relying on a third party framework of some kind. And either I ran into too many problems with it because it wasn't of high enough quality, or it wasn't maintained anymore. Or it just, you know, you know, got discontinued or stop working on some new OS or some new platform or some new condition.
00:09:00 ◼ ► And it was just not being touched. And so I kind of learned the lesson there, you know, use the use the system stuff as much as possible. And whatever is not the system stuff, ideally use your own code as much as possible.
00:09:12 ◼ ► And I know that's there's lots of arguments against that for various reasons. But that's that's why I do that is because I've been burned before. And when you rely on someone else's, you know, for instance, if you're using someone else's UI framework, instead of the native platform UI frameworks,
00:09:26 ◼ ► you're creating a possible problem for yourself in the future. If either the platform makes that not work anymore, for some reason, or the platform introduces some cool new capabilities, your customers are going to want to you are going to want you to use that you can't use from that framework, or the frameworks vendor just stops working on it, they move on to something else, or they go take it in a different direction that you don't, you know, that you that you can't go with them or whatever.
00:09:50 ◼ ► So a lot of those decisions are, you know, the basic technical needs of the app, you know, what, what language you're using, what framework using how you're building it. But sometimes you make a decision to kind of enforce better discipline on yourself.
00:10:05 ◼ ► And that might be something like, you know, a simple thing, build settings, treating warnings as as errors, you know, then you are, you're kind of saying, I'm trying to prevent a lot of technical debt built up here, or a lot of, you know, bad practices to build up here.
00:10:20 ◼ ► So I will enforce some rule here, whether it's, you know, warnings as errors, whether it's turning on the Swift concurrency warning, because I strongly recommend that you do, you know, any of those kind of things are trying to avoid future or current technical debt, or bad practices with things.
00:10:36 ◼ ► Now that, that does restrain you in certain ways. So for instance, if you're treating warnings as errors, like that's going to slow you down during development, it's going to prevent you from doing certain things at all. But again, in the future, you're hoping you arrive at a point that is, that is, you know, desirable or better.
00:10:54 ◼ ► Now, what caught me on this track was, well, I mentioned, you know, what if your points, what if your point algorithm for your path, what if there's some kind of dynamic provider for that, but it's asynchronous, and you can only call it from asynchronous contexts?
00:11:06 ◼ ► Well, in my overcast rewrite, I've also been writing this SQLite framework called Blackbird alongside of it, and I decided to make Blackbird use a Swift actor as its main database, you know, blocking mechanism.
00:11:24 ◼ ► So, because SQL handles can only be accessed by one thread at a time, so I'm using an actor to manage the SQLite access, which means that for most times you're accessing the database, you are accessing it in an async context, which means that if there's some like, you know, system function or callback where you need to fetch a record from the database and return it synchronously, you basically can't.
00:11:48 ◼ ► Now, the reason I made this decision was to try to build, you know, one of the problems, I'm kind of fighting the last fight here, current overcast versions, and from 1.0 all the way to now, we're using the SQLite framework I built for Objective-C a million years ago called FC model.
00:12:06 ◼ ► And FC model, for various reasons, decides instead of using an actor, because it didn't exist back then, FC model serializes access to SQLite by making all database accesses happen on the main thread.
00:12:23 ◼ ► But the reason I did it is because it saves a lot of bugs, that you never have bugs of like things that responded to database events, being on background threads and updating the UI or having race conditions in the UI or anything like that.
00:13:28 ◼ ► So those decisions I made early on really, really restricted me and had the problem that now the current version of overcast in the App Store, it's using the main thread for all of its database reads and a lot of its processes as a result.
00:16:28 ◼ ► Yeah, and so I think there's definitely an element of that that is, yes, the early decisions you make have these profound impacts. Like in your case, it's the, you know, there are these whole sort of structural things that you can or can't do based on a choice you make when you're setting up your database tool at the beginning of a project.
00:16:46 ◼ ► And I think what it immediately makes me think of is like, so how do we know what are good choices to make? Like that seems like the crux of this issue is ultimately like what we want to do then is we want to make the right choice all the time.
00:17:06 ◼ ► Exactly right. Like how hard can that be? And I think obviously like we're being, you know, you know, we're being silly about that because the reality is like, how do you know? It's like, well, some of it is the trial and error of, you know, years of being in this career.
00:17:20 ◼ ► In terms of I've made every mistake, you know, I've made every mistake you could imagine probably at least once and many, many of them many times because like was I did, didn't learn the first time.
00:17:30 ◼ ► And so, you know, I've written code that was, well, did a lot of work on the main thread and it completely locks up the app and it's kind of terrible or awful or I didn't consider some of the implications of something would happen in performance or like it works fine when you only have 10 things.
00:17:55 ◼ ► So some of it is just, I think, an experience of I have a better intuition now for when I'm making a decision at the beginning of a process for how large of an impact this is going to have.
00:18:14 ◼ ► If so, I should be really thoughtful about that and try and think not just about like the immediate problems ahead of yourself, but this is where I think something that I started, I've started to do more and more is to think of the like, how could I see what features do I think this app is going to have in two or three years?
00:18:31 ◼ ► And how would I want to set structure things to be there, not in the sense of like over engineering everything that assuming that it's going to be around forever and, you know, deal with all these super complicated things.
00:18:42 ◼ ► But understand that, like, if you have obvious features or obvious directions that this is going to go or obvious challenges or issues that you would run into, like in the case of a podcast player, it's like, well, what is it?
00:18:53 ◼ ► Does this get, you know, scale to people who are subscribing to 100 shows to 1000 shows to 10,000 shows like and you have to pick a limit ultimately and say like, this is where I'm making my choices inside of this is the sort of expectation that I have.
00:19:08 ◼ ► And if I can build the decisions or make choices now that are set is setting that likely path, not just in the near future, but in the semi near future, then I'm, you know, probably going to be in a good place.
00:19:19 ◼ ► And it's like that intuition is just really tough. And it's one of these things that I bring it up as a something to be thoughtful of, because especially early in my career, I don't think I thought as enough about the long term choices of the, you know, of the decisions I was making.
00:19:34 ◼ ► And there are a few times that would have that it came back to bite me like there's I remember there was some project that I built worked on once this is back before my iOS days where I was a web developer, and it was I decided you know what I'm going to cash all of the API, like requests that people are making in a MySQL database that contains the main data of the of the app.
00:19:55 ◼ ► So anytime someone makes a request, I'm going to cash exactly what I'm sending back to them as a string that I'll put into the MySQL database. And then if they ask for the same thing in the future, I'll send it back.
00:20:04 ◼ ► naively, I thought that would be great, rolled it out to it rolled it out to production, and the entire production database was just completely destroyed because of the amount of, you know, heavy read and write of these giant, gigantic string values that were being pushed back and forth to it.
00:20:19 ◼ ► And the entire system was taken down and I had to roll back the change. And that was embarrassing. And it's like, it's like, so now I know don't do that. If you need a cache layer, you need to cache it somewhere else. You can't cache it in your main database. Like I've just learned that.
00:20:33 ◼ ► But I think there's just it's that sense of I at the beginning of that I just didn't really think I wasn't being thoughtful about like, well, what is this going to have? What implications is this going to have on performance on maintainability on all of those kinds of those kind of aspects?
00:20:46 ◼ ► I mean, recently, even I ran into a similar thing. I'm not in the same as like, so in the same kind of way that I'm working on all these mapping features for pedometer++, I'm working on a thing that lets you download offline maps.
00:20:59 ◼ ► Makes sense. Seems like an obvious feature I should have considered, you know, a year ago when I was working on this feature to start with. But as part of that, what I need to know is how which, you know, which map tiles have you downloaded? Like, that's a very simple fundamental question to, you know, which downloads which map tiles need to be downloaded.
00:21:18 ◼ ► And I had no mechanism or provision for that whatsoever in the app. And so right now, the best I can do is I need to ask the file system, does this file exist? Which is actually one of these things where you talk about, oh, you know, modern devices are pretty fast.
00:21:45 ◼ ► It was not internet now and I'm having to pay off this like poor choice I made back then, that rather than keeping track in some kind of, you know, like, what I'm ultimately doing is I'm making a little SQLite database that is a directory essentially of which files have been downloaded and which ones haven't.
00:22:01 ◼ ► Because you can ask SQLite, you know, which of these 38,000 things exist, and it'll come back to you trivially. Like, that's what it does. That's its whole thing. The file system, that's not what it does.
00:22:11 ◼ ► And so now I'm having to pay back this poor choice, whereas what I should have done, I should have anticipated that this was going to happen and, you know, built a system that when it downloads a file, it keeps track of that somewhere, other than the database.
00:22:23 ◼ ► And so now I'm having to pay that off and I'm having to build this thing, I have to do a migration where, you know, I sort of synchronously have to like pause the app to go and work out which files exist.
00:22:31 ◼ ► And so, you know, it's like, that's the disaster that I'm having to undo. And then it's like, I think I could have anticipated this one. In fact, I vaguely remember thinking about this back then, but being like, Oh, whatever, it'll be fine.
00:22:43 ◼ ► And whenever I tell myself, Oh, whatever, it'll be fine. Essentially, what I'm saying is it's like, future Dave, future Dave's gonna have a bad day. Like, I'm just making a problem for myself in the future.
00:22:53 ◼ ► And unfortunately, being an independent developer, that problem is me. Like, I'm gonna have to deal with that. There's no other person who I'm putting in a rough spot. I'm just setting myself up for failure in the future.
00:23:03 ◼ ► Yeah, I think a lot of times we spend a lot of our mental energy when doing a new project worrying about what if this doesn't work? What will make this not work? But we often don't think enough about what if this succeeds and I have to be, you know, not only maintaining this in five years, ten years, but what if this grows?
00:23:28 ◼ ► Like, how does this scale? You know, what if I get a lot of users and in some part of them uses the app way more heavily than I've planned for? Like, you know, as you were saying, what if somebody has downloaded thousands of those map tiles, you know, like as part of a huge, you know, traveling or, you know, that's possible.
00:23:45 ◼ ► What if somebody has 500 podcast subscriptions? Like, that's possible. And again, not even as uncommon as you would think. And so you have to think, you know, whatever decisions I'm making here, is this, like, how manageable can I make things in five or ten years by decisions I'm making now?
00:24:04 ◼ ► And a lot of that comes down to, again, just, you know, first of all, wise tech decisions early on, like, you know, not building against some weird third party framework for your entire UI or whatever. But also, I think it's important to try to not close doors unnecessarily.
00:24:19 ◼ ► So, and this is very difficult, because, you know, we don't have foresight, really. We don't have perfect predictions. We can't tell what the future will bring. But you can kind of tell, like, you know, am I, is some decision I'm making, is this going to make it difficult to change later?
00:24:34 ◼ ► So for instance, like, I mean, this is, God, this is one of the reasons why I hate running servers. Because I have all this user data in my servers, in these giant MySQL servers, and I, some of those decisions I made early on with, like, how the data is structured, how it works, how things are optimized so the servers don't get crushed.
00:24:52 ◼ ► Some of those decisions make it more difficult for me to either change the way certain things behave or to add certain types of new features, because they will, they would either require some massive migration on the servers that is just too cumbersome and time consuming and dangerous to attempt, or they would crush the servers under too much load, and I don't want to deal with that.
00:25:13 ◼ ► And so the whole decision of how, you know, first of all, you know, the server database schema and layout and how certain things work, that has massive long running implications if you actually do last a long time.
00:25:35 ◼ ► And, you know, if I was doing things different, if I was doing things fresh today, from the entire product perspective, like if I didn't already have users, that's another thing, like, you know, when you already have customers and users of your app, it becomes harder to make changes down the road.
00:25:50 ◼ ► Because people don't like change, they don't like for things to all of a sudden be different in the app they've been using for years, if it's not the way they want it to be, or if it's different from the way they were using it, or if it makes something that they did either not possible anymore or more cumbersome.
00:26:08 ◼ ► So even having users can restrict you a lot. But, you know, again, like the earlier on you make some of the decisions, the better. But ultimately, you're not going to be perfect at making the decisions. So the more flexible you can be, the better. The more potential flexibility you can leave on the table, the better.
00:26:24 ◼ ► So, you know, make decisions early on and build things early on to be changeable and to be flexible and to be resilient to change. And I know that's easier said than done, but to keep that in your mind, like, you know, what would happen if I decided to change the entire way this data is stored?
00:26:43 ◼ ► What would happen if I moved from a client-side model to a server-side model or vice versa? Like, can I build the database layer, can I build the structure of the app conceptually, or can I build, you know, the way I'm processing this to account for those kind of changes?
00:26:57 ◼ ► What if iCloud gets really good in the future? Or what if iCloud gets really bad in the future? How do I account for those? How do I build the app to be a little flexible on that kind of thing?
00:27:13 ◼ ► Yeah, and I think there's definitely this element of trying to wait to make decisions until you have more information is maybe one way I think that the difficulty with so many of these things is that, yeah, the more you can sort of preserve optionality for the future is awesome.
00:27:41 ◼ ► I think I've been doing this for the summer with my backpack-driven development where I just tend to build lots of small prototypes and experience lots of different things in these kind of like development spikes into a problem.
00:27:51 ◼ ► Rather than trying to sort of tackle it properly from the beginning, I tend to go around and explore a lot as I'm trying to gather information so that I can make a better, more informed decision for the future.
00:28:15 ◼ ► But either way, even if you're not making that a specific goal that you're doing, the later you can push a decision into the development process, the more information you inevitably and naturally will have.
00:28:49 ◼ ► But you can set yourself up for making a better and more informed choice that you're going to regret less by waiting until you have the most information you possibly can before you make that choice.