ATP

00:00:00 ◼   ► Unfortunately, the barbecue place I think is closed.

00:00:02 ◼   ► - Is that part of the ATP tradition?

00:00:04 ◼   ► - Yeah, usually my editing reward,

00:00:06 ◼   ► barbecue places are typically only open

00:00:07 ◼   ► from Thursday through Sunday.

00:00:08 ◼   ► That's just like a barbecue cultural thing.

00:00:10 ◼   ► I don't know why. - What?

00:00:11 ◼   ► - So usually, my Thursday tradition is I edit the show,

00:00:15 ◼   ► somewhere around noon, hopefully I'm done,

00:00:17 ◼   ► or at least I'm almost done, I take a barbecue break.

00:00:20 ◼   ► That's my reward to myself for a solid morning of work.

00:00:24 ◼   ► (laughs)

00:00:25 ◼   ► - But no more of that, right?

00:00:27 ◼   ► - What am I gonna eat?

00:00:28 ◼   ► - Yeah, I don't know, I don't know.

00:00:30 ◼   ► - God, I can't make this myself.

00:00:33 ◼   ► A lot of the stuff that I eat, I can make myself.

00:00:35 ◼   ► I can't make this myself.

00:00:36 ◼   ► - You got your own coffee roaster.

00:00:37 ◼   ► It's only a matter of time

00:00:38 ◼   ► for you to get your own smoker in the backyard.

00:00:40 ◼   ► - Yep, that's true.

00:00:41 ◼   ► (electronic beeping)

00:00:42 ◼   ► - So tonight, we have a special episode for you,

00:00:45 ◼   ► and we are not really gonna be covering the coronavirus

00:00:48 ◼   ► and all of that terrible news,

00:00:50 ◼   ► but we're hoping that with this show

00:00:52 ◼   ► and hopefully future shows,

00:00:54 ◼   ► we'll be able to give you a little break from worrying,

00:00:57 ◼   ► because gosh knows, I'm doing enough for everyone.

00:00:59 ◼   ► So with that in mind,

00:01:01 ◼   ► we actually have something extremely special planned,

00:01:04 ◼   ► and I wish I could say that we planned it

00:01:05 ◼   ► specifically during a pandemic,

00:01:07 ◼   ► but thankfully we didn't, but nevertheless, here we are.

00:01:10 ◼   ► And instead of some cutesy, funny intro,

00:01:12 ◼   ► we're just gonna get right to it

00:01:13 ◼   ► and say hello again to Chris Lattner.

00:01:16 ◼   ► - Hi guys, it's great to be back.

00:01:18 ◼   ► - Well, thank you, thank you for coming,

00:01:20 ◼   ► and we are very excited to have you.

00:01:22 ◼   ► And I figure we could start by just asking,

00:01:24 ◼   ► what are you up to, man?

00:01:27 ◼   ► - Lots of things, aside from sheltering in place.

00:01:29 ◼   ► Staying very busy, I started a new job at Sci-5

00:01:33 ◼   ► a couple of months ago, just under two months ago now.

00:01:37 ◼   ► And so I'm learning a lot of new things

00:01:39 ◼   ► and I'm exploring new areas,

00:01:41 ◼   ► also staying in touch with a lot of other familiar things.

00:01:44 ◼   ► Just generally keeping really busy.

00:01:47 ◼   ► Actually, this weekend finished a package shed,

00:01:51 ◼   ► which is a little hut looking thing

00:01:53 ◼   ► that has a tiny roof on it

00:01:55 ◼   ► that keeps packages from getting rained on.

00:01:58 ◼   ► - Marco just made a house for his garbage, very similar.

00:02:00 ◼   ► Yes. - Yeah, yeah.

00:02:02 ◼   ► - Yeah, does yours also keep raccoons out?

00:02:04 ◼   ► - No, it doesn't, but the kids like playing it.

00:02:08 ◼   ► - So last time we talked to you, it was like three years ago,

00:02:10 ◼   ► and I think we talked to you like right after you left Apple.

00:02:13 ◼   ► Since we last talked, you went to Tesla,

00:02:16 ◼   ► then you left Tesla and went to Google,

00:02:18 ◼   ► then you left Google and went to Sci-5.

00:02:20 ◼   ► Can you give us whatever kind of summary

00:02:23 ◼   ► you feel comfortable with?

00:02:25 ◼   ► Explain that journey to us.

00:02:27 ◼   ► - Okay, so how should I say this?

00:02:29 ◼   ► Let's start with Tesla.

00:02:30 ◼   ► The journey into Tesla was a big adventure,

00:02:33 ◼   ► one that I knew was fraught with peril and opportunity.

00:02:36 ◼   ► The Tesla Autopilot team had been through several leaders

00:02:41 ◼   ► and they each had short time horizons.

00:02:44 ◼   ► And my mental thought process going into this is that,

00:02:48 ◼   ► you know, success over the long term wasn't highly probable,

00:02:52 ◼   ► but I bet I would have a lot of really interesting

00:02:55 ◼   ► life experiences, learn a lot,

00:02:56 ◼   ► and then have some more stories, which I do,

00:02:59 ◼   ► that I don't really wanna share.

00:03:01 ◼   ► (laughing)

00:03:02 ◼   ► The exit was Elon and I had specific differences

00:03:07 ◼   ► of opinion about leadership,

00:03:08 ◼   ► decided that we could not get along together,

00:03:12 ◼   ► and then he and I decided to part ways.

00:03:15 ◼   ► The entire Autopilot team is very challenging,

00:03:18 ◼   ► but also really awesome and really exciting team.

00:03:21 ◼   ► That's a very specifically weird dynamic

00:03:24 ◼   ► and Tesla is a very fascinating place.

00:03:27 ◼   ► - I always wonder for people who have experiences

00:03:32 ◼   ► and jobs at important companies during exciting times,

00:03:35 ◼   ► so on and so forth, very often like most of them will,

00:03:38 ◼   ► like you will not wanna talk about specifics

00:03:39 ◼   ► 'cause like whatever, you move on, you learn something,

00:03:42 ◼   ► you don't wanna dish a bunch of dirt or whatever.

00:03:44 ◼   ► But I do always wonder if there comes a time where we'll,

00:03:48 ◼   ► we've referred to this a lot on the show

00:03:49 ◼   ► where they'll be like, oh, everyone's retired

00:03:51 ◼   ► and it's 50 years after all their careers happened

00:03:54 ◼   ► and they wanna write about, here's what really happened,

00:03:56 ◼   ► here's what it was like in the early days making the iPhone.

00:03:58 ◼   ► A couple people have had books like that,

00:04:00 ◼   ► like Ken Kuchen, did we go on that?

00:04:02 ◼   ► Do you ever think about sometime in your old age

00:04:05 ◼   ► when you're retired and the world has moved on

00:04:08 ◼   ► and we're all using our holographic AR glasses, phones,

00:04:11 ◼   ► or whatever, would you write a tech memoir,

00:04:14 ◼   ► here's what it was like to be at these important companies

00:04:17 ◼   ► during these important times and here are my experiences

00:04:19 ◼   ► or do you just have no interest

00:04:20 ◼   ► in ever sharing that information?

00:04:23 ◼   ► - I have no objection to sharing that.

00:04:24 ◼   ► I think that for me it's just a question of time.

00:04:28 ◼   ► It's also a question of talent.

00:04:29 ◼   ► I don't know that I'd be any good at writing such a thing.

00:04:32 ◼   ► It would probably turn out really dry and boring.

00:04:34 ◼   ► - Well, they have people for that though.

00:04:35 ◼   ► - Yeah, I suppose.

00:04:37 ◼   ► I guess, I mean, I can tell you things that I liked

00:04:39 ◼   ► and disliked about Tesla if that's what you're asking.

00:04:41 ◼   ► - Yeah, I know it was more of a meta question of like,

00:04:44 ◼   ► 'cause you don't wanna talk about it

00:04:45 ◼   ► like when you're still in your career

00:04:46 ◼   ► 'cause you don't wanna burn any bridges

00:04:48 ◼   ► and so on and so forth,

00:04:49 ◼   ► but if the job that you had was of historical significance,

00:04:53 ◼   ► and I think many of your jobs have been,

00:04:55 ◼   ► it's the type of information that is valuable to history

00:04:59 ◼   ► and useful for people in the future to know

00:05:01 ◼   ► what was it like.

00:05:03 ◼   ► If you could read Andy Grove's book or whatever

00:05:05 ◼   ► about Intel in the early days

00:05:07 ◼   ► or you wanna read about the creation of the transistor

00:05:09 ◼   ► or the creation of Unix or what it was like to make C

00:05:12 ◼   ► or the early days of the Google search engine,

00:05:14 ◼   ► I think a lot of your experience already falls

00:05:17 ◼   ► into the category of being worthy of being recorded

00:05:20 ◼   ► from your personal perspective when you're retired sometime.

00:05:24 ◼   ► And again, we say, oh, I don't have enough time.

00:05:25 ◼   ► Someday, I don't know if this is true,

00:05:27 ◼   ► maybe you think this is not true,

00:05:27 ◼   ► I assume someday you will retire.

00:05:29 ◼   ► Now I'm starting to question that of saying

00:05:31 ◼   ► maybe you're just gonna drop dead

00:05:33 ◼   ► in front of the keyboard someday, but assuming--

00:05:36 ◼   ► - I think that's quite possible, yeah.

00:05:38 ◼   ► - Assuming you ever do retire,

00:05:40 ◼   ► I think we would all love to read that book.

00:05:41 ◼   ► - Yeah, I mean, I'd be very open to writing that.

00:05:43 ◼   ► I just need to have the spare time

00:05:44 ◼   ► to actually collect the thoughts and--

00:05:46 ◼   ► - It's like retirement to you.

00:05:48 ◼   ► Spare time is what you get when you stop working.

00:05:50 ◼   ► (laughing)

00:05:52 ◼   ► - I mean, there's an aspect of my personality

00:05:54 ◼   ► that I don't seek out the easy, cushy jobs.

00:05:57 ◼   ► - I know, I know.

00:05:58 ◼   ► - I seek out the hard things that seem worthwhile.

00:06:01 ◼   ► - All right, so from now on, we can go to Google now,

00:06:03 ◼   ► which is a hard thing that seemed worthwhile, I guess.

00:06:06 ◼   ► - Yeah, well, so I guess a couple of different things.

00:06:07 ◼   ► So when I joined Google, I joined the TensorFlow team,

00:06:11 ◼   ► and actually at the time, I had talked to Sci-5,

00:06:15 ◼   ► and I had decided that Sci-5 was too early on,

00:06:18 ◼   ► and I didn't really understand where they were going,

00:06:20 ◼   ► and it didn't seem like the right thing.

00:06:21 ◼   ► And when I was talking with Google,

00:06:26 ◼   ► the pitch was pretty simple.

00:06:27 ◼   ► I was very interested in machine learning technology.

00:06:29 ◼   ► That's something I was interested in back at Apple,

00:06:32 ◼   ► but wasn't able to do anything with it.

00:06:34 ◼   ► In the case of Tesla, one of the things I did

00:06:37 ◼   ► was I force pivoted the technology stack

00:06:40 ◼   ► off of this other machine learning framework called Caffe

00:06:43 ◼   ► onto TensorFlow.

00:06:44 ◼   ► And so we moved to TensorFlow to get multi-GPU

00:06:46 ◼   ► training support and other things like that.

00:06:49 ◼   ► And in the process of doing that,

00:06:51 ◼   ► I learned quickly that TensorFlow is a good thing,

00:06:54 ◼   ► but it was a ways away from being a great thing.

00:06:58 ◼   ► And so one of the things that attracted me to Google

00:07:00 ◼   ► was that TensorFlow is an incredibly important

00:07:02 ◼   ► technology platform.

00:07:06 ◼   ► It's also very much a developer tool,

00:07:07 ◼   ► which I have a lot of experience with.

00:07:10 ◼   ► The machine learning aspect, I think,

00:07:12 ◼   ► was very appealing to me.

00:07:13 ◼   ► I had a good friend that said,

00:07:14 ◼   ► "Hey, if you wanna learn about machine learning technology,

00:07:16 ◼   ► "go to one of the best places in the world that's doing it."

00:07:18 ◼   ► And I think that was, I don't regret that at all.

00:07:21 ◼   ► And so the calculus going to Google was,

00:07:26 ◼   ► Google's also a legendary, amazing place to work,

00:07:29 ◼   ► a lot of smart people.

00:07:30 ◼   ► I know many people that are there.

00:07:32 ◼   ► And so I was very excited about that.

00:07:33 ◼   ► And I'm also very excited that I joined

00:07:35 ◼   ► and had a great time there and did some amazing things.

00:07:38 ◼   ► - Some of the stuff you did at Google was,

00:07:40 ◼   ► I guess, working on TensorFlow,

00:07:41 ◼   ► but also there was a Swift angle on that as well.

00:07:44 ◼   ► It was kind of, I don't know if,

00:07:46 ◼   ► well, you can tell me which was the cart

00:07:48 ◼   ► and which was the horse,

00:07:49 ◼   ► but basically modifications were made to Swift

00:07:51 ◼   ► that made it easier to work with machine learning stuff.

00:07:54 ◼   ► And they weren't features just for that purpose,

00:07:56 ◼   ► but they happened to lend themselves to that purpose,

00:07:58 ◼   ► and you happened to be doing that with Swift at Google.

00:08:01 ◼   ► So it's all, how does that work out, how did that connect?

00:08:04 ◼   ► - Yeah, I can explain how that all went down.

00:08:06 ◼   ► So what I was hired to do, my starter job, you could say,

00:08:10 ◼   ► is Google was developing this machine learning accelerator

00:08:14 ◼   ► called a TPU.

00:08:16 ◼   ► There's multiple generations of TPU.

00:08:17 ◼   ► The one I was working on was the first training accelerator.

00:08:20 ◼   ► The idea of TPUs is to use custom silicon,

00:08:23 ◼   ► a specifically machine learning design processor,

00:08:25 ◼   ► to be able to do machine learning training,

00:08:28 ◼   ► both faster than you can do with a GPU,

00:08:30 ◼   ► but also, and I think more importantly,

00:08:32 ◼   ► at a much bigger scale.

00:08:33 ◼   ► And they were just at the point in time

00:08:34 ◼   ► where the hardware was now getting installed.

00:08:38 ◼   ► They had significant software challenges,

00:08:40 ◼   ► integration challenges, programming challenges,

00:08:42 ◼   ► and they wanted to launch it in cloud.

00:08:44 ◼   ► And so making a public product out of something

00:08:47 ◼   ► that's not quite ready yet was a big challenge.

00:08:51 ◼   ► And so my starter project was to get cloud TPUs to market.

00:08:56 ◼   ► You can now get access to them now through Google Cloud.

00:08:59 ◼   ► They're a great product.

00:09:00 ◼   ► They're really, really awesome.

00:09:02 ◼   ► And so I started working on that.

00:09:04 ◼   ► Along the way, there's this discussion about,

00:09:06 ◼   ► okay, well, how do we get machine learning technology

00:09:08 ◼   ► to go to the next step?

00:09:10 ◼   ► So TPUs are an amazing hardware platform,

00:09:13 ◼   ► but they don't really touch on the programming model.

00:09:15 ◼   ► Now, as part of building the TPU platform,

00:09:18 ◼   ► there are many, many things about it

00:09:20 ◼   ► that are quite different than the normal places

00:09:23 ◼   ► that machine learning frameworks came from.

00:09:24 ◼   ► So one example of that is that they are thousands of chips.

00:09:28 ◼   ► It's like a supercomputer,

00:09:29 ◼   ► petaflops of compute in many racks of a machine.

00:09:33 ◼   ► And so the way you think about this

00:09:35 ◼   ► and the way you wanna program it is slightly different.

00:09:37 ◼   ► At the same time, there are other trends in the industry,

00:09:40 ◼   ► including a framework that was new at the time

00:09:42 ◼   ► called PyTorch, which came out of the Facebook

00:09:45 ◼   ► AI research group.

00:09:46 ◼   ► And PyTorch was pushing towards more dynamic,

00:09:50 ◼   ► more natural language processing models,

00:09:54 ◼   ► things like this, where you don't have as,

00:09:57 ◼   ► where the programming model isn't as static

00:09:58 ◼   ► as TensorFlow is at the time.

00:10:00 ◼   ► And so there's a bunch of interesting discussions about,

00:10:02 ◼   ► well, how do we get the programming model to move forward?

00:10:07 ◼   ► And it became apparent to me, you might not be surprised,

00:10:09 ◼   ► that Python was not really great for this, right?

00:10:12 ◼   ► Because a lot of where the Python world was coming from

00:10:16 ◼   ► is saying, because, so I should say,

00:10:20 ◼   ► a lot of where TensorFlow 1 was coming from

00:10:22 ◼   ► is saying Python is a slow language.

00:10:25 ◼   ► And so because it's a slow language,

00:10:26 ◼   ► what we'll do is we will use Python

00:10:29 ◼   ► so that you can kind of script together

00:10:31 ◼   ► some code that constructs a graph.

00:10:34 ◼   ► And then after you've constructed the graph,

00:10:35 ◼   ► we'll go use C++ code to run it really fast.

00:10:38 ◼   ► Okay, now this is one of the reasons that TensorFlow 1

00:10:43 ◼   ► was a very, what I call a static programming model,

00:10:45 ◼   ► is because you create this graph,

00:10:47 ◼   ► and anything you can encode in the graph, you're good.

00:10:50 ◼   ► You can just use graph nodes

00:10:52 ◼   ► and you can express these computations this way.

00:10:54 ◼   ► But if you can't express in the graph, you're kind of stuck.

00:10:56 ◼   ► You have to go hack TensorFlow and rebuild it yourself.

00:10:58 ◼   ► You have to go become a machine learning framework engineer

00:11:01 ◼   ► and learn about compilers, all this stuff.

00:11:03 ◼   ► And that's beyond the abilities of most machine learning,

00:11:07 ◼   ► data scientists type people, right?

00:11:09 ◼   ► And as it should be.

00:11:10 ◼   ► And so what PyTorch did very early on is it said,

00:11:14 ◼   ► okay, well, we'll shift to this dynamic programming model

00:11:16 ◼   ► where PyTorch, if you squint and look at it the right way,

00:11:20 ◼   ► it's basically the thinnest layer of Python

00:11:22 ◼   ► that you can have on top of a really fast C++ runtime.

00:11:26 ◼   ► And so both PyTorch and TensorFlow of the day,

00:11:30 ◼   ► we're looking at, given that Python is the way it is,

00:11:34 ◼   ► how do we work around the limitations

00:11:35 ◼   ► and how do we make something that is acceptable?

00:11:38 ◼   ► And even today, if you want to deploy

00:11:41 ◼   ► a machine learning model, you don't deploy Python.

00:11:43 ◼   ► And so if you look at both the TensorFlow

00:11:46 ◼   ► and the PyTorch standard ways of using them,

00:11:48 ◼   ► what you do is you write your training logic in Python,

00:11:52 ◼   ► and then you use some deployment mechanism.

00:11:54 ◼   ► In the case of TensorFlow, it's a TensorFlow graph.

00:11:56 ◼   ► In the case of PyTorch, it's their TorchScript solution.

00:12:00 ◼   ► And then you end up loading it up

00:12:02 ◼   ► into a bunch of C++ code,

00:12:03 ◼   ► and then you deploy the C++ code.

00:12:06 ◼   ► And so where Swift for TensorFlow came from

00:12:07 ◼   ► is this idea of saying, hey,

00:12:09 ◼   ► well, if we take a more modern language,

00:12:10 ◼   ► we can have better language-integrated features,

00:12:13 ◼   ► which I'm happy to talk about

00:12:14 ◼   ► if you want to geek out about that.

00:12:15 ◼   ► You can have deployment built in,

00:12:18 ◼   ► so you write it in one thing, output a different thing,

00:12:22 ◼   ► bolt it together with C++ code.

00:12:24 ◼   ► You can just have one solution that scales.

00:12:26 ◼   ► And by doing this, you allow people to move faster,

00:12:28 ◼   ► you get research flowing into production faster,

00:12:29 ◼   ► and you get a lot of other great benefits from that.

00:12:32 ◼   ► - So you're basically like writing in Swift,

00:12:35 ◼   ► and Swift is actually also the thing that run,

00:12:38 ◼   ► but you're still calling into libraries

00:12:40 ◼   ► and other languages from Swift?

00:12:42 ◼   ► - Yeah, so let me dive into how Swift for TensorFlow works.

00:12:46 ◼   ► So Swift for TensorFlow is a project with, I think,

00:12:50 ◼   ► several major components.

00:12:52 ◼   ► One of the interesting things about machine learning

00:12:54 ◼   ► is that you, and there's many different forms,

00:12:57 ◼   ► but one of the most popular forms uses

00:12:59 ◼   ► what's called backpropagation.

00:13:01 ◼   ► And so you do, when you're training,

00:13:05 ◼   ► you shove a bunch of data into a model,

00:13:07 ◼   ► and as you're shoving data into the model,

00:13:08 ◼   ► you're accumulating what are called gradients,

00:13:10 ◼   ► which are the updates for the weights in the model.

00:13:13 ◼   ► The weights are basically the parameters that you,

00:13:16 ◼   ► actually, let me take a step back.

00:13:18 ◼   ► What is a machine learning model?

00:13:19 ◼   ► A machine learning model is a set of computation.

00:13:23 ◼   ► It's a function, and the thing that's,

00:13:25 ◼   ► the distinguishing feature of a machine learning model

00:13:28 ◼   ► is that instead of it being a function

00:13:30 ◼   ► where you write it all manually in code,

00:13:32 ◼   ► you write the structure manually,

00:13:34 ◼   ► but then you learn and train the weights.

00:13:37 ◼   ► And so I look at a machine learning model

00:13:39 ◼   ► when you deploy it as, it's a function

00:13:41 ◼   ► that has all this trainable state behind it.

00:13:44 ◼   ► Now, where does that trainable state come from?

00:13:46 ◼   ► Well, the way machine learning training works

00:13:48 ◼   ► is that you take a version of the model you wanna deploy,

00:13:53 ◼   ► and you start shoving lots of data through it,

00:13:55 ◼   ► and then you use what's,

00:13:57 ◼   ► and I don't wanna go into calculus here,

00:13:58 ◼   ► but you accumulate what are called gradients.

00:14:02 ◼   ► Gradients are, you can think of it as like derivatives.

00:14:06 ◼   ► If you think about calculus one,

00:14:07 ◼   ► it's, you know, you can take F and you get F prime,

00:14:10 ◼   ► which is the derivative of a function.

00:14:12 ◼   ► The gradients in a machine learning case

00:14:13 ◼   ► are just higher dimensional versions of that same idea.

00:14:17 ◼   ► And by using those in your machine learning training system,

00:14:20 ◼   ► what you do is you shove a bunch of data through it,

00:14:22 ◼   ► you see, hey, I can get closer to a better answer

00:14:25 ◼   ► if I nudge all the weights in this direction,

00:14:27 ◼   ► then you do it again, you nudge all the weights,

00:14:29 ◼   ► you do it again, you nudge all the weights, you do it again.

00:14:32 ◼   ► Eventually your training converges

00:14:33 ◼   ► and you have a set of weights,

00:14:34 ◼   ► and now you can ship those weights as a binary blob

00:14:37 ◼   ► and deploy the quote unquote graph for the model,

00:14:41 ◼   ► but also the weights that go with it.

00:14:43 ◼   ► Now, the way that it works

00:14:45 ◼   ► is that you have to do that calculus thing.

00:14:46 ◼   ► You have to be able to compute the gradient

00:14:48 ◼   ► or you have to have compute the backwards version

00:14:51 ◼   ► or the derivative version of a function.

00:14:53 ◼   ► So there's lots of different ways of doing this.

00:14:56 ◼   ► And so one of the contributions of Swift for TensorFlow

00:14:58 ◼   ► is this idea of differentiable programming.

00:15:02 ◼   ► We joke this is one of the differentiating features,

00:15:05 ◼   ► which is a terrible pun.

00:15:06 ◼   ► And so in most machine learning frameworks that exist today,

00:15:12 ◼   ► the way this works is you build a graph

00:15:14 ◼   ► and then you go do these graph transformations.

00:15:16 ◼   ► Well, the graph is just a data structure in memory.

00:15:18 ◼   ► You do these transformations and you get a new graph out.

00:15:21 ◼   ► Another way to do it is you do it dynamically.

00:15:23 ◼   ► And if you do it dynamically,

00:15:24 ◼   ► again, it's a very runtime sort of a thing.

00:15:27 ◼   ► The problem with both approaches

00:15:29 ◼   ► because they're runtime things

00:15:30 ◼   ► is that when you make a mistake,

00:15:32 ◼   ► so for example, you forget to transpose a matrix

00:15:36 ◼   ► or something and they do a matrix multiply,

00:15:37 ◼   ► the sizes don't line up, you get a runtime error.

00:15:40 ◼   ► And when you get the runtime error,

00:15:42 ◼   ► it's difficult to reason about where it came from

00:15:43 ◼   ► depending on the exact details of the framework.

00:15:46 ◼   ► And so what Swift for TensorFlow does is it says,

00:15:48 ◼   ► hey, well, what we can do is we can take this idea

00:15:50 ◼   ► of taking a function and calculating its derivative,

00:15:55 ◼   ► build just that piece into the Swift language.

00:15:57 ◼   ► So this is a first-class language feature.

00:16:00 ◼   ► And by doing that, now you have a very generic

00:16:02 ◼   ► language feature that could be used in many domains.

00:16:04 ◼   ► Machine learning is just one of them.

00:16:06 ◼   ► And you can get a much better user experience.

00:16:09 ◼   ► You can get a nice type-directed way of doing this

00:16:12 ◼   ► so that it's extensible.

00:16:14 ◼   ► So you can say, hey, it works on float

00:16:17 ◼   ► and it works on tensor,

00:16:17 ◼   ► but I wanna define a quaternion type.

00:16:20 ◼   ► I can go do that and make my thing also differentiable.

00:16:23 ◼   ► And that's something that doesn't really exist

00:16:26 ◼   ► in modern machine learning frameworks.

00:16:28 ◼   ► What you'll see in them is that you'll see

00:16:29 ◼   ► that tensor is differentiable,

00:16:31 ◼   ► but the normal float type isn't.

00:16:33 ◼   ► Or there's special hacks around certain corner cases,

00:16:36 ◼   ► but it's very difficult to extend these things.

00:16:38 ◼   ► And again, there's many different systems

00:16:41 ◼   ► and this is generalization,

00:16:42 ◼   ► but by building in the language, the goal,

00:16:45 ◼   ► and I think the reality is that you get a much better

00:16:48 ◼   ► better user experience.

00:16:49 ◼   ► So that's one of the features.

00:16:52 ◼   ► Now, what we did is we've designed all these features

00:16:55 ◼   ► to be orthogonal from each other,

00:16:57 ◼   ► and we want to upstream these things.

00:16:59 ◼   ► And so the differentiable programming work

00:17:02 ◼   ► has long been a collaboration with the Swift community.

00:17:04 ◼   ► And I think the team's planning on pushing it

00:17:07 ◼   ► through Swift evolution soon.

00:17:08 ◼   ► And so it's been really exciting.

00:17:10 ◼   ► - So for me, I write regular Swift stuff

00:17:14 ◼   ► in iOS apps and things of that nature.

00:17:16 ◼   ► And the only, I haven't really ever dabbled

00:17:19 ◼   ► with machine learning and listening to you talk,

00:17:21 ◼   ► my own ignorance is becoming ever more evident.

00:17:23 ◼   ► But nevertheless, the only thing that I was aware of

00:17:27 ◼   ► that has kind of bubbled into my Swift universe,

00:17:30 ◼   ► my perspective of Swift, is some of the Python interop.

00:17:33 ◼   ► And that doesn't sound like what you're talking about yet.

00:17:36 ◼   ► Is that fair to say?

00:17:37 ◼   ► - That'd be the next step.

00:17:39 ◼   ► So the cool thing about the differentiable

00:17:41 ◼   ► programming features is that they're very relevant

00:17:43 ◼   ► to the numeric world.

00:17:44 ◼   ► And so there's a couple of different subgroups

00:17:47 ◼   ► in the numeric world that are fairly narrow,

00:17:50 ◼   ► but in the case of machine learning,

00:17:51 ◼   ► they're narrow, but really commercially important.

00:17:55 ◼   ► And so the inspiration for this feature

00:17:57 ◼   ► and the inspiration for the design

00:17:59 ◼   ► actually came from the old days.

00:18:00 ◼   ► It came from Fortran, where back in the old days,

00:18:04 ◼   ► you have a bunch of numeric programming stuff.

00:18:06 ◼   ► And so a lot of the techniques were pioneered

00:18:10 ◼   ► and figured out back in the Fortran days,

00:18:12 ◼   ► and then the world promptly forgot about them.

00:18:14 ◼   ► And so pulling those forward,

00:18:16 ◼   ► making them work in a modern language,

00:18:18 ◼   ► making it work with all the different constraints

00:18:19 ◼   ► that are just very different was a big challenge

00:18:22 ◼   ► and is a big challenge.

00:18:23 ◼   ► And building language features is hard,

00:18:25 ◼   ► but it's been a really interesting project on its own.

00:18:28 ◼   ► - Were you familiar with Fortran

00:18:30 ◼   ► from whatever your past travels had been,

00:18:33 ◼   ► or did somebody say to you,

00:18:33 ◼   ► "Oh, you should look at this weird old,"

00:18:35 ◼   ► well, I mean, I'm sure you were familiar with it,

00:18:36 ◼   ► but you should look at this weird old language.

00:18:38 ◼   ► They had some good ideas,

00:18:38 ◼   ► and you had to dig into it today, in 2020 or 2018 or whatever.

00:18:43 ◼   ► - Well, so these features

00:18:46 ◼   ► weren't part of the Fortran language.

00:18:47 ◼   ► They were part of the Fortran community.

00:18:49 ◼   ► One of the interesting things about the Fortran community

00:18:51 ◼   ► back in the day is that there were a lot of tools

00:18:54 ◼   ► that read in Fortran code, transformed it,

00:18:56 ◼   ► and then wrote it back out.

00:18:57 ◼   ► And so these are effectively source code preprocessors.

00:19:00 ◼   ► And so some of the source code preprocessors

00:19:02 ◼   ► were doing this kind of stuff,

00:19:03 ◼   ► and they're widely used because, again,

00:19:05 ◼   ► the Fortran world is,

00:19:06 ◼   ► there's a lot of numeric people working in that world,

00:19:08 ◼   ► back then, but also today.

00:19:10 ◼   ► I would also like to say I was not the one

00:19:12 ◼   ► that designed all the differentiable programming features

00:19:15 ◼   ► in Swift, it was a team effort,

00:19:16 ◼   ► and I contributed to that,

00:19:18 ◼   ► and I helped with some of the design points,

00:19:19 ◼   ► but I will just be very honest,

00:19:21 ◼   ► that calculus is not my strong point.

00:19:23 ◼   ► (laughing)

00:19:24 ◼   ► I'm very okay with that.

00:19:25 ◼   ► - We are sponsored this week by Indeed.com.

00:19:30 ◼   ► Now, I'll be honest with you,

00:19:31 ◼   ► when they booked this spot a few months ago,

00:19:33 ◼   ► they were gonna run a regular ad,

00:19:34 ◼   ► but as the coronavirus outbreak hit,

00:19:37 ◼   ► it just didn't seem right anymore.

00:19:39 ◼   ► So if your job has been affected by coronavirus,

00:19:41 ◼   ► they've put together a guide to help,

00:19:43 ◼   ► and I'm willing to it from here.

00:19:44 ◼   ► That's all for now.

00:19:45 ◼   ► So stay safe, everyone,

00:19:46 ◼   ► and thanks to Indeed.com for sponsoring our show.

00:19:49 ◼   ► - So we both started to talk about the Python interop.

00:19:56 ◼   ► I'd love to hear a little more about that,

00:19:58 ◼   ► 'cause I've glanced at it,

00:19:59 ◼   ► but I haven't personally had any particular need to use it,

00:20:02 ◼   ► and I've only written, I don't know,

00:20:04 ◼   ► a couple hundred lines of Python in my life.

00:20:05 ◼   ► So I'm vaguely familiar with Python

00:20:07 ◼   ► in kind of some of its tenets,

00:20:10 ◼   ► but this is all kind of outside my typical wheelhouse.

00:20:13 ◼   ► Nevertheless, I find it really fascinating,

00:20:16 ◼   ► the idea of extending Swift or changing Swift

00:20:18 ◼   ► in order to make it interop better.

00:20:20 ◼   ► So how did the Python interop stuff come to be?

00:20:23 ◼   ► I mean, it seems fairly obvious,

00:20:25 ◼   ► but I'd love to hear your perspective of the journey

00:20:27 ◼   ► and what was done to Swift to make that better.

00:20:29 ◼   ► - I think you start from the premise.

00:20:31 ◼   ► So the premise was,

00:20:33 ◼   ► Swift is a good language for hopefully machine learning,

00:20:36 ◼   ► which initially in the project, it was a theory.

00:20:39 ◼   ► It was not a proven fact,

00:20:40 ◼   ► but it was pretty clear

00:20:42 ◼   ► that the entire world revolves around Python.

00:20:45 ◼   ► And so just having something better is not enough.

00:20:48 ◼   ► You need to provide a path

00:20:49 ◼   ► for people to be able to move over.

00:20:51 ◼   ► You need to be able to make it

00:20:52 ◼   ► so you can migrate existing code,

00:20:54 ◼   ► and you kind of have to integrate

00:20:56 ◼   ► with all the, not just the machine learning technologies,

00:20:59 ◼   ► but the huge ecosystem that has been built

00:21:02 ◼   ► around the Python machine learning world.

00:21:04 ◼   ► So this includes all the plotting libraries

00:21:06 ◼   ► and all the analysis and data loading and slicing and dicing.

00:21:09 ◼   ► There's just a huge ecosystem out there.

00:21:11 ◼   ► And so interop-ing with Python

00:21:13 ◼   ► was a pretty clear goal from the beginning.

00:21:16 ◼   ► But then you ask the question of like,

00:21:18 ◼   ► what is the best way to do that?

00:21:20 ◼   ► Now, if you look at Swift,

00:21:21 ◼   ► Swift already has language interoperability support

00:21:24 ◼   ► for C, Objective-C, those kinds of languages, right?

00:21:27 ◼   ► And the way that works is a very expensive,

00:21:31 ◼   ► very complicated integration with the Clang compiler.

00:21:35 ◼   ► And so Apple has invested a tremendous amount of money

00:21:37 ◼   ► in to making sure that all of its frameworks

00:21:40 ◼   ► map over beautifully into Swift.

00:21:42 ◼   ► And all these things can,

00:21:44 ◼   ► there's like a thousand attributes that you can use

00:21:46 ◼   ► to customize how it gets imported

00:21:47 ◼   ► and all that kind of stuff.

00:21:49 ◼   ► And I think that makes sense for the C world,

00:21:52 ◼   ► but doing that for Python seemed very untenable

00:21:55 ◼   ► for a couple of reasons.

00:21:55 ◼   ► One of which is the complexity,

00:21:59 ◼   ► let's just say could not pay for it.

00:22:02 ◼   ► It didn't, it was not that important.

00:22:04 ◼   ► It was, it started, this all started as a research project.

00:22:06 ◼   ► But second of all,

00:22:07 ◼   ► a major difference between Python and the C languages

00:22:10 ◼   ► is that Python is fully dynamically typed.

00:22:13 ◼   ► And so in C languages, you have type signatures,

00:22:17 ◼   ► you have API declarations,

00:22:18 ◼   ► you have all this stuff to tie into,

00:22:20 ◼   ► but in Python, you just don't have that.

00:22:22 ◼   ► And so kind of going through this,

00:22:25 ◼   ► what I realized quickly is that

00:22:26 ◼   ► that dynamic nature of Python was both a huge curse

00:22:31 ◼   ► in terms of working the way that the C importer worked,

00:22:34 ◼   ► but it was also an amazing blessing

00:22:35 ◼   ► because it made everything way simpler.

00:22:38 ◼   ► And so without going to how it works,

00:22:40 ◼   ► I'll explain the outcome.

00:22:42 ◼   ► So right now there's,

00:22:45 ◼   ► you can go open a Jupyter workbook,

00:22:47 ◼   ► which is a online notebook environment.

00:22:49 ◼   ► It's kind of like a playground on the web.

00:22:51 ◼   ► You can go through the Python interoperability tutorial.

00:22:54 ◼   ► And if you do that, what you'll see is you'll see

00:22:56 ◼   ► the syntax looks almost exactly like Python.

00:22:59 ◼   ► So this comes to the,

00:23:01 ◼   ► this builds on the fact that the Python expression syntax

00:23:04 ◼   ► and the C expression syntax, or sorry,

00:23:07 ◼   ► the Python expression syntax

00:23:08 ◼   ► and the Swift expression syntaxes are very similar.

00:23:11 ◼   ► Like you use plus in both languages to add things.

00:23:14 ◼   ► You have dot notation, you have function call notation,

00:23:17 ◼   ► you have parentheses and stuff like that.

00:23:19 ◼   ► It all works roughly the same way.

00:23:22 ◼   ► But in the case of Python, it quote unquote just works.

00:23:26 ◼   ► What you do is you import the Python module

00:23:28 ◼   ► and now you have full access to the entire Python ecosystem

00:23:31 ◼   ► and everything works.

00:23:32 ◼   ► You don't have to do type annotations.

00:23:33 ◼   ► You don't need header files.

00:23:34 ◼   ► You don't need to go change your Python code.

00:23:37 ◼   ► It just works.

00:23:39 ◼   ► And the way that works,

00:23:41 ◼   ► you might wonder about this

00:23:41 ◼   ► because Python is such a dynamic,

00:23:43 ◼   ► such a unique language in its own right.

00:23:46 ◼   ► The way it works is that Swift just links

00:23:48 ◼   ► in the Python interpreter.

00:23:50 ◼   ► And so when you import the Python module,

00:23:52 ◼   ► the Python module and the Swift Python module

00:23:55 ◼   ► just links to the Python interpreter.

00:23:56 ◼   ► And so you're literally just linking the Python interpreter.

00:23:59 ◼   ► So now you get literally everything in Python,

00:24:02 ◼   ► you can now talk to it.

00:24:03 ◼   ► And the way Python works under the covers

00:24:05 ◼   ► is the Python is a relatively, I should back out of that.

00:24:10 ◼   ► I was about to say it's a relatively simple language

00:24:12 ◼   ► and that's probably not quite true,

00:24:14 ◼   ► but it's a language that is built on top of C

00:24:17 ◼   ► and it has a C API for everything.

00:24:19 ◼   ► And so there's a C API to call a function in Python.

00:24:22 ◼   ► There's a C API to do like a dot access, like X.Y,

00:24:26 ◼   ► and there's C APIs for all these things.

00:24:28 ◼   ► And so that Python module is just using Swift's existing

00:24:33 ◼   ► C interop to import all those APIs

00:24:35 ◼   ► and directly call into them.

00:24:37 ◼   ► It's a very beautiful thing.

00:24:38 ◼   ► And that Swift, the Python module in Swift

00:24:42 ◼   ► is only something like 1200 lines of Swift code.

00:24:44 ◼   ► It's pretty simple.

00:24:45 ◼   ► - That's bananas.

00:24:46 ◼   ► I mean, there's a similar language you could do this for.

00:24:48 ◼   ► Could you make Swift interoperate with PHP just for me?

00:24:51 ◼   ► (laughing)

00:24:53 ◼   ► - So let's talk about how this works.

00:24:55 ◼   ► (laughing)

00:24:57 ◼   ► So now the way this works is if you go look

00:25:00 ◼   ► at that Python module, and again, it's just Swift code.

00:25:02 ◼   ► So you can go take a look at it.

00:25:04 ◼   ► What you'll see is you'll see a bunch of like weird

00:25:07 ◼   ► boilerplate stuff that is there to make things work.

00:25:11 ◼   ► But at the end of the day, just calls them the C APIs.

00:25:14 ◼   ► So now how do you provide the feel of Python code in Swift?

00:25:19 ◼   ► Well, Swift has already a fairly hackable syntax.

00:25:22 ◼   ► So you have like plus, you can override plus.

00:25:25 ◼   ► You have the ability to define new operators

00:25:28 ◼   ► and things like that if you want,

00:25:29 ◼   ► but you actually don't really need that

00:25:31 ◼   ► because Python operators and Swift operators

00:25:32 ◼   ► are roughly the same or Swift has a superset.

00:25:35 ◼   ► But there's big problems when you start

00:25:37 ◼   ► to save function calls.

00:25:39 ◼   ► And so as of Swift two and a half years ago,

00:25:43 ◼   ► you couldn't just, you didn't have a notion

00:25:46 ◼   ► of a callable type.

00:25:48 ◼   ► You couldn't say I have a value

00:25:49 ◼   ► and I want the function call operator on that value to work.

00:25:53 ◼   ► That's not a thing.

00:25:54 ◼   ► Function call at the time meant either call function

00:25:58 ◼   ► for call method or initialize a type.

00:26:00 ◼   ► But that was not a user extensible part of the language.

00:26:04 ◼   ► And so when starting to talk through this,

00:26:08 ◼   ► had many conversations with people and said,

00:26:10 ◼   ► okay, well, how do we do Python interoperability?

00:26:13 ◼   ► We talked through, and this included

00:26:15 ◼   ► with the Swift core team at the time.

00:26:17 ◼   ► Like what are the best ways of doing this?

00:26:19 ◼   ► And people generally all agreed

00:26:22 ◼   ► that doing a Python feature was a bad move.

00:26:24 ◼   ► We should not do that.

00:26:26 ◼   ► It turns out there are lots of dynamic languages.

00:26:28 ◼   ► There's Ruby out there, there's PHP, there's JavaScript.

00:26:31 ◼   ► There's lots of interesting dynamic languages out there.

00:26:34 ◼   ► And we didn't wanna have a Python feature.

00:26:36 ◼   ► And so pulling us all back around,

00:26:40 ◼   ► if you go look at that, that is what led to a couple

00:26:43 ◼   ► of very specific features being added to Swift,

00:26:45 ◼   ► including one that just shipped in Swift 5.2,

00:26:48 ◼   ► which went out yesterday as of this recording, I think.

00:26:51 ◼   ► And these features were the dynamic callable

00:26:53 ◼   ► and the dynamic member lookup features.

00:26:56 ◼   ► And what those two features do is it allows any type,

00:26:59 ◼   ► and we use it for Python,

00:27:01 ◼   ► but you can use it in your own Swift code now.

00:27:03 ◼   ► Any type can just overload the call operator

00:27:06 ◼   ► and the member lookup operator.

00:27:08 ◼   ► The member lookup operator is the X.Y kind of syntax,

00:27:12 ◼   ► and turn it into a method call.

00:27:14 ◼   ► And now when you do X.Y on a Python-y thing,

00:27:18 ◼   ► it goes and does that C function that does X.Y for Python

00:27:22 ◼   ► and then returns the result.

00:27:24 ◼   ► And the way this whole system bakes out,

00:27:27 ◼   ► which is really beautiful, is you look at Python

00:27:31 ◼   ► and people say Python has no types, right?

00:27:34 ◼   ► That's a thing that many people say,

00:27:37 ◼   ► and they say Swift has types, right?

00:27:40 ◼   ► And Swift having types and Python not having types

00:27:43 ◼   ► mean they're incompatible, right?

00:27:45 ◼   ► Well, the way I look at it

00:27:46 ◼   ► and the way that I can explain it is saying

00:27:48 ◼   ► Python has one type.

00:27:51 ◼   ► That one type is implicit,

00:27:54 ◼   ► and so you never utter it generally in Python,

00:27:57 ◼   ► but there is a type and Python has a little object model

00:28:00 ◼   ► and has a little data representation,

00:28:02 ◼   ► and it's very beautiful and consistent

00:28:04 ◼   ► in its own way internally,

00:28:05 ◼   ► and I actually have a lot of respect

00:28:07 ◼   ► for the internals of how Python works.

00:28:09 ◼   ► And so when you import that into Swift,

00:28:11 ◼   ► it is the Python object type.

00:28:14 ◼   ► And so there's one type in Swift

00:28:16 ◼   ► that is all the Python stuff,

00:28:18 ◼   ► and it's all completely dynamic within that type.

00:28:21 ◼   ► And so if you have a Python object and you say,

00:28:24 ◼   ► myPythonObject.x, what that does is it fires off

00:28:27 ◼   ► that C API call and it returns a new Python object.

00:28:31 ◼   ► And then you say, parenthesis 42,

00:28:33 ◼   ► and it does a function call on that Python object.

00:28:36 ◼   ► And what this means is you get all the,

00:28:38 ◼   ► you're true to Python through and through,

00:28:41 ◼   ► because if in Python you get some crazy runtime error,

00:28:44 ◼   ► well, hey, you'll get the crazy runtime error here too.

00:28:46 ◼   ► It's the same model, not in your Swift code.

00:28:50 ◼   ► Now, one of the things that I didn't really anticipate,

00:28:52 ◼   ► but it has worked out really well,

00:28:54 ◼   ► and this comes back to this idea of

00:28:57 ◼   ► you get beautiful designs if you build simple things

00:28:59 ◼   ► that compose correctly,

00:29:01 ◼   ► is that in Python you can have a Python array,

00:29:04 ◼   ► you can have Python integers,

00:29:06 ◼   ► you can have Python, all the Python things.

00:29:09 ◼   ► Well, Swift also has integers, it also has arrays.

00:29:12 ◼   ► And so one of the really interesting things

00:29:14 ◼   ► about Python interoperability when you embed into Swift

00:29:16 ◼   ► is you get, I think, perhaps the world's

00:29:19 ◼   ► most beautiful progressive typing system for Python,

00:29:23 ◼   ► where you can say, hey, I have a Python dictionary

00:29:27 ◼   ► of Python strings to Python arrays,

00:29:30 ◼   ► or you can say, I have a Swift dictionary of Python arrays,

00:29:33 ◼   ► or Python strings to Python arrays,

00:29:36 ◼   ► or you can say, I have a Swift dictionary

00:29:38 ◼   ► of Swift strings to Python arrays,

00:29:41 ◼   ► and you can type statically or dynamically

00:29:44 ◼   ► as much as you want at any level,

00:29:45 ◼   ► because it all interoperates correctly

00:29:47 ◼   ► in the same type system.

00:29:48 ◼   ► It's a really, really interesting and very beautiful thing.

00:29:52 ◼   ► And the fact that it just works kind of blows people's minds.

00:29:55 ◼   ► Now, coming back to PHP, well,

00:29:57 ◼   ► so that Python module in Swift is 1200 lines of code.

00:30:01 ◼   ► PHP has its own interpreter.

00:30:03 ◼   ► You could build exactly the same kind of a thing

00:30:05 ◼   ► talking to PHP, that's by design.

00:30:07 ◼   ► That's one of the nice things

00:30:08 ◼   ► about the language features being very orthogonal.

00:30:11 ◼   ► And then the question is, how good does it feel?

00:30:15 ◼   ► I'm not a PHP expert, thankfully.

00:30:16 ◼   ► (laughing)

00:30:19 ◼   ► But if the basic grammar structure of PHP

00:30:22 ◼   ► is similar to the basic grammar structure of Swift,

00:30:24 ◼   ► it will work out really nicely.

00:30:25 ◼   ► If you were talking to Objective-C or something like that,

00:30:30 ◼   ► it would be somewhat less beautiful.

00:30:34 ◼   ► But the system's set up so that you could talk

00:30:36 ◼   ► to small talky languages, you could talk

00:30:38 ◼   ► to many different kinds of things,

00:30:41 ◼   ► and I think it's pretty cool.

00:30:43 ◼   ► - See, everyone, you heard it here first.

00:30:45 ◼   ► I think most people are gonna look at this proposal

00:30:47 ◼   ► as like, you have a sewage treatment plant over here,

00:30:50 ◼   ► and you have a nice, fresh, clean ocean over here.

00:30:53 ◼   ► Let's build a canal to connect the two.

00:30:55 ◼   ► (laughing)

00:30:56 ◼   ► - I mean, if you go back and you look at,

00:30:58 ◼   ► so the Python interoperability,

00:31:00 ◼   ► all the language features are in Swift now.

00:31:02 ◼   ► Like, there's no, this is a done deal.

00:31:04 ◼   ► If you go back and you look at the, gosh, when was that?

00:31:07 ◼   ► That must have been December 2017-ish, something like that.

00:31:11 ◼   ► That's when we were talking about, on the Swift forums,

00:31:14 ◼   ► adding dynamic member lookup to Swift.

00:31:17 ◼   ► And the threads, there are many mega threads on this,

00:31:19 ◼   ► and it was hugely controversial,

00:31:22 ◼   ► and the arguments against it at the time,

00:31:24 ◼   ► or one of the major arguments against it at the time,

00:31:26 ◼   ► was like, hey, if you allow people to overload operator dot,

00:31:29 ◼   ► which is kind of what this is, they're gonna misuse it.

00:31:32 ◼   ► And then nobody's gonna be able to reason about anything,

00:31:34 ◼   ► because everybody will use it for all the things,

00:31:37 ◼   ► and they'll horribly pollute all the codebase everywhere,

00:31:40 ◼   ► and you're not gonna be able to reason about anything.

00:31:42 ◼   ► - I mean, can't you make that argument

00:31:43 ◼   ► about almost every Swift feature?

00:31:45 ◼   ► - Bingo, that's exactly right.

00:31:46 ◼   ► And so you can make that argument

00:31:48 ◼   ► about any feature in any language.

00:31:51 ◼   ► You can misuse anything.

00:31:53 ◼   ► And if you just use integers for everything,

00:31:56 ◼   ► well, that's not gonna be great for your numeric code either,

00:31:59 ◼   ► but you could do it, theoretically.

00:32:01 ◼   ► And so I think that the feature and the argument

00:32:05 ◼   ► has stood the test of time,

00:32:06 ◼   ► where we've had it for, I think, a couple of years now.

00:32:09 ◼   ► People have done really interesting things.

00:32:11 ◼   ► There's definitely JavaScript interoperability things

00:32:14 ◼   ► for marching through JSON files that use this,

00:32:17 ◼   ► and it's enabled some really beautiful and expressive APIs,

00:32:20 ◼   ► but I haven't seen people going overboard

00:32:23 ◼   ► and using it for everything.

00:32:25 ◼   ► It's kind of the same argument as, oh my god,

00:32:26 ◼   ► because you allow emojis in your identifiers,

00:32:28 ◼   ► everything's gonna be pile of poop, right?

00:32:31 ◼   ► - Oh, so you've been looking at my code.

00:32:33 ◼   ► - When you make, this is kind of like the,

00:32:36 ◼   ► I don't know, the destiny of any community built up

00:32:39 ◼   ► or in a language with a particular set of features, right?

00:32:40 ◼   ► So Swift was born as a language with,

00:32:43 ◼   ► that preferred to have errors,

00:32:48 ◼   ► call it a compile time rather than runtime,

00:32:50 ◼   ► and it therefore attracted a community of people

00:32:53 ◼   ► who value that as an attribute in their language.

00:32:57 ◼   ► So you come in a couple years later and say,

00:32:59 ◼   ► you know what, what about, there's some cases

00:33:01 ◼   ► where you won't tell until you make the call

00:33:02 ◼   ► whether there's an error, and they're like, wait a second,

00:33:04 ◼   ► that's not what I signed up for.

00:33:05 ◼   ► I came here for a language where you couldn't do that.

00:33:09 ◼   ► When I have something dot something

00:33:10 ◼   ► and the compiler says it's good, it's good, right?

00:33:13 ◼   ► If I wanted the other thing, I would go to Objective-C

00:33:15 ◼   ► and be sending messages to nil.

00:33:16 ◼   ► I'm here in Swift, I'm a fan of Swift,

00:33:18 ◼   ► I'm on the Swift forums, and so you kind of,

00:33:21 ◼   ► coming down from the mountain and saying,

00:33:23 ◼   ► but what if, dynamic callable, but what if,

00:33:25 ◼   ► integration with languages that aren't like that?

00:33:28 ◼   ► And it was like, well, interoperability is fine,

00:33:30 ◼   ► but don't pollute my language with, you know,

00:33:33 ◼   ► now when I make a call on something or do a member lookup,

00:33:36 ◼   ► I don't even know if that's gonna work

00:33:38 ◼   ► until it hits that line of code?

00:33:39 ◼   ► I understand why people found that heretical,

00:33:42 ◼   ► but I think that same sort of pushback is why thus far

00:33:46 ◼   ► it doesn't feel like it's been a problem,

00:33:48 ◼   ► because if you are attracted to Swift

00:33:51 ◼   ► and are a super fan of Swift,

00:33:52 ◼   ► you're not gonna do that willy-nilly,

00:33:53 ◼   ► you're not gonna say, hey, here's this cool library

00:33:55 ◼   ► for doing this thing in Swift, and by the way,

00:33:57 ◼   ► everything's dynamic, and dot means nothing,

00:34:00 ◼   ► and you can write any method after the dot

00:34:01 ◼   ► and I'll dynamically create it on the fly for you.

00:34:03 ◼   ► Like, people don't do that, or if they did that,

00:34:05 ◼   ► their library wouldn't become popular, so it's kind of--

00:34:07 ◼   ► - Yeah, nobody would use it.

00:34:08 ◼   ► - Yeah, so like, it's, people aren't dying to do it,

00:34:11 ◼   ► and it has never been the culture of Swift,

00:34:13 ◼   ► so even though that feature exists,

00:34:15 ◼   ► I don't think it's even an attractive nuisance

00:34:16 ◼   ► at this point.

00:34:18 ◼   ► - Yeah, I mean, another really interesting to me objection

00:34:22 ◼   ► at the time was people would say things along the lines

00:34:25 ◼   ► of interoperability with Python makes sense

00:34:28 ◼   ► for the machine learning community, but if you do this,

00:34:30 ◼   ► then people will just leave the code in Python,

00:34:32 ◼   ► and they won't ever move it to a beautiful Swift API,

00:34:35 ◼   ► because it will look like Python,

00:34:37 ◼   ► and that's not a Swiftie design, right?

00:34:40 ◼   ► And this is true, right?

00:34:43 ◼   ► I mean, the Python integrated,

00:34:46 ◼   ► if you use NumPy, which is a very popular Python library

00:34:50 ◼   ► for numerical programming, and Swift,

00:34:52 ◼   ► it does not look like a natural Swift API.

00:34:55 ◼   ► The naming conventions and the keyword arguments

00:34:57 ◼   ► get used differently and all that kind of stuff.

00:35:00 ◼   ► - Sounds familiar.

00:35:01 ◼   ► - Yeah, so, exactly, this is also like Objective-C, right?

00:35:05 ◼   ► Where, at least if you don't do all the work

00:35:07 ◼   ► to annotate your APIs, you get something

00:35:09 ◼   ► that doesn't look at all like Swift,

00:35:11 ◼   ► and what I said at the time, which I still believe

00:35:16 ◼   ► is two things, one is, hey, well, interoperability

00:35:18 ◼   ► is the first thing that gives you the ability

00:35:20 ◼   ► to define wrappers, and so if you wanna use

00:35:22 ◼   ► some yucky external code, and it's yucky

00:35:27 ◼   ► for whatever reason, being able to wrap it up

00:35:29 ◼   ► without having to drop two different completely

00:35:33 ◼   ► foreign universes is really useful,

00:35:34 ◼   ► because then you can define a Swift API

00:35:36 ◼   ► that wraps the underlying thing.

00:35:37 ◼   ► The other thing is that, because it doesn't feel natural,

00:35:41 ◼   ► people will want to do that work to create those wrappers,

00:35:45 ◼   ► but when you start doing that, you start to realize,

00:35:47 ◼   ► well, actually, that Python API I'm calling into

00:35:49 ◼   ► is just a Python wrapper on top of C anyways,

00:35:53 ◼   ► in many cases, and so, instead of wrapping the Python,

00:35:57 ◼   ► maybe I should wrap the C, and again,

00:35:59 ◼   ► what you allow people to do is, over time,

00:36:01 ◼   ► the community can go build amazing new plotting libraries

00:36:04 ◼   ► and data analysis libraries and things like this,

00:36:08 ◼   ► but you just allow each individual person

00:36:11 ◼   ► to make decisions that make sense to them,

00:36:14 ◼   ► and you're never blocked, it's very pragmatic.

00:36:16 ◼   ► You can always get stuff done, but then,

00:36:18 ◼   ► if you have lots of spare time, you wanna go design

00:36:20 ◼   ► the world's best plotting library, you can do that,

00:36:22 ◼   ► and so now, that feeling unnatural thing

00:36:26 ◼   ► actually kind of helps with that, I think.

00:36:29 ◼   ► - This is an interesting test case,

00:36:31 ◼   ► because in the case of Objective-C and Swift,

00:36:33 ◼   ► a lot of the similar problems, but the big difference is,

00:36:37 ◼   ► there was a sort of command and control structure

00:36:40 ◼   ► that basically dictated Swift as a thing that we're doing,

00:36:43 ◼   ► and until you're otherwise, this is the future,

00:36:45 ◼   ► so get on board the train, and pretty soon,

00:36:48 ◼   ► you basically have no choice.

00:36:50 ◼   ► Like, there's a dictator involved,

00:36:52 ◼   ► and that dictator being Apple.

00:36:53 ◼   ► Like, if you're gonna develop for Apple platforms,

00:36:55 ◼   ► it's gonna be in Swift eventually, so get used to it

00:36:58 ◼   ► over the course of many years, and you know,

00:37:00 ◼   ► in the world of Python and machine learning,

00:37:02 ◼   ► I imagine there is no dictatorial force like that,

00:37:04 ◼   ► so even though the same thing could happen,

00:37:06 ◼   ► it's certainly not gonna happen with the same speed,

00:37:09 ◼   ► but all the same tools apply, all the same techniques,

00:37:12 ◼   ► all the same sort of what you were saying about,

00:37:13 ◼   ► but it has to happen at the rate of cats being herded,

00:37:17 ◼   ► and not at the rate of the dictator saying,

00:37:21 ◼   ► this is gonna happen over the course of the next few years

00:37:23 ◼   ► unless something goes terribly wrong,

00:37:24 ◼   ► so get on that train, so it'll be interesting to see how,

00:37:28 ◼   ► if that goes faster or slower,

00:37:30 ◼   ► or if it never actually happens,

00:37:32 ◼   ► or if Swift is just another player

00:37:33 ◼   ► in a big soup of languages, but how do you feel

00:37:36 ◼   ► about not being able to just mandate it

00:37:38 ◼   ► as was possible in the Apple days?

00:37:41 ◼   ► - Well, so we could talk about Swift for TensorFlow

00:37:43 ◼   ► more in a second, but let me push back on you, John,

00:37:45 ◼   ► because you're right that Apple could have done that,

00:37:48 ◼   ► but they didn't, so until Swift UI,

00:37:52 ◼   ► there's never been a thing that you could only do in Swift.

00:37:55 ◼   ► - Yeah, I know, but that's a multi-year plan.

00:37:57 ◼   ► - Well, but even with Swift UI,

00:37:59 ◼   ► you can still build UIs without using it,

00:38:01 ◼   ► and lots of people still use UIKit,

00:38:03 ◼   ► so what I'm saying is, for years, Apple did not do that.

00:38:08 ◼   ► There was no arm twisting, thou shalt use Swift.

00:38:11 ◼   ► It was always a--

00:38:13 ◼   ► - Well, so there's not arm twisting,

00:38:14 ◼   ► there's a little bit of arm grabbing,

00:38:16 ◼   ► like at a certain point on your second year at WWDC,

00:38:19 ◼   ► and all the slides are in Swift,

00:38:21 ◼   ► that's the type of thing where it's like,

00:38:23 ◼   ► they control the platform, and they're clearly saying,

00:38:25 ◼   ► "Oh, everything's available in both languages."

00:38:27 ◼   ► The year Swift was introduced,

00:38:29 ◼   ► every slide had both Objective-C and Swift,

00:38:31 ◼   ► I'm presumably dictated from on high,

00:38:32 ◼   ► and for the years that followed,

00:38:34 ◼   ► Objective-C slowly disappeared

00:38:36 ◼   ► from all the slides at WWDC,

00:38:38 ◼   ► and Swift became the only thing,

00:38:39 ◼   ► and then pretty soon the default was Swift

00:38:41 ◼   ► for a new project in Xcode,

00:38:42 ◼   ► and that's what I'm talking about,

00:38:44 ◼   ► like that there is a large--

00:38:45 ◼   ► - Yeah, yeah, I agree with you there.

00:38:46 ◼   ► - A large, not so invisible hand

00:38:49 ◼   ► pushing in one clear direction,

00:38:51 ◼   ► and it wasn't like, yeah, they didn't come down

00:38:53 ◼   ► and just say, "We're changing this overnight,"

00:38:55 ◼   ► 'cause you couldn't, you had to make sure,

00:38:56 ◼   ► "Hey, is the Swift thing gonna work out?

00:38:57 ◼   ► "Do people like it?"

00:38:58 ◼   ► There's lots of unknowns,

00:38:59 ◼   ► but the direction has always been clear.

00:39:00 ◼   ► That's why I said, in the absence of anything else,

00:39:03 ◼   ► this is gonna happen, whether you like it or not,

00:39:06 ◼   ► because there are still people like,

00:39:07 ◼   ► "Oh, I liked Objective-C better," and whatever,

00:39:09 ◼   ► but whether you like it or not,

00:39:11 ◼   ► if the majority of the community

00:39:12 ◼   ► does not scream bloody murder,

00:39:14 ◼   ► and Swift works out fine, which it seems to,

00:39:17 ◼   ► we're going to get to the point very quickly

00:39:18 ◼   ► when, a couple years down the line,

00:39:20 ◼   ► "Oh, here's a new API that you can't use for Objective-C,"

00:39:23 ◼   ► and by the time that happens, it's almost a non-event.

00:39:26 ◼   ► I mean, some people grumbled about Swift UI,

00:39:28 ◼   ► but Swift had been so clearly the message from Apple

00:39:32 ◼   ► for so many years that I think,

00:39:35 ◼   ► by this point, if you weren't on board with that change,

00:39:37 ◼   ► like, I don't know if you've been beaten down

00:39:39 ◼   ► by the overwhelming tide of Swift on Apple platforms,

00:39:42 ◼   ► or you just got used to it, or learned to love it,

00:39:45 ◼   ► or left to go to a different platform, right?

00:39:47 ◼   ► Whereas, like I said, in machine learning,

00:39:49 ◼   ► you can have enthusiasm for Swift,

00:39:50 ◼   ► but it's really up to the individual people, as you said,

00:39:52 ◼   ► to say, "Is this what I wanna do?

00:39:54 ◼   ► "Do I wanna make a wrapper for this?

00:39:55 ◼   ► "Do I mind that there's Python in between?

00:39:57 ◼   ► "Do I find the Swift thing benefits me in any way,

00:39:59 ◼   ► "or do I just wanna do it in Python,

00:40:01 ◼   ► "or do I like the Swift so much

00:40:02 ◼   ► "that I'm gonna disintermediate the Python

00:40:04 ◼   ► "and go right from Swift to C for my new library?"

00:40:07 ◼   ► And that's up to the individual, and I mean,

00:40:09 ◼   ► I suppose as a cultural influence as well,

00:40:11 ◼   ► I'm not involved in the machine learning community,

00:40:13 ◼   ► but if they have big conferences and the same type of thing,

00:40:15 ◼   ► here is, you know, let me show a demo

00:40:17 ◼   ► of this cool thing I did in machine learning,

00:40:18 ◼   ► and it just so happens that all of their demo

00:40:20 ◼   ► and slides and code is in Swift, that sends a message,

00:40:23 ◼   ► especially if it's something cool,

00:40:25 ◼   ► but it's sort of on a case-by-case basis

00:40:27 ◼   ► up to the individuals.

00:40:28 ◼   ► - Well, so I think that, I mean,

00:40:30 ◼   ► to agree with you and then disagree with you,

00:40:34 ◼   ► I think you're right that the machine learning community

00:40:36 ◼   ► is definitely not top-down controlled,

00:40:38 ◼   ► and even if somebody were to do that,

00:40:40 ◼   ► you would have a thousand people

00:40:43 ◼   ► all doing different things anyways,

00:40:45 ◼   ► because there isn't a lot of consistency in that universe,

00:40:47 ◼   ► but the same thing is true about server development

00:40:49 ◼   ► and many other segments where Swift is,

00:40:52 ◼   ► what I would say, organically growing slowly over time,

00:40:55 ◼   ► and so I don't think that's unique

00:40:56 ◼   ► to the machine learning community.

00:40:57 ◼   ► - Yeah, that's true.

00:40:58 ◼   ► We talked about Swift on the server recently,

00:41:00 ◼   ► and it is very similar in that Apple's not super interested

00:41:02 ◼   ► in, they don't have a server platform

00:41:04 ◼   ► that they're pressing on everybody,

00:41:05 ◼   ► so they're not a force there,

00:41:07 ◼   ► and then it's really more like machine learning

00:41:09 ◼   ► where it's up to individuals.

00:41:10 ◼   ► - And even if they did, if they came out and said,

00:41:12 ◼   ► "We think that this is the right thing to do for server,"

00:41:14 ◼   ► everybody'd say like, "Okay, cool, Apple, whatever.

00:41:16 ◼   ► "I'm using my Django thing," or whatever,

00:41:20 ◼   ► and there's not that, to your point,

00:41:22 ◼   ► there's not that top-down, single-leader type of thing,

00:41:26 ◼   ► but the thing I wanna push back on you again

00:41:29 ◼   ► is who fundamentally has the control there?

00:41:31 ◼   ► Who has the power?

00:41:33 ◼   ► So is it Apple, or is it the community?

00:41:36 ◼   ► Because Apple has pushed technologies, as you know,

00:41:39 ◼   ► in the past that haven't worked out,

00:41:41 ◼   ► and so if Apple started pushing it that first year,

00:41:45 ◼   ► and it was kind of a soft push,

00:41:46 ◼   ► like that first year in particular was a,

00:41:48 ◼   ► "Hey, we have a thing.

00:41:50 ◼   ► "We hope you like it.

00:41:50 ◼   ► "We think it's great, but let us know what you think,"

00:41:54 ◼   ► and if the community barfed all over it

00:41:57 ◼   ► and said, "This is terrible.

00:41:58 ◼   ► "We don't want anything to do with it,"

00:41:59 ◼   ► well, Apple would have, course correction didn't change,

00:42:01 ◼   ► and so I think the community and the community reaction

00:42:05 ◼   ► and feedback and things like that also have a huge impact

00:42:08 ◼   ► on Apple's decision-making process as well.

00:42:12 ◼   ► - Yeah, I mean, there's definitely a give and take there,

00:42:14 ◼   ► but that's why I was often the qualifier

00:42:16 ◼   ► is assuming things worked out,

00:42:17 ◼   ► assuming there wasn't just open revolt

00:42:19 ◼   ► in the streets or whatever,

00:42:21 ◼   ► but there was definitely pushback,

00:42:23 ◼   ► but there was enough promise that Apple said,

00:42:25 ◼   ► "We're gonna power through the pushback,"

00:42:27 ◼   ► because certainly when Swift was introduced

00:42:29 ◼   ► and for years afterwards, there were lots of complaints

00:42:31 ◼   ► from people who were very experienced with Objective-C

00:42:33 ◼   ► about how Swift was inadequate for their purposes

00:42:35 ◼   ► and Objective-C was better

00:42:37 ◼   ► and didn't understand why we were making this change,

00:42:39 ◼   ► but there was enough promise and enough people who liked it

00:42:42 ◼   ► that Apple was able to make the decision to say,

00:42:45 ◼   ► "We're gonna keep going.

00:42:46 ◼   ► "This looks like, I know there's complaints,

00:42:48 ◼   ► "but we're gonna keep going,"

00:42:50 ◼   ► and so to just power through that.

00:42:52 ◼   ► - Yeah, yeah, I'm sure that none of the people on this call

00:42:55 ◼   ► would have those complaints early on.

00:42:57 ◼   ► - No, no, early on. (laughing)

00:42:59 ◼   ► We had complaints like that three shows ago.

00:43:03 ◼   ► We'll get to that in a minute.

00:43:04 ◼   ► All right, so I think we should--

00:43:05 ◼   ► - Yeah, one of the things I love to tell people

00:43:07 ◼   ► is that pick any technology that I know,

00:43:10 ◼   ► and I can both love it and hate it at the same time.

00:43:12 ◼   ► - Well, that's what comes from using it, yeah.

00:43:15 ◼   ► - Yeah, exactly, and if you can't,

00:43:17 ◼   ► then you're just being religious or something.

00:43:20 ◼   ► - Or you haven't used it long enough yet.

00:43:21 ◼   ► - Yeah, and so, sorry, so those are the language features

00:43:24 ◼   ► in the Swift for TensorFlow project.

00:43:26 ◼   ► There's other pieces as well,

00:43:27 ◼   ► and so one of those is the API,

00:43:29 ◼   ► and so Swift for TensorFlow has a big API,

00:43:31 ◼   ► and that API then wraps the TensorFlow APIs,

00:43:34 ◼   ► and saying it wraps the TensorFlow APIs

00:43:37 ◼   ► doesn't do justice to how cool it is.

00:43:39 ◼   ► I don't, it probably doesn't make sense

00:43:41 ◼   ► to deep dive on all this stuff, given the audience,

00:43:43 ◼   ► but it's really cool stuff.

00:43:47 ◼   ► It uses, again, that really powerful

00:43:49 ◼   ► and very efficient nature that Swift gives API developers.

00:43:53 ◼   ► The next step up then is the community aspect of it,

00:43:56 ◼   ► and one of the things I'm very happy about

00:43:57 ◼   ► with the Swift for TensorFlow community

00:43:59 ◼   ► is that it has attracted a lot of really interesting

00:44:02 ◼   ► and really smart people that have contributed a huge amount,

00:44:05 ◼   ► and one of the things that we did at Google

00:44:07 ◼   ► and is continuous to this day is

00:44:09 ◼   ► there is a roughly weekly public video chat

00:44:12 ◼   ► that you can dial into and talk to the team,

00:44:15 ◼   ► and all the episodes are recorded,

00:44:18 ◼   ► and you can go watch all the technical discussions

00:44:22 ◼   ► about differentiable programming

00:44:24 ◼   ► or APIs for reinforcement learning

00:44:28 ◼   ► and all these different things, and they're very public,

00:44:31 ◼   ► and this has been a really great thing

00:44:33 ◼   ► for engaging a community and helping build

00:44:35 ◼   ► and helping learn, and particularly

00:44:37 ◼   ► in the machine learning community,

00:44:38 ◼   ► where there's so many different people

00:44:40 ◼   ► and perspectives and talents,

00:44:42 ◼   ► and there isn't obviously one right way to do it

00:44:44 ◼   ► because it's not a mature field.

00:44:47 ◼   ► I think that was very, very helpful.

00:44:49 ◼   ► - You had those video calls at Apple too, though, right?

00:44:52 ◼   ► - Are you kidding me?

00:44:53 ◼   ► - I am kidding.

00:44:54 ◼   ► Just imagine, though, this is the difference

00:44:56 ◼   ► between the two companies.

00:44:57 ◼   ► You're talking about a technology

00:44:59 ◼   ► and a product that you're working on developing

00:45:02 ◼   ► and that you're having routine interactions

00:45:04 ◼   ► with official members of Google and members of the public

00:45:08 ◼   ► talking about the technology in an open forum.

00:45:10 ◼   ► It's an alien concept to the Apple way of doing things.

00:45:13 ◼   ► - Yeah, and again, I think that Apple's moving a lot.

00:45:17 ◼   ► In particular, in the Swift world,

00:45:20 ◼   ► I think it's being quite progressive.

00:45:23 ◼   ► - They got a blog.

00:45:24 ◼   ► (laughing)

00:45:26 ◼   ► - It's just like a light and day difference

00:45:28 ◼   ► in terms of the significance placed on such things.

00:45:31 ◼   ► Then it's just a different philosophy.

00:45:33 ◼   ► I don't think that either is right or wrong,

00:45:35 ◼   ► but it's quite different.

00:45:36 ◼   ► So the project continues today.

00:45:38 ◼   ► It's an exciting project.

00:45:40 ◼   ► Just announced their new release a week ago,

00:45:44 ◼   ► two weeks ago, something like that.

00:45:45 ◼   ► And so they're making really great strides

00:45:47 ◼   ► integrating new things.

00:45:48 ◼   ► One of the challenges there is that the runtime implementation

00:45:51 ◼   ► that they're building on top of the classic TensorFlow runtime

00:45:53 ◼   ► is not perfectly suited for their task.

00:45:56 ◼   ► And so they're moving to new technology stacks

00:45:59 ◼   ► and doing cool stuff.

00:46:01 ◼   ► Anyways, if you're interested in that space,

00:46:03 ◼   ► it's a very vibrant and very cool project.

00:46:06 ◼   ► And it's still, I would say,

00:46:07 ◼   ► a little bit ahead of its time,

00:46:09 ◼   ► but the technology pieces are falling in place now

00:46:12 ◼   ► and it's gonna be a very exciting 2020.

00:46:14 ◼   ► - So it sounds like you're super into this stuff,

00:46:16 ◼   ► but then you left to go to Sci-5.

00:46:19 ◼   ► Explain that.

00:46:20 ◼   ► - That's only the second project I did at Google.

00:46:24 ◼   ► So there's another project called MLIR.

00:46:27 ◼   ► Have you heard anything about that?

00:46:28 ◼   ► - I have.

00:46:30 ◼   ► - Do you wanna talk about that or is that too geeky

00:46:34 ◼   ► even for John Syracuso?

00:46:35 ◼   ► (laughing)

00:46:36 ◼   ► - I remember you being super excited about CIL

00:46:39 ◼   ► back in the day.

00:46:40 ◼   ► And you just love intermediary language.

00:46:42 ◼   ► This is what you love.

00:46:43 ◼   ► - So here's the deal with MLIR

00:46:45 ◼   ► without diving too deep into it.

00:46:48 ◼   ► If you work on compilers for too long,

00:46:50 ◼   ► you start, just like if you work on anything for too long,

00:46:54 ◼   ► you start pattern recognizing across different systems,

00:46:57 ◼   ► you start realizing they're all the same.

00:46:58 ◼   ► Actually, what you realize is they're 50% the same

00:47:01 ◼   ► and 50% different.

00:47:04 ◼   ► But because you're building a new thing,

00:47:05 ◼   ► you build a new thing from scratch,

00:47:07 ◼   ► and the 50% that's the same never gets factored

00:47:09 ◼   ► across the rest of the universe.

00:47:11 ◼   ► And so that's what compilers are today.

00:47:13 ◼   ► And so if you look at LVM, for example,

00:47:16 ◼   ► it has what's called an intermeter representation.

00:47:19 ◼   ► It's the data structure that the entire LVM universe

00:47:22 ◼   ► works on as a ton of infrastructure

00:47:24 ◼   ► that's been built up across now.

00:47:25 ◼   ► LVM from 20 years old this year, incidentally,

00:47:28 ◼   ► which is kind of scary.

00:47:30 ◼   ► But, and so LVM has a bunch of this stuff.

00:47:33 ◼   ► Then you go look at Swift.

00:47:34 ◼   ► Swift has CIL.

00:47:36 ◼   ► CIL is it's compiler representation

00:47:38 ◼   ► for doing high level language optimizations

00:47:41 ◼   ► and doing arc optimizations and that kind of stuff.

00:47:44 ◼   ► And that representation has to reinvent

00:47:46 ◼   ► a huge amount of basic compiler stuff.

00:47:49 ◼   ► And you go look at machine learning compilers.

00:47:50 ◼   ► You go look at TensorFlow.

00:47:51 ◼   ► You go look at all these different domain specific worlds.

00:47:55 ◼   ► You look at Julia and Rust

00:47:58 ◼   ► and all these different compilers are doing the same thing

00:48:00 ◼   ► over and over and over and over and over again.

00:48:02 ◼   ► And so what MLIR does is ML,

00:48:04 ◼   ► the ML stands for multi-layer.

00:48:07 ◼   ► It's also designed to be reinterpreted in many ways.

00:48:09 ◼   ► If the acronym ultimately fails itself,

00:48:11 ◼   ► we can say it's Moore's law or mid-level

00:48:16 ◼   ► or machine learning or whatever the cool thing is

00:48:19 ◼   ► of the day, Bitcoin.

00:48:20 ◼   ► Wait, that doesn't work.

00:48:21 ◼   ► And so what MLIR does, it says, okay, cool.

00:48:24 ◼   ► Let's make it so that instead of building

00:48:26 ◼   ► an instance of a compiler,

00:48:27 ◼   ► we build a compiler construction toolkit.

00:48:30 ◼   ► The way MLIR works is you define

00:48:32 ◼   ► in a declarative specification what your IR is,

00:48:36 ◼   ► what that intermediate representation is,

00:48:38 ◼   ► what the type system is, what the instructions are

00:48:40 ◼   ► that go into it, things like that.

00:48:41 ◼   ► And then you get a tremendous amount of infrastructure

00:48:43 ◼   ► for free for doing things like testing, location tracking.

00:48:47 ◼   ► So you get debugging optimized code,

00:48:48 ◼   ► something LVM has never really been great at.

00:48:51 ◼   ► You get a ton of the things, multi-threaded compilation,

00:48:54 ◼   ► like all these things that are actually hard

00:48:58 ◼   ► and you have to design in from the beginning

00:49:00 ◼   ► to make them really great.

00:49:00 ◼   ► And most people don't think about that.

00:49:03 ◼   ► And so when you're building a new compiler,

00:49:04 ◼   ► typically just like you're building a system,

00:49:07 ◼   ► you're racing the market to get your thing to work.

00:49:09 ◼   ► You're not investing in that core infrastructure.

00:49:11 ◼   ► And so MLIR allows you to do that.

00:49:14 ◼   ► And we built and started this at Google.

00:49:18 ◼   ► It's now an open source project.

00:49:20 ◼   ► It's contributed back to LVM.

00:49:21 ◼   ► So it's now an official LVM project.

00:49:24 ◼   ► And one of the cool things about MLIR

00:49:25 ◼   ► is it's only 18 months old-ish at this point

00:49:29 ◼   ► and 18 months-ish from the first white paper.

00:49:32 ◼   ► And it's already being pervasively adopted

00:49:35 ◼   ► across the industry by all the big companies

00:49:38 ◼   ► for lots of different things.

00:49:39 ◼   ► And it's been just an incredible growth.

00:49:42 ◼   ► And it's just really cool to see that happen.

00:49:45 ◼   ► Interesting you described it as a compiler construction toolkit.

00:49:48 ◼   ► Wasn't that basically the pitch for LVM back in the day?

00:49:50 ◼   ► In a very different world, but similar idea,

00:49:53 ◼   ► a bunch of libraries you can use to build a compiler.

00:49:55 ◼   ► Yeah, so the difference with LVM is, I think, fairly big.

00:50:01 ◼   ► So I love LVM.

00:50:02 ◼   ► Please don't misread any bad things I say about LVM

00:50:05 ◼   ► as me disliking LVM.

00:50:08 ◼   ► LVM is really-- or the LVM IR, what people think about when

00:50:12 ◼   ► they talk about the core LVM.

00:50:15 ◼   ► LVM is a really good way to talk to CPUs.

00:50:17 ◼   ► Or they're talking-- the basic model of LVM

00:50:20 ◼   ► is C with vectors, roughly.

00:50:22 ◼   ► And so if you have a problem that looks like C with vectors,

00:50:25 ◼   ► LVM is a really good solution, because you

00:50:27 ◼   ► can use a large number of code generators

00:50:30 ◼   ► for all the different popular processors

00:50:33 ◼   ► and things like this.

00:50:34 ◼   ► And you could build really cool high-leverage things

00:50:36 ◼   ► on top of that.

00:50:38 ◼   ► LVM has not been very successful when you talk to accelerators.

00:50:41 ◼   ► LVM has also been completely useless for high-level language

00:50:44 ◼   ► things.

00:50:46 ◼   ► It was never designed to do that.

00:50:48 ◼   ► And so the difference between MLIR and LVM

00:50:51 ◼   ► is that MLIR is designed to solve all the world's problems.

00:50:54 ◼   ► And in fact, LVM is an instance within MLIR.

00:50:58 ◼   ► MLIR also models LVM as well as many other things.

00:51:01 ◼   ► Are you holding out one more letter?

00:51:03 ◼   ► So you've got LVM-- what was that, low-level virtual

00:51:05 ◼   ► machine or whatever-- ML, which is like mid-level.

00:51:10 ◼   ► And I guess the HL is like languages like Swift?

00:51:14 ◼   ► Or do we have room for one more letter in there?

00:51:16 ◼   ► I don't know.

00:51:17 ◼   ► I'm pretty good at four-letter technologies.

00:51:19 ◼   ► [LAUGHTER]

00:51:20 ◼   ► [MUSIC PLAYING]

00:51:22 ◼   ► We are sponsored this week by Jamf Now.

00:51:25 ◼   ► Jamf Now makes it easy to set up, manage, and secure

00:51:28 ◼   ► your Apple devices.

00:51:29 ◼   ► Jamf Now is mobile device management security

00:51:32 ◼   ► that helps you sleep better at night.

00:51:34 ◼   ► From enforcing passcodes and encryption

00:51:36 ◼   ► to remotely locking or wiping a device,

00:51:38 ◼   ► Jamf Now ensures you have the MDM security settings you need

00:51:42 ◼   ► so your Apple devices and the information on them

00:51:44 ◼   ► is untouchable, just the way you want it.

00:51:46 ◼   ► With Jamf Now, you can set passcode requirements,

00:51:49 ◼   ► place a device in lost mode, even remote lock or wipe

00:51:52 ◼   ► a device as needed from anywhere.

00:51:55 ◼   ► Jamf Now helps secure devices so you can focus on your business

00:51:58 ◼   ► instead.

00:51:59 ◼   ► And it's super easy to use.

00:52:01 ◼   ► There is no IT experience needed.

00:52:03 ◼   ► So our listeners, you can start securing your business today

00:52:07 ◼   ► by setting up your first three devices for free.

00:52:10 ◼   ► Add more, starting at just $2 a month per device.

00:52:14 ◼   ► So create your free account today at jamf.com/atp.

00:52:19 ◼   ► That's J-A-M-F dot com slash ATP.

00:52:22 ◼   ► Thank you so much to Jamf Now for sponsoring our show.

00:52:25 ◼   ► [MUSIC PLAYING]

00:52:29 ◼   ► So yeah, so Google's a great place.

00:52:30 ◼   ► I had a lot of fun there.

00:52:32 ◼   ► There's a lot of really talented people.

00:52:34 ◼   ► I love that they're ambitious and willing to swing

00:52:37 ◼   ► hard to do amazing things.

00:52:40 ◼   ► And so it's really great.

00:52:42 ◼   ► But due to an unfortunate incident with the free food,

00:52:44 ◼   ► you were forced to leave and go to a new company.

00:52:47 ◼   ► Well, just because something is good

00:52:48 ◼   ► doesn't mean it can't be better, right?

00:52:51 ◼   ► All right, well, so what's your sci-fi pitch?

00:52:53 ◼   ► I mean, the downsides of Google are things

00:52:56 ◼   ► like it's a gigantic company with many tentacles.

00:52:59 ◼   ► It's--

00:52:59 ◼   ► Tentacles?

00:53:00 ◼   ► --I should say it.

00:53:01 ◼   ► Well, it's got its tentacles into everything.

00:53:03 ◼   ► It's trying to be and do everything in the industry.

00:53:07 ◼   ► I think the other bigger issue is it's just

00:53:08 ◼   ► kind of becoming bureaucratic, like many big companies do.

00:53:12 ◼   ► And so certain aspects of that, like the performance review

00:53:16 ◼   ► system, is just a nightmare.

00:53:19 ◼   ► And so there are certain aspects of that that are just

00:53:21 ◼   ► standard big company life.

00:53:23 ◼   ► And with sci-fi, the appeal is it's a much smaller company.

00:53:27 ◼   ► It's hundreds of people.

00:53:30 ◼   ► And it's not an early stage startup,

00:53:32 ◼   ► but it's way smaller than one of the big companies.

00:53:37 ◼   ► But it's really got an ambitious charter

00:53:40 ◼   ► of reinventing how semiconductors are made.

00:53:42 ◼   ► And so what my team does at sci-fi

00:53:45 ◼   ► is really rethinking how people design, build processors,

00:53:51 ◼   ► but also just accelerators in general from the beginning.

00:53:55 ◼   ► And there's a tremendous number of compiler, language,

00:53:58 ◼   ► lots of other familiar problems, as well as

00:54:00 ◼   ► a lot of very developer tool-y kinds of problems

00:54:03 ◼   ► where you're trying to enable a new kind of user

00:54:07 ◼   ► to be productive and be able to do things

00:54:09 ◼   ► they couldn't do before.

00:54:10 ◼   ► So it's a lot of fun.

00:54:11 ◼   ► It's really exciting.

00:54:12 ◼   ► And it's also a space where the existing tools, whether they

00:54:16 ◼   ► be proprietary or open source, all

00:54:18 ◼   ► have different kinds of problems.

00:54:19 ◼   ► And there's a huge opportunity, or so it seems,

00:54:21 ◼   ► to make the world better.

00:54:23 ◼   ► It sounds like there's a lot of synergy with the MLR stuff

00:54:25 ◼   ► you were talking about.

00:54:26 ◼   ► Like any kind of industry like that,

00:54:28 ◼   ► that is especially something about building hardware,

00:54:31 ◼   ► probably has a whole bunch of software tools.

00:54:33 ◼   ► But the love and attention those software tools

00:54:35 ◼   ► get for this very narrow market is probably not very big.

00:54:38 ◼   ► And so people just deal with the tools

00:54:40 ◼   ► that they have from a limited number of vendors

00:54:42 ◼   ► that are not nearly as friendly and as polished as the tools

00:54:45 ◼   ► for a larger platform, let's say.

00:54:48 ◼   ► Yeah, that's exactly right.

00:54:49 ◼   ► And there's also an aspect of MLR

00:54:52 ◼   ► is fundamentally transformative compiler technology,

00:54:55 ◼   ► in my opinion.

00:54:56 ◼   ► And I say that having worked on a lot of compilers.

00:54:59 ◼   ► There's nothing else like it.

00:55:00 ◼   ► And it really opens the door to a lot

00:55:02 ◼   ► of really interesting new kinds of ways to solve old problems.

00:55:06 ◼   ► And not all the tools in this chip design space

00:55:11 ◼   ► are built on state-of-the-art technology.

00:55:13 ◼   ► And so there's unique opportunities when you say,

00:55:16 ◼   ► hey, let's take a thing that didn't exist until 18 months

00:55:18 ◼   ► ago and apply it in a new way to a field that

00:55:20 ◼   ► is really, really important and has a lot of smart people

00:55:25 ◼   ► working in it.

00:55:26 ◼   ► And just like the TensorFlow and the Swift for TensorFlow

00:55:29 ◼   ► projects and things like that are about building tools

00:55:31 ◼   ► for data scientists, here's about building tools

00:55:33 ◼   ► for chip designers.

00:55:35 ◼   ► Does MLR have a animal mascot logo thing?

00:55:39 ◼   ► It has a abstracted cuboid M. It's a geometric design.

00:55:45 ◼   ► It's not nearly as cute as the LVM dragon.

00:55:47 ◼   ► Yeah, a giant silver dragon.

00:55:50 ◼   ► We got to workshop the MLR thing.

00:55:53 ◼   ► You're very enthusiastic about it, but it's no dragon.

00:55:55 ◼   ► Seriously.

00:55:56 ◼   ► And John, did you even catch that somewhere along the way,

00:55:59 ◼   ► the LVM dragon got its head installed right side up?

00:56:01 ◼   ► Yes.

00:56:02 ◼   ► I brought this up with you many times.

00:56:03 ◼   ► I'm glad to know that that was corrected.

00:56:05 ◼   ► Yes.

00:56:06 ◼   ► So there's progress being made in all fronts.

00:56:09 ◼   ► Wow.

00:56:10 ◼   ► So to come back to Sci-5, to the best year willing and capable

00:56:14 ◼   ► of saying, who is the kind of customer

00:56:16 ◼   ► that you guys are courting?

00:56:17 ◼   ► And I don't necessarily mean like name Tesla or something

00:56:20 ◼   ► like that.

00:56:20 ◼   ► I'm just saying like, what kinds of more specific problems

00:56:23 ◼   ► are you looking to solve?

00:56:24 ◼   ► I understand what you were saying about making chip design

00:56:27 ◼   ► better and faster and whatnot.

00:56:29 ◼   ► But how does that apply to an actual thing

00:56:32 ◼   ► I can hold in my hand?

00:56:33 ◼   ► Sure, so I think that there is a difference between what

00:56:36 ◼   ► Sci-5 is about and what my role in Sci-5 is.

00:56:40 ◼   ► Oh, OK.

00:56:41 ◼   ► Sci-5 as a company has a couple of different things

00:56:43 ◼   ► going on.

00:56:45 ◼   ► It is-- are you familiar with the RISC-V instruction

00:56:48 ◼   ► set and that movement?

00:56:49 ◼   ► I know that it is a thing, but I don't know much more than that.

00:56:52 ◼   ► OK, so let me give a quick dive into that.

00:56:56 ◼   ► So RISC-V, it's an instruction set like ARM or like x86

00:57:00 ◼   ► from Intel and AMD or MIPS or PowerPC.

00:57:03 ◼   ► It's like one of those kinds of things.

00:57:05 ◼   ► So if you talk about the others, so you talk about MIPS, x86 ARM,

00:57:10 ◼   ► PowerPC, like all these things, they're all proprietary.

00:57:13 ◼   ► And so they're all owned by a very large company,

00:57:17 ◼   ► and that very large company controls its destiny.

00:57:20 ◼   ► And so this control comes from multiple--

00:57:24 ◼   ► in multiple ways.

00:57:25 ◼   ► So they have a roadmap, and they publish new specifications.

00:57:28 ◼   ► And if you ask nicely, they will listen to you,

00:57:31 ◼   ► and they will consider your input.

00:57:32 ◼   ► But they decide what to do about it.

00:57:35 ◼   ► But the other thing about it is that they're

00:57:37 ◼   ► the guardians of all the chips.

00:57:40 ◼   ► And so you either have to buy a chip from them,

00:57:44 ◼   ► or you have to buy a license to make a chip using

00:57:47 ◼   ► their instruction set.

00:57:48 ◼   ► And so this has been a very successful model

00:57:51 ◼   ► for these companies for a really long time.

00:57:54 ◼   ► Look at what Intel has done with x86, right,

00:57:57 ◼   ► and the Wintel monopoly back in the day.

00:58:00 ◼   ► Or look at what ARM is doing now for cell phones, for example.

00:58:04 ◼   ► But this is not really great if you want to do something really

00:58:07 ◼   ► custom, or if you don't want to be locked

00:58:10 ◼   ► into one particular vendor.

00:58:11 ◼   ► And there are actual instances where vendors go out of business

00:58:14 ◼   ► and then your instruction set and all the software built

00:58:17 ◼   ► around it are stranded.

00:58:18 ◼   ► Things like HP even had an instruction set called PA-Risk.

00:58:24 ◼   ► And they eventually abandoned it with Itanium,

00:58:28 ◼   ► and the whole debacle around that,

00:58:29 ◼   ► which really left their server users in kind of a weird spot.

00:58:34 ◼   ► And so you could argue that building your whole world

00:58:38 ◼   ► on top of a single proprietary vendor

00:58:40 ◼   ► leaves you tied to their destiny, right?

00:58:43 ◼   ► I think it's true for all proprietary things.

00:58:45 ◼   ► So what RISC-V is is RISC-V came out of Berkeley.

00:58:48 ◼   ► And it was originally a group of academics

00:58:51 ◼   ► that were working on RISC instruction sets.

00:58:53 ◼   ► And they were the pioneers of RISC back in the day.

00:58:57 ◼   ► And so RISC-V is the fifth incarnation

00:59:00 ◼   ► of the Berkeley research group's design for RISC processors.

00:59:06 ◼   ► And what they did with RISC-V, which was really cool

00:59:10 ◼   ► and interesting, is they open sourced

00:59:12 ◼   ► and opened the entire design process for the instruction set.

00:59:16 ◼   ► And so not only is it a patent-free,

00:59:20 ◼   ► non-licensed, open kind of instruction set,

00:59:26 ◼   ► but there are open design processes,

00:59:28 ◼   ► kind of like Swift Evolution,

00:59:29 ◼   ► for the instruction set themselves.

00:59:30 ◼   ► And so you too can design your own RISC-V processor

00:59:35 ◼   ► and decide to add new instructions

00:59:36 ◼   ► without talking to anybody if you want.

00:59:39 ◼   ► But the community also realizes that if everybody does that,

00:59:42 ◼   ► well, you get huge fragmentation.

00:59:44 ◼   ► And so it's better for people to work together

00:59:46 ◼   ► in a collaborative, cross-industry way

00:59:48 ◼   ► to define new extensions to that.

00:59:51 ◼   ► And so RISC-V is, I think,

00:59:53 ◼   ► it's still a bit early in certain ways,

00:59:55 ◼   ► but it's really eating the industry

00:59:57 ◼   ► in a very interesting way.

01:00:00 ◼   ► And it's this wave that's really kind of taking over things.

01:00:03 ◼   ► Now, Sci-Fi was founded by the creators of RISC-V.

01:00:07 ◼   ► And so the founders of Sci-Fi were from that research group.

01:00:10 ◼   ► They had designed the RISC-V instruction set.

01:00:13 ◼   ► And so they founded the company initially around that.

01:00:16 ◼   ► And their idea was to productize

01:00:19 ◼   ► and commercialize the RISC-V design,

01:00:22 ◼   ► build actual processors for it,

01:00:23 ◼   ► and they've been doing that for quite some time.

01:00:25 ◼   ► Sci-Fi has since evolved.

01:00:29 ◼   ► And so, yes, it is the RISC-V company,

01:00:33 ◼   ► or the leading RISC-V company.

01:00:36 ◼   ► Yes, it has some amazing RISC-V CPU designs,

01:00:40 ◼   ► which you can license and put them in your own chips,

01:00:42 ◼   ► but it's also what is called now an idea to silicon company.

01:00:45 ◼   ► And so they have all the functions for chip design in-house.

01:00:48 ◼   ► And so you can walk up and say,

01:00:49 ◼   ► "Hey, I'm gonna build a toaster or a microwave,

01:00:52 ◼   ► and I need this and this,

01:00:53 ◼   ► and it needs to be Bluetooth enabled,"

01:00:54 ◼   ► because of course it does.

01:00:56 ◼   ► (laughing)

01:00:57 ◼   ► But I don't want it to be big,

01:00:58 ◼   ► so take out this and that and the other feature,

01:01:00 ◼   ► and the entire stack and the entire system

01:01:02 ◼   ► can be completely customized to your needs.

01:01:04 ◼   ► And if you wanna own the design

01:01:06 ◼   ► because you're into owning,

01:01:08 ◼   ► you're an ML accelerator,

01:01:09 ◼   ► machine learning accelerator company,

01:01:12 ◼   ► and you're building your secret sauce,

01:01:13 ◼   ► and that's your accelerator,

01:01:14 ◼   ► but yeah, you still need RAM interfaces

01:01:17 ◼   ► and all the other standard things

01:01:18 ◼   ► that go with this PCI interface and stuff like that.

01:01:20 ◼   ► Well, you can own or customize however much you want,

01:01:23 ◼   ► but Sci-Fi can then bring it to market for you.

01:01:26 ◼   ► Now, Sci-Fi is not unique in that respect,

01:01:29 ◼   ► but what they're doing,

01:01:31 ◼   ► and one of the things that I'm very excited about

01:01:33 ◼   ► and what my team is driving is

01:01:35 ◼   ► working on all the tooling, the methodology,

01:01:37 ◼   ► the cloud platforms, all this stuff

01:01:38 ◼   ► to make it just better, faster, cheaper,

01:01:40 ◼   ► improving turnaround time,

01:01:42 ◼   ► making it so the tools work together,

01:01:43 ◼   ► there's more commonality,

01:01:45 ◼   ► and just making that just way better

01:01:49 ◼   ► than what you get out of a bunch of existing tools

01:01:52 ◼   ► that are kind of cobbled together.

01:01:55 ◼   ► So it's a really exciting time.

01:01:56 ◼   ► They're really hard problems.

01:01:58 ◼   ► It's an industry that is very mature,

01:02:00 ◼   ► and there's a lot of very established, very good players,

01:02:03 ◼   ► and there's a lot of great tools,

01:02:05 ◼   ► but there's also, I think,

01:02:07 ◼   ► a good opportunity to look at the big space

01:02:09 ◼   ► and try to understand how it all fits together

01:02:11 ◼   ► and come up with new ideas.

01:02:13 ◼   ► So we'll see how it goes.

01:02:14 ◼   ► It's a long-term journey.

01:02:16 ◼   ► - You're mentioning the open source instruction said,

01:02:20 ◼   ► and the fact that companies like this can exist

01:02:24 ◼   ► because risk five is not proprietary,

01:02:26 ◼   ► so they can build products based on it

01:02:29 ◼   ► and use that as a tool set to,

01:02:31 ◼   ► if a customer comes to them,

01:02:32 ◼   ► so they want a thing that does whatever,

01:02:34 ◼   ► they don't have to license an instruction set

01:02:36 ◼   ► from somebody they can use risk five.

01:02:37 ◼   ► Makes me think about Apple's usual MO.

01:02:41 ◼   ► Well, they have two different MOs.

01:02:42 ◼   ► One, the more traditional one,

01:02:44 ◼   ► was that they come up with something themselves,

01:02:48 ◼   ► and it would be their thing, and they would control it,

01:02:50 ◼   ► and they're happy with that, and everyone's great.

01:02:51 ◼   ► But in the modern world, Apple can't invent

01:02:55 ◼   ► everything themselves, so they had two options.

01:02:58 ◼   ► The old one was, okay, well,

01:03:00 ◼   ► we'll use something from a third party.

01:03:03 ◼   ► And then the newer option is,

01:03:04 ◼   ► a proprietary thing from a third party,

01:03:06 ◼   ► and the newer option is we'll use an open thing

01:03:09 ◼   ► that nobody owns in particular.

01:03:11 ◼   ► So example of the third-party one is like,

01:03:13 ◼   ► we're gonna put CPUs in our computers,

01:03:15 ◼   ► we'll buy them from Motorola,

01:03:16 ◼   ► we'll buy them from IBM, whatever.

01:03:19 ◼   ► And that has been a relationship

01:03:20 ◼   ► that's been a little bit fraught.

01:03:21 ◼   ► The open source one is we're gonna build

01:03:22 ◼   ► a new operating system, we'll build it on top of BSD,

01:03:24 ◼   ► we'll build it on top of Mach, right?

01:03:26 ◼   ► We'll build a new compiler, toolchain on top of LVM,

01:03:29 ◼   ► which is open, and even going to the extent

01:03:32 ◼   ► of building some things themselves in-house,

01:03:34 ◼   ► either based on open source projects like WebKit from KHTML,

01:03:37 ◼   ► or saying we're gonna make a new language,

01:03:38 ◼   ► and also that language is gonna be open source,

01:03:40 ◼   ► even though that was from the outside,

01:03:42 ◼   ► touch and go until the official word

01:03:44 ◼   ► that was gonna be open, that's a choice that they've made.

01:03:47 ◼   ► But still, within Apple, there are situations

01:03:49 ◼   ► where they're still in the old pattern,

01:03:51 ◼   ► which is we're using a thing, it's super important,

01:03:54 ◼   ► we don't own it, and it's proprietary in some form.

01:03:57 ◼   ► And one example of that would be the ARM instruction set

01:03:59 ◼   ► that Apple has this license for

01:04:00 ◼   ► with whoever owns the ARM stuff, right?

01:04:03 ◼   ► And that's, you know, they make their own ARM CPUs,

01:04:07 ◼   ► A whatever, blah, blah, blah, but they don't,

01:04:10 ◼   ► Apple does not own the ARM instruction set,

01:04:12 ◼   ► nor is it open source that anybody can use for free,

01:04:14 ◼   ► like RISC-V.

01:04:15 ◼   ► So depending on how the RISC-V thing goes,

01:04:19 ◼   ► and we've been talking for a while in this program

01:04:20 ◼   ► about speculating transitions of the Mac

01:04:23 ◼   ► to the ARM platform, and in our various discussions,

01:04:26 ◼   ► we have brought up the idea of like,

01:04:27 ◼   ► why is everyone thinking it has to be ARM,

01:04:29 ◼   ► or why does everyone think it has to be away from x86,

01:04:31 ◼   ► or whatever, couldn't Apple just come up

01:04:32 ◼   ► with its own instruction set,

01:04:33 ◼   ► what about the inertia of ARM, and all this other stuff?

01:04:36 ◼   ► I can imagine a future where RISC-V either has the cloud,

01:04:41 ◼   ► or Apple decides to give it the, you know,

01:04:44 ◼   ► to hitch its wagon to that star,

01:04:46 ◼   ► kind of like it did with KHTML,

01:04:47 ◼   ► which did not have a lot of cloud in the web browser

01:04:49 ◼   ► community before Apple sort of adopted it,

01:04:52 ◼   ► but kept it open, then Apple could have an instruction set

01:04:56 ◼   ► that is no longer at the mercy of some company

01:04:59 ◼   ► that is not Apple, but that is nevertheless

01:05:01 ◼   ► a private company, and I always think ARM

01:05:02 ◼   ► is gonna go out of business, and honestly,

01:05:04 ◼   ► Apple could probably buy them

01:05:05 ◼   ► if they really got into a pinch,

01:05:07 ◼   ► but I do wonder about, you know,

01:05:09 ◼   ► if the timelines are lined up differently,

01:05:12 ◼   ► we could all be using iPhones with chips inside them

01:05:15 ◼   ► that run some kind of RISC-V instruction set,

01:05:17 ◼   ► and it wouldn't be that different of a world,

01:05:20 ◼   ► and I bet Apple, it would fit better

01:05:21 ◼   ► with the current model of Apple,

01:05:23 ◼   ► which is, if we don't own it totally in-house,

01:05:25 ◼   ► it should be open.

01:05:27 ◼   ► - Well, so I don't know anything about Apple's plans

01:05:29 ◼   ► in this space, so I can't comment to that, obviously,

01:05:32 ◼   ► but I think there's a couple of different factors

01:05:35 ◼   ► that I would weigh into this, so Apple,

01:05:37 ◼   ► Apple's big enough they could theoretically do anything,

01:05:40 ◼   ► right, and so one of the things that impresses me

01:05:43 ◼   ► about Apple is that in the face of immense resources,

01:05:48 ◼   ► they're still incredibly strategic, right,

01:05:51 ◼   ► and it's very, this is something that I don't think

01:05:53 ◼   ► Google is quite as good at, is that Google tries

01:05:56 ◼   ► to do everything because they can, right,

01:05:59 ◼   ► and that's not always great.

01:06:02 ◼   ► - I think Google shut down that kite-based Wi-Fi thing,

01:06:06 ◼   ► didn't they, so they're raining it in a little bit.

01:06:09 ◼   ► - Yeah, so anyways, I don't wanna speak negatively

01:06:11 ◼   ► about Google, but the thing with instruction sets

01:06:15 ◼   ► is that there's a huge software ecosystem

01:06:17 ◼   ► that goes with that instruction set,

01:06:20 ◼   ► and so one of the things that's interesting about RISC-V

01:06:23 ◼   ► is that you could look at this and say,

01:06:25 ◼   ► well, the RISC-V instruction set's not that interesting,

01:06:29 ◼   ► right, and I think the people behind RISC-V

01:06:31 ◼   ► would say that's the whole idea,

01:06:32 ◼   ► it should not be interesting, it should be very

01:06:34 ◼   ► straightforward to compile for and things like this,

01:06:36 ◼   ► but the point is not that the instruction set

01:06:39 ◼   ► is magic in some way, right, the point is that

01:06:42 ◼   ► at the state of evolution that processors are at,

01:06:46 ◼   ► the instruction set isn't the most interesting piece,

01:06:48 ◼   ► it's about the software that lives on top of it,

01:06:51 ◼   ► and at any point in time, somebody could go invent

01:06:53 ◼   ► their own instruction set, that does happen,

01:06:56 ◼   ► particularly in like grad school,

01:06:57 ◼   ► if you're an electrical engineer and you're designing a chip,

01:07:00 ◼   ► you can invent your own instruction set,

01:07:02 ◼   ► but if you do that, you have no software.

01:07:04 ◼   ► Where's the web browser, where's, you know,

01:07:06 ◼   ► all the other things that go with this,

01:07:08 ◼   ► where's the compiler for, the C compiler,

01:07:10 ◼   ► all these things, and so the cool thing about RISC-V here

01:07:13 ◼   ► is that you can say, I'm gonna design my own CPU,

01:07:17 ◼   ► but I get the software, and that I think is very,

01:07:20 ◼   ► very interesting and cool.

01:07:22 ◼   ► Now, coming back to Apple, Apple doesn't need that,

01:07:24 ◼   ► they could invent their own thing,

01:07:25 ◼   ► they have multiple times before,

01:07:27 ◼   ► so I don't know how they weigh all these things,

01:07:29 ◼   ► but yeah, I think for the general industry,

01:07:34 ◼   ► it's a pretty big factor.

01:07:38 ◼   ► - We are sponsored this week by Linode,

01:07:40 ◼   ► my favorite web host.

01:07:41 ◼   ► Whether you're working on a personal project

01:07:43 ◼   ► or managing your enterprise's infrastructure,

01:07:45 ◼   ► Linode Cloud Hosting has the pricing, support, and scale

01:07:48 ◼   ► you need to take your project to the next level.

01:07:51 ◼   ► They have 11 data centers worldwide so far

01:07:53 ◼   ► and they're always adding more.

01:07:54 ◼   ► All these are filled with enterprise grade hardware

01:07:57 ◼   ► and their next generation network to back it all up.

01:07:59 ◼   ► So Linode Cloud Hosting delivers the server performance

01:08:02 ◼   ► you expect at prices that frankly, you don't.

01:08:05 ◼   ► I've been a Linode customer myself

01:08:07 ◼   ► since long before they were sponsoring,

01:08:09 ◼   ► about eight or nine years now.

01:08:10 ◼   ► That's where I host my personal server for my blog

01:08:13 ◼   ► and the livestream for the show.

01:08:14 ◼   ► It's also where I host all of Overcast

01:08:16 ◼   ► and that makes up about 25 servers there

01:08:19 ◼   ► and I've been with them this entire time

01:08:21 ◼   ► and I am so happy with Linode.

01:08:23 ◼   ► They are amazing value for the money.

01:08:25 ◼   ► Like, you get such good specs,

01:08:27 ◼   ► you look around the industry

01:08:28 ◼   ► and for the entire time I've been with them,

01:08:30 ◼   ► they have been top of the line

01:08:32 ◼   ► in both performance and in value.

01:08:35 ◼   ► They really are great.

01:08:36 ◼   ► I've never found anybody who can beat them

01:08:37 ◼   ► consistently on value and so that's why I'm there

01:08:40 ◼   ► 'cause I spend a lot of money on servers

01:08:41 ◼   ► and I get a lot for it there

01:08:43 ◼   ► and I wouldn't get that much for my money

01:08:45 ◼   ► or I'd have to spend a lot more,

01:08:46 ◼   ► pretty much every else I've ever seen.

01:08:48 ◼   ► And they have great support if I need it,

01:08:49 ◼   ► they have great performance,

01:08:50 ◼   ► they have all these different options.

01:08:52 ◼   ► So anything you need from basic things

01:08:54 ◼   ► like their wonderful $5 a month Nano plan

01:08:56 ◼   ► which gives you a gig of RAM these days

01:08:58 ◼   ► and over time when they're able to offer more for the money,

01:09:01 ◼   ► they just upgrade their plans at the same prices

01:09:04 ◼   ► and you can just upgrade your servers

01:09:06 ◼   ► with one click to the new plans, it's incredible.

01:09:08 ◼   ► So see for yourself, go to linode.com/atp

01:09:12 ◼   ► and when you create a new account,

01:09:13 ◼   ► use promo code ATP2020 to get a $20 credit

01:09:17 ◼   ► for your next project.

01:09:19 ◼   ► Linode is so great, I highly recommend it.

01:09:21 ◼   ► If you need servers, check out Linode Cloud Hosting.

01:09:24 ◼   ► Once again, linode.com/atp,

01:09:27 ◼   ► promo code ATP2020 when creating a new account

01:09:30 ◼   ► to get a $20 credit.

01:09:31 ◼   ► Thank you so much to Linode for hosting all my stuff

01:09:34 ◼   ► and sponsoring our show.

01:09:35 ◼   ► - So Chris, you haven't been at Apple for several years now

01:09:42 ◼   ► but you are very much a veteran of WWDC

01:09:44 ◼   ► and you have presented at several,

01:09:48 ◼   ► to the best of my recollection if not all the ones

01:09:50 ◼   ► when you were at Apple.

01:09:51 ◼   ► So as someone who has been on the big stage,

01:09:55 ◼   ► the biggest stage in fact at WWDC,

01:09:58 ◼   ► knowing now that it's gone online only,

01:10:01 ◼   ► how do you feel about that?

01:10:03 ◼   ► Do you think it's workable?

01:10:05 ◼   ► What would Apple do about labs?

01:10:07 ◼   ► Like do you have any thoughts, opinions,

01:10:09 ◼   ► anything you'd like to discuss with regard to WWDC?

01:10:12 ◼   ► - Yeah, again, disclaimer, I haven't been there for years

01:10:16 ◼   ► and I don't know how they think about it

01:10:17 ◼   ► but the way I think about it is,

01:10:19 ◼   ► I think it's actually a huge opportunity in certain ways.

01:10:22 ◼   ► So WWDC as a thing has needed to evolve anyways in my opinion

01:10:28 ◼   ► because you've got too much demand and too little supply.

01:10:31 ◼   ► The labs, if you're one of the lucky people

01:10:33 ◼   ► to get the lottery tickets and you get into the labs,

01:10:36 ◼   ► it's incredibly valuable but what about everybody else?

01:10:39 ◼   ► Right, and the talks, the talks are great

01:10:44 ◼   ► but also variable depending on the day each speaker has

01:10:46 ◼   ► and so this gives you a chance to improve the quality

01:10:49 ◼   ► of the talks and maybe relieve some of the pressure on that.

01:10:53 ◼   ► Now the question for me is,

01:10:55 ◼   ► does what do they do with the keynote

01:10:57 ◼   ► and other things like that which were kind of positioned

01:10:59 ◼   ► as being technical content

01:11:01 ◼   ► but they're really marketing talks, right?

01:11:03 ◼   ► And so how does that all work?

01:11:05 ◼   ► And I'm not sure about that

01:11:06 ◼   ► but I think that overall the event will be great

01:11:09 ◼   ► and maybe this will force an evolution that allows labs

01:11:12 ◼   ► or the replacement for labs to be more scalable

01:11:15 ◼   ► to more people and maybe that will be a good thing.

01:11:18 ◼   ► Now I think that the other side of this is

01:11:22 ◼   ► that this whole shelter in place,

01:11:24 ◼   ► people can't talk to each other,

01:11:26 ◼   ► you can't go to conferences really impacts a lot of things

01:11:29 ◼   ► including WWDC where the community aspects

01:11:31 ◼   ► of pulling people physically together

01:11:33 ◼   ► and you go have a beer with each other

01:11:34 ◼   ► and you see each other once a week,

01:11:37 ◼   ► that I don't think is really replaceable

01:11:40 ◼   ► and I would argue that in the case of WWDC

01:11:44 ◼   ► that's already being diluted anyways

01:11:46 ◼   ► just because the community is too big for the venue

01:11:51 ◼   ► but I think that's something that is an industry,

01:11:54 ◼   ► it's gonna be really interesting to see

01:11:55 ◼   ► what this summer or what happens with all the events.

01:12:00 ◼   ► It's really hard for people on the event planning side

01:12:02 ◼   ► thanks to dealing with the uncertainty.

01:12:04 ◼   ► - One of the things we've been talking about

01:12:05 ◼   ► speaking of like limited resources and supply and demand

01:12:08 ◼   ► is exactly how much of Apple employees

01:12:13 ◼   ► and engineers time WWDC takes.

01:12:16 ◼   ► We're always sort of speculating like,

01:12:18 ◼   ► does not having to do a presentation on stage live

01:12:22 ◼   ► in the moment with all the rehearsals and everything,

01:12:24 ◼   ► does that free up any engineering resources

01:12:27 ◼   ► or is it more or less about the same amount of time

01:12:29 ◼   ► 'cause now they're just gonna have to be standing

01:12:30 ◼   ► in front of a camera, pre-recording things or whatever.

01:12:33 ◼   ► How would you characterize the amount of time

01:12:35 ◼   ► you felt like it took you and your teams

01:12:38 ◼   ► to prepare for WWDC?

01:12:40 ◼   ► Did you feel like you could better spend that time

01:12:42 ◼   ► doing something else or did you feel like,

01:12:43 ◼   ► look, this is stuff we have to do anyway

01:12:45 ◼   ► and it's good that we're doing it?

01:12:47 ◼   ► - I think it depends on what,

01:12:49 ◼   ► so again, this seems like a huge opportunity to me

01:12:52 ◼   ► because the dynamic that happens with WWDC

01:12:55 ◼   ► is that there are multiple things

01:12:58 ◼   ► that Apple gets out of WWDC.

01:13:01 ◼   ► One of which is having that Monday morning keynote

01:13:04 ◼   ► forces all the builds to converge.

01:13:06 ◼   ► Beta one has to happen.

01:13:11 ◼   ► And that is an incredibly valuable forcing function

01:13:14 ◼   ► for engineering management to have a clearly defined,

01:13:19 ◼   ► easy to articulate goal that everybody understands

01:13:22 ◼   ► and can rally behind

01:13:23 ◼   ► and you just don't have the ambiguity there.

01:13:25 ◼   ► Now, the bad thing about WWDC historically

01:13:27 ◼   ► is that crunch time,

01:13:29 ◼   ► and it is a little bit of crunch time as you can imagine,

01:13:31 ◼   ► happens exactly the same time

01:13:33 ◼   ► you're preparing the technical talks.

01:13:35 ◼   ► So you're trying to, on the one hand,

01:13:36 ◼   ► have amazing technical content that you're preparing.

01:13:40 ◼   ► On the other hand, you're trying to fix

01:13:41 ◼   ► and manage the fixing of all the bugs

01:13:43 ◼   ► and this is very difficult to juggle.

01:13:45 ◼   ► Now, if they decided to have the keynote on day X

01:13:49 ◼   ► and then have the technical talks roll out two weeks later,

01:13:52 ◼   ► you could really change that dynamic

01:13:54 ◼   ► and maybe that would be better for the sanity of everybody.

01:13:58 ◼   ► I don't think it necessarily dramatically reduces

01:14:01 ◼   ► the amount of time that it would take to produce that.

01:14:04 ◼   ► The thing that you get back, I think,

01:14:06 ◼   ► is the 1000 Apple engineers or however many it is

01:14:10 ◼   ► that actually attend a week of the conference.

01:14:12 ◼   ► And so you get that back,

01:14:14 ◼   ► but I think that is probably minor

01:14:17 ◼   ► when you look across all of Apple's engineering efforts

01:14:20 ◼   ► at this point.

01:14:21 ◼   ► So I don't know, I think it'll be really interesting

01:14:23 ◼   ► to see what they do.

01:14:24 ◼   ► And Apple's a very strategic and very smart company

01:14:27 ◼   ► and has a lot of very smart people.

01:14:28 ◼   ► I'm sure they're looking at how to turn this

01:14:31 ◼   ► into a new opportunity

01:14:32 ◼   ► and what new things they can do with the format

01:14:34 ◼   ► and how they can delight people in new ways.

01:14:38 ◼   ► - If you had to put money for or against

01:14:40 ◼   ► there ever being another in-person thing like WWDC again,

01:14:44 ◼   ► would you say for or against?

01:14:47 ◼   ► You gotta put money out of your $1, $1 bet.

01:14:50 ◼   ► - Well, so this comes back to your observation about Swift,

01:14:54 ◼   ► which is you do a thing and then see how it works out.

01:14:57 ◼   ► Right?

01:14:58 ◼   ► And so I think that if they do WWDC virtually this year

01:15:01 ◼   ► and it sucks, then it's probably gonna go physical again.

01:15:04 ◼   ► - That's not how betting works.

01:15:07 ◼   ► - I don't know.

01:15:07 ◼   ► I would wager that it doesn't return

01:15:11 ◼   ► to its original format.

01:15:13 ◼   ► So if there is an in-person event,

01:15:15 ◼   ► I think it would be significantly different

01:15:17 ◼   ► than what the historical events have been.

01:15:19 ◼   ► - I'll see.

01:15:21 ◼   ► - So we'll see.

01:15:22 ◼   ► I don't know.

01:15:23 ◼   ► - It's been funny for me because as I found out

01:15:26 ◼   ► about this news, which we all kind of expected,

01:15:28 ◼   ► on the one hand, I was kind of happy

01:15:29 ◼   ► because I've been lucky enough to go to many

01:15:33 ◼   ► of the last WWDCs and have a ticket.

01:15:36 ◼   ► But nevertheless, it is certainly way, way, way too little

01:15:41 ◼   ► for the amount of demand that there is, as you said, Chris.

01:15:43 ◼   ► And so on the one side for the actual event itself,

01:15:45 ◼   ► I was like, oh, you know what?

01:15:46 ◼   ► I'll probably do the best.

01:15:47 ◼   ► But then on the other side,

01:15:48 ◼   ► and this is another thing you said a minute ago,

01:15:50 ◼   ► not being able to rub shoulders with not only my two co-hosts

01:15:54 ◼   ► but a lot of the people, some of whom are within Apple

01:15:56 ◼   ► that I know and some of our mutual friends

01:15:59 ◼   ► that all kind of descend on San Jose that one week in June,

01:16:03 ◼   ► I'm really gonna miss that quite a bit

01:16:04 ◼   ► if that doesn't ever return.

01:16:06 ◼   ► And I don't know how the community will replace it,

01:16:07 ◼   ► but it is definitely a yin and yang sort of thing

01:16:11 ◼   ► that I'm very curious to see what'll happen in 2021,

01:16:14 ◼   ► assuming we all survive till then.

01:16:16 ◼   ► But moving along--

01:16:18 ◼   ► - Well, so let's flip that question around.

01:16:21 ◼   ► What opportunity does this produce

01:16:22 ◼   ► for things like AltConf, right?

01:16:25 ◼   ► Because there are people that are interested

01:16:27 ◼   ► in pulling people together,

01:16:28 ◼   ► and there are a lot of interesting community events,

01:16:31 ◼   ► and maybe this is an opportunity for them to benefit

01:16:34 ◼   ► and for the community to self-organize

01:16:36 ◼   ► in ways that Apple perhaps wasn't super great at

01:16:39 ◼   ► in the first place.

01:16:40 ◼   ► - Yeah, I couldn't agree more.

01:16:41 ◼   ► And I think some of the trouble is

01:16:43 ◼   ► any sort of physical self-organizing

01:16:45 ◼   ► isn't happening quite obviously, but--

01:16:46 ◼   ► - Well, not this summer.

01:16:48 ◼   ► But I think the question is, as you project forward,

01:16:50 ◼   ► I'm really curious to see what the pandemic does

01:16:54 ◼   ► to human culture.

01:16:55 ◼   ► I mean, I think it's a really interesting,

01:16:58 ◼   ► you think about, I don't know,

01:17:00 ◼   ► pick your timeframe, two years from now,

01:17:01 ◼   ► three years from now when all the dust has settled,

01:17:03 ◼   ► things have gone more or less back to normal,

01:17:05 ◼   ► we have reassembled the US economy and the world economy.

01:17:08 ◼   ► What's different, right?

01:17:11 ◼   ► And 9/11, just within America,

01:17:14 ◼   ► had a fairly profound impact on a lot of little things

01:17:18 ◼   ► across our world.

01:17:20 ◼   ► And this is a global event, right?

01:17:23 ◼   ► This is a global event that's a much bigger impact.

01:17:26 ◼   ► And I think it's gonna be very interesting

01:17:27 ◼   ► to see how this changes the way we look at things.

01:17:30 ◼   ► It could have widespread effects

01:17:31 ◼   ► on how people think about in-person meetings.

01:17:34 ◼   ► You're working from home,

01:17:36 ◼   ► maybe far more socially acceptable

01:17:38 ◼   ► outside of software and other industries

01:17:41 ◼   ► where it's been more common.

01:17:43 ◼   ► I think it could have really interesting effects

01:17:45 ◼   ► on society and culture in general.

01:17:48 ◼   ► - Well, those things have really changed

01:17:49 ◼   ► if Apple starts letting people work from home.

01:17:51 ◼   ► (laughing)

01:17:54 ◼   ► - It does happen, it does happen.

01:17:55 ◼   ► So I feel like, Chris, in the first part of our conversation,

01:17:59 ◼   ► talking about your past of the last three or so years,

01:18:02 ◼   ► I keep coming back to something you said to us

01:18:06 ◼   ► the last time we spoke,

01:18:07 ◼   ► and this is not the only time you've said it.

01:18:08 ◼   ► You said in so many words,

01:18:10 ◼   ► you want Swift to take over the world.

01:18:11 ◼   ► And I feel like in many ways,

01:18:14 ◼   ► your roles at certainly Google in particular,

01:18:19 ◼   ► it is you trying to get Swift to take over the world.

01:18:22 ◼   ► So three years on from when the four of us last spoke,

01:18:25 ◼   ► how do you feel that goal is coming along?

01:18:28 ◼   ► Do you feel like you've made tremendous progress,

01:18:29 ◼   ► not enough progress?

01:18:31 ◼   ► Where do you feel like it lies?

01:18:32 ◼   ► - Well, so I think there's different ways

01:18:35 ◼   ► of interpreting this because my,

01:18:38 ◼   ► let's come back to why I believe that, right?

01:18:40 ◼   ► So my goal is never to take Swift

01:18:43 ◼   ► and shove it down the throats of people who don't want it.

01:18:46 ◼   ► Right?

01:18:47 ◼   ► My goal is for Swift to be such an amazing thing

01:18:50 ◼   ► that people want to use it, okay?

01:18:53 ◼   ► And when you look at this

01:18:55 ◼   ► and you look at the machine learning community,

01:18:58 ◼   ► when I was starting there

01:19:00 ◼   ► and when I was talking with people

01:19:01 ◼   ► when we were talking about this whole Python problem

01:19:03 ◼   ► and things like this,

01:19:04 ◼   ► and the general sentiment was everybody uses Python.

01:19:07 ◼   ► Yeah, it's not great, but everybody uses it.

01:19:09 ◼   ► The world will never change.

01:19:11 ◼   ► And I come at that and say,

01:19:12 ◼   ► hey, well, the world doesn't change

01:19:13 ◼   ► unless somebody steps up to change it.

01:19:15 ◼   ► You know?

01:19:16 ◼   ► And this is merely hard.

01:19:18 ◼   ► Let's work backwards from this.

01:19:19 ◼   ► Let's look at all the different problems,

01:19:20 ◼   ► including the migration problem and social acceptance

01:19:22 ◼   ► and da, da, da, da, interoperability,

01:19:25 ◼   ► like all of these things and solve the problems.