Under the Radar

Under the Radar 14: Linux Servers for App Developers


  welcome to under the radar a show but

  independent ios and app development I'm

  Marco Arment and I'm David Smith under

  the radar is never longer than 30


  so let's get started so last week we

  mentioned at the end how we were gonna

  tell this week tips and and stories and

  setups of how to run your own Linux

  servers because this is now you know in

  the wake of the parse announcement last

  week this is now kind of in the news and

  we think it's kind of a core competency

  that many app developers should know

  just how to run your own servers and and

  once you know this it enables your apps

  to be better to be more capable and and

  if you if you do kind of a standard

  server route without doing these like

  kind of more abstract proprietary

  services like parse it gives you a lot

  of flexibility with what you can do in

  the future and it kind of decouples you

  from depending on a particular third

  party to keep their their platform going

  yeah and this is probably fair

  disclaimer to say this is going to be an

  episode that goes a bit into the weeds

  like Dennis had some pretty low level

  stuff and if that's for you awesome if

  it isn't for you you know maybe stick

  around you might learn something and

  know that you know some some episodes

  are gonna be high level some episodes

  are gonna be low level and hopefully we

  can have a nice balance as a result so

  at a high level it's probably worth

  starting of just kind of talking about

  the the spectrum of hosting before we

  kind of end it before why we do it the

  way we do it

  so ultimately hosting see in many ways

  comes down to levels of abstraction like

  how abstracted away if they're like the

  actual computer that you're running your

  servers and your backends and systems on

  how much that is abstracted away from

  you so like at the lowest level you have

  a but I think it would be called

  colocation or something like that where

  you have a computer that's like your

  computer that you went to Best Buy or

  whatever to buy and you're taking it and

  you're putting it somewhere typically in

  some kind of data center that you put in

  a data center so that it has reliable

  power and reliable internet and reliable

  air-conditioning and all those types of

  things but like it's your machine most

  people don't do that I would really not

  recommend doing something like that most

  people with a very least we do but I

  think most people call like bare metal

  managed servers where it's not really

  your machine it's a service that they

  buy machines they manage machines if

  harddrive dies in the machine they'll go

  and fix it for you but it's you have

  actual like a dedicated piece of

  hardware you know a set of hard drives a

  motherboard a CPU that's exactly yours

  and you probably don't want that either

  I've I've gone down that road it's fine

  but it has a lot of difficulties that we

  aren't really offset by its costs both

  in monetary and like ease of use so most

  people who didn't even doing any kind of

  hosting themselves will use something

  that's more like a VPS which stands for

  virtual private server which is like

  basically like it's a machine from your

  apps perspective like you are your app

  sees it's a hard drive it sees a CPU it

  sees memory but all of those aren't

  actually physical things they're it's

  running on a shared environment with a

  bunch of other systems that I'll see the

  same things and share resources and you

  know you can use something like the node

  for this or digitalocean is another one

  that's pretty common in many ways ec2

  kind of runs like this but it's a bit

  different but you have these kinds of

  systems where you have a virtual Linux

  machine that you can do and you have

  some nice advantages that bit because

  because it's virtual you can do things

  like pick it up and move it around you

  know like you take an image of your hard

  drive and you can copy it onto another

  one and things like that that are really

  easy to manage and from a cost

  perspective it's way cheaper because

  you're sharing you know server resources

  and then above that you have things like

  virtual server environments things like

  a juror or Heroku or total abstractions

  like cloud cater parse or those types of

  things that are completely abstracting

  away from you and if I said for most

  people you probably want to start at a

  VPS like you want to find out where

  you're comfortable if you if the thought

  of administering a Linux server even

  after this episode is completely

  overwhelming like maybe go to something

  like Heroku or Azure where you have a

  little bit of control over what you're

  doing but mostly there's something there

  but it's still at least general-purpose

  but I think we both run for like reap

  yes sir vers because once you know what

  you're doing it's really cheap it's

  really efficient and you have a lot of

  flexibility that you could move

  providers you can spin up new servers

  and take them down really quickly and

  easily and you know especially like the

  cost is amazing it's incredible the

  amount of performance you can get from a

  really basic set up I mean at this point

  I think I administer 19 Virtual Private

  servers or something like that to ramai

  apps and it's a pretty big number but

  the actual load of that of actually

  doing that is pretty straightforward oh

  yeah and I mean and things have gotten

  so cheap over the years like I now pay

  for all of overcast which is something

  like something like 11 VPS instances on

  Linode I now pay for all of them what I

  used to pay for one database server for

  Instapaper back in the day and that

  wasn't that long ago that's like five

  years ago

  so what you can get now with with at the

  VPS level is just incredibly good

  I was the biggest fan of bare metal

  dedicated servers before but now even I

  have converted because you can just you

  know the competition especially between

  Linode and digitalocean has has really

  made incredibly fast Hardware available

  for almost no money and it also makes it

  much easier to learn because you know it

  lets you and we'll get into this in a

  little bit but it lets you you know if

  you can just like in you know in a

  simple command on a web interface so not

  involving buying anything not involving

  emailing anybody or placing any orders

  with anybody just in a web interface you

  can quickly create an instance of a

  server then you can do things like let

  me see how easy is it to set up PHP or

  whatever like you can do things like

  that or you can say you know I wonder

  how whether whether this new district

  district of linux makes more sense to me

  or or I wonder if my software will run

  on this other distro or this other web

  server or whatever you can just spin up

  a new instance in minutes and you know

  run your install script and and you know

  try poke around try things out see what

  works you would you know see what breaks

  and if if at any point you decide this

  wasn't for you or this didn't go well

  you can just delete the instance and

  you've paid up to that point maybe 50

  cents like it's it's crazy how easy this

  is to to both administer servers and to

  learn how to administer servers now in

  the world of VPS yeah because it's

  lovely think of most of these are

  charged by hour of use or something like

  that and so you can they're just kind of

  like utility in that way they

  you can just spin them up when you want

  him but take him down when you don't

  want them anymore and like it's very

  straightforward and easy to do in a way

  that is kind of scary the first time you

  do it that you just like hit a button

  and it's like oh here's a machine go

  ahead and ssh into it and get started

  yeah I mean it's so easy that like you

  know one way you could have done it

  before is by running something like

  VMware fusion or parallels on your

  computer and virtualizing little Linux

  setups on your computer and doing that

  but now it's so cheap and easy to go do

  it on linen or digital ocean that it's

  actually easier to do that than it is to

  do it on your local computer like now I

  do all my experimentation on VPS I don't

  even have our tools agent software

  installed anymore on my Mac yeah and I'm

  the same way like the only thing I used

  my Mac for when I'm doing this kind of

  thing is like the first level of the

  nice thing about using listen like Linux

  is that it's you know these are the

  basic underpinning is underneath our

  Macs are like it's you know it's some

  kind of UNIX space thing I couldn't tell

  you all of the history of it but

  functionally like your Mac is running

  the same basic stack kernel the types of

  things that your ultimate server is

  gonna run and so like I use my Mac

  because I can learn and kind of

  prototype and try some of the things

  that ultimately I'm gonna run on my

  linic I might like main so server

  locally like I have a development

  database that runs on my on my machine I

  have a web have a web server that runs

  on my local machine which makes

  prototyping really quick but as soon as

  I want to do anything that's going to be

  simulating my end environment I just

  yeah I just spin up an instance and run

  it and I have a couple of just instances

  that you're just bringing up you're like

  try it out if it works great if it

  doesn't you know you can just delete it

  and start again so wait once you've

  decided you're going to build and build

  in a Linux instance and get started like

  I'm the actual process is pretty

  straightforward she'll go to a virtual

  private server provider and just sort of

  you'll be set up your instance it'll

  give you some credentials and an IP

  address and a few things that you'll

  have for like how to log into it and

  then you just log in and get started and

  for most applications you're going to

  install at least two things on that

  you're going to install a web server and

  a database of some kind and starting out

  you could probably put them both on the

  same machine

  eventually you'll probably do a thing

  where you'll split the web database onto

  its own machine and put a one or more

  web servers in front of it but if you

  know the very least you're going to

  install some kind of web server

  environment some kind of database the

  web server that you choose it's entirely

  up to you like I use Ruby on Rails for

  all my backends I think all of your

  ziering PHP yeah I mean and I have many

  opinions on like how you choose these

  tools I will summarize them for the sake

  of this being a short show I will

  summarize them as what you want most of

  all when choosing server software if you

  don't want to be administering and

  tweaking with your server constantly

  what you want is old boring and popular

  those three things old boring and

  popular new and trendy does not

  necessarily always mean better in this

  case you know especially like what what

  makes some people think that the new

  stuff is better is vague claims of it

  being quote scalable and beware of any

  claims of things being scalable because

  in reality scaling web services up not

  only is it the kind of thing that you

  will probably never need for almost any

  project that you do but when you do need

  it it is way more complicated and there

  is no silver bullet for scaling there is

  nothing that automatically scales unless

  it also automatically scales on for all

  of your money away but there there is

  nothing that automatically and easily

  scales scaling is always complicated and

  always tricky it's just you know the new

  things that claim scalability just scale

  differently and it the challenges are

  just different challenges the newer

  something is the trendy or new or

  something is that means you know the

  fewer people have experience with it and

  and the the fewer large installations it

  has and so that means there's less

  knowledge in the world about how it

  fails or what the anti-patterns

  of it might be so as you go a lower

  level in the stack being this kind of

  conservative preferring things that are

  old boring and popular it becomes more

  important so like if you want to use

  like a brand new cutting-edge CSS

  library go for it doesn't matter really

  you know if it works for you go for it

  your risk is pretty low there but you

  you probably would not want to use

  a beta version of the Linux kernel you

  know it's like as you go down the stack

  you want to be more conservative because

  it's just it's just kind of a bad idea

  to take risks lower down and I include

  things like the database in that you

  know MySQL and Postgres are great

  they're battle-tested they are old

  boring and popular that is what you want

  in your database just like you wouldn't

  want a trendy new kind of breaks on your

  car like you want to time-tested breaks

  that everyone knows works and everyone

  knows that you aren't going to be

  pushing the boundaries of them you know

  you want a really boring database and I

  would say you want a really boring

  language - because let really new

  cutting-edge languages are still in flux

  and so if you want to have this kind of

  like set it and forget it kind of server

  approach you don't want to have to deal

  with a young languages frequent changes

  and frequent needs for being updated you

  know you want to minimize anything that

  will require your attention or that

  might break your app so you want to

  minimize the need for you know backward

  compatible updates and things like that

  or crashes and you know having to

  install little bug fixes constantly for

  the for the language of database you're

  using you want to avoid all of that

  because new really new means untested or

  less tested another thing that you hit

  with new stuff is in the language layer

  new means that you have fewer

  third-party libraries and and generally

  less third-party code and this is a

  problem like I I'm using PHP forever and

  PHP third-party code is mostly garbage

  but there is so much of it out there

  that any any need that I have I can find

  something to do it and it isn't always

  good but I can find something and as

  I've started writing some newer things

  and go and go I like as a language

  conceptually but there because it's such

  a such a younger language and it isn't

  that popular yet and might never become

  popular who knows but because it's it's

  younger and less popular there is very

  little third-party go code for some very

  common things that I keep meeting to do

  and I keep every time I need to write

  something new and go I hit this wall of

  oh I'll just load up the you know

  markdown module and I look and there's

  like three of them that are all kind of


  and you know and I haven't looked in a

  while for markdown so maybe that's a bad

  example but I keep hitting that wall

  whenever I do anything in a new language

  if oh I just want to see how oh how do I

  resize an image oh well you kind of

  can't unless you do this and so it's

  always easier to use old boring and

  popular stuff it may be less interesting

  and if you are the kind of person who

  wants you know cutting-edge languages

  and who find that satisfying then you

  will trade time and maintenance for that

  and so if for the goal of

  low-maintenance and and you know low

  server maintenance I highly suggest that

  you use other parts of your app to get

  the kind of like newness out of your

  system like yeah use the higher-level

  parts or use maybe the iOS side of it if

  if you have an iOS and a web app use

  that side of it to do new cool stuff and

  make the server as boring old and

  popular as possible because the old

  boring popular stuff works it really

  works it hardly ever breaks and if and

  you will never be the biggest person

  using it because long before you know

  even suppose that you build something

  with old boring stuff suppose it gets

  like the size of Etsy right Etsy is huge

  it's not the biggest website on the on

  the internet but it's huge right it gets

  the size of Etsy if you get that big

  will you be the biggest user of anything

  in your stack because once you're the

  biggest user of like your database or

  something then you basically become a

  developer of that database you will hit

  all the bugs you will have to go in and

  make patches yourself to customize

  things that aren't working or that

  aren't working well or aren't working

  right for you you know so like Facebook

  if Facebook rolls out tons of patches

  and things Twitter has rolled out tons

  of patches or things like memcache I

  mean like there's there's all these like

  big companies all these big web

  companies end up having to basically

  partially take over parts of the stack

  that depend on open source wise and make

  their own contributions and fixes

  because they hit tons of bugs you never

  want to be the biggest user of anything

  in your stack and you know when you

  start out of course you won't be the

  biggest user of basically anything but

  you could become the biggest user like

  you'd be surprised you know you hear

  about new trendy things but you'd be

  surprised how few big companies actually

  use them so you never want to be that

  because he was ever anything you want

  Facebook to have fixed every bug in that

  stack years before you ever hit it

  yeah I think we're ready rubber both

  coming at it from the perspective of

  being apt developers like what makes our

  apps cool is not the back end stack

  that's got nothing to do with like the

  thing that I'm gonna be marketing my app

  has like my customer shouldn't care no

  one like it's only it exists for the

  sole purpose of just getting a job done

  like my most of my backends are the most

  ridiculously boring simple applications

  as I can possibly make it I put all the

  much logic and smarts as I can in the

  apps itself like the database is just

  it's mostly just a nice wrapper for a

  database and something that's simple and

  straightforward that I can set up and

  then I never think about again like the

  only time I ever really think about my

  databases is or like my servers is like

  when there's some big security thing

  that I need to make sure was Auto

  updated or I'm adding major new

  functionality to the front end to my

  applications and then I have to go back

  and like you know add something to it

  but otherwise I just wanted as simple

  and basic as can be it's just like a big

  dish just like a big database that I

  just a database that I can put things in

  and then later get them out and as long

  as it can do that I'm good and if

  anything what are you using old boring

  and popular if anything ever goes wrong

  or if you have any questions whatsoever

  or anything comes up you can just search

  the web and you have hundreds of answers

  from the thousands of other people who

  have used that already and have

  documented it like three years earlier I

  think like it's just it's so much easier

  to to make your your your server-side

  stack something that you can easily

  forget about most of the time and you

  know you don't even have to deal with it

  exactly and I guess once you decide on

  which one you want to do it's like it

  the process from there is honestly

  pretty straightforward you pick

  something you install it usually these

  kind of things that are pretty

  straightforward install process like if

  you're you know there's you be some kind

  of version of like go get me my sequel

  go get me Postgres go get me nginx go

  get me apache and the server will go get

  it install it configure it you'll change

  a few configuration files and you'll be

  up and running because it's boring and

  the process that I'm getting it set up

  has been done by thousands of people so


  automated inscripted and the only thing

  that I had mentioned there is one thing

  that I learned early on is once I've set

  up a server the first kind of what the

  first time for I always a document how I

  did it like the steps I did and then I

  will usually destroy what I did the

  first time and do it again

  to make sure I actually know what I'm

  doing to avoid the situation where down

  the road you are trying to like recreate

  a web server and you can never get it to

  work quite right and it turns out it's

  because the first time you did it you

  did this step that you forgot to

  document and so that was like no one

  will approach first little pro tip is

  like the first time you when you think

  you got it right that's great

  now do it again because if you if it

  works the second time then you know you

  actually understand your city or your

  setup and could we'll be able to do it

  multiple times if down the road you need

  to because when you need to do it down

  the road it's usually under some kind of

  duress because you're like wow things

  are going really well I needed another

  web server to keep up with the load like

  that's a good thing but it's also a bit

  of an urgent thing yeah and VPS does

  make this so much easier because not

  only can you like you know you can just

  clone whatsoever you already have if you

  need to but also like you can do things

  like you know automate this with a big

  shell script as you as you figure out

  like this is what I do like whenever I

  have to install you know something new

  or change something I format everything

  I'm doing as a shell script as I'm going

  and then I can then I can rerun it on a

  fresh server from a VPS like you know it

  just what you said that you can kind of

  verify your work and it's it's so much

  easier man it's like doing this and I

  should also say like anything that is

  not really easy to install from a

  package manager like if if suppose

  you're looking at you know I'm looking

  at some PHP library to manipulate an

  image or something you know that this is

  just a weird example and if there's like

  three of them and one of them is in the

  repository that my server comes with

  like the the UM or apt-get repo like one

  of them is in the repo and I can install

  it with what with you know one word on

  the command line and one of them needs

  to be built from source I'm going with

  the one in the repo because like that's

  like anything as a general rule of thumb

  anything that you need to build from

  source or manually download from

  somewhere that is not in your package

  manager try to avoid needing that and

  and everything becomes a lot easier and

  package managers are not flawless you

  know sometimes

  you'll hit some weird conflict but

  that's very rare

  if you especially if you stick with the

  with like the the default repositories

  that come configured with it because

  those are so heavily tested by other

  people and again old boring and popular

  that you might not be able to use the

  newest version of something or or a

  cutting-edge feature of something or or

  an obscure module of something but if

  you can if you can get away with not

  using it you are saving yourself time in

  the future

  exactly we're sponsored this week by our

  friends at a glue the intranet you will

  actually like with a glue you don't have

  to be stuck at your desk to do your work

  you can manage your task list from your

  laptop during a meeting you can share

  status updates from your phone as you're

  leaving a client site you can even

  access the latest version of a file from

  your home whether you're wearing pants

  or pajamas or anything nobody will even

  know these days everything is mobile

  your work should be - now if you've ever

  looked at a corporate intranet you can

  possibly thought whoever designed this

  must truly hate me and everyone I know

  with a glue those days are over

  igloo Internet's are actually places you

  want to be they're surprisingly

  configurable you can even customize it

  and rebrand it to give it the look and

  feel of your team they have group spaces

  role-based access permissions easy to

  use drag-and-drop widget editors so you

  can reorganize the whole platform to fit

  exactly how your teams work now of

  course today Mobile is everywhere igloo

  is great on mobile they are fully

  responsive they work on every device any

  screen size new devices come out they

  work on day one and they have all these

  great integrations with modern services

  that you actually use things like box

  Google Drive Dropbox all of this is

  integrated into one big easy to secure

  platform they do things like 256 bit

  encryption single sign-on Active

  Directory integrations you if you know

  what any of those things are you know

  how safe and secure a glue is so share

  your files with your co-workers for

  everyone to collaborate on do everything

  you want inside your intranet with a

  glue break away from the internet that

  you hate go sign up for a glue right now

  and you can try for free for any team

  with 210 people for as long as you want

  so I put a glue software comm slash

  radar once again that's a glue software

  comm slash radar start your free trial

  today thank you so much to a Lou for

  supporting under the radar and all of

  real afm okay so once you now have this

  the server set up its running your

  systems up and going

  the thing that of course seems worth

  pointing out is some of the lessons

  we've learned about how things typically

  go wrong when they actually you know

  said if you got this deployed you got it

  running hopefully you have something

  running to monitor it I think I use

  Pingdom I don't know what you use but

  you have some kind of call where I also

  use big dumb there you go they're a

  great service it just you give them a

  URL and they will hit it and one thing I

  always like with those kinds of URLs is

  don't point it to like the root URL of

  your site to make sure that it's up like

  have a special URL that it hits that

  will test your whole stack you know so

  it'll go down to the database make sure

  the database is running it if you have

  like a certain search you know query

  system like it'll go and hit that and

  then report back results so that you're

  actually testing your whole thing but

  you know you have something that's

  testing it every now and then you'll get

  a little message that says oh it's down

  like why is that and usually on the nice

  thing like it doesn't actually happen

  very often like for running of as many

  services I think we both run like

  outages are pretty rare things are

  pretty reliable and stable these days

  like in general like you every now and

  then you'll hit some kind of network

  outage in the data center which you

  can't really do anything about which is

  awesome because you can't do anything

  about it yeah you know you go you go on

  Twitter and you apologize to your users

  and you just wait for the host to fix it

  yeah and that's great it's it's there's

  nothing to be done and that's gonna

  happen no matter where you are to like

  originally I think oh well you know

  maybe I if I was using something else I

  wouldn't have this problem but it's like

  everything has this problem like I've

  been on a lot of hosts before they've

  all had outages yeah like those are just

  built into it but like beyond that

  there's not that many of things that

  happen like usually the issues I run

  into you like the most common thing that

  I'll hit is something like a machine

  will run out of disk space which seems

  kind of basic but it's happened to me

  several times where I turns out I have

  some log file that isn't being rotated

  correctly and it's you know it's just

  like my machine it's completely out of

  space and so some things it'll just like

  start falling over in a really confusing

  way so I just go and check on my servers

  every now and then and just like make

  sure that they have space and the space


  getting smaller and smaller because most

  servers should just be very stable and

  unchanging and once you do that things

  are pretty good and then you run into

  unfortunately the the next most likely

  case of problems is probably what we

  would call human error and unfortunately

  you are the human that is causing the


  so we've when you're administering your

  own servers you have all these little

  opportunities for you to do something

  that ends up breaking your back-end I've

  run a deployment to one of my to my via

  RSS syncing system feed Wrangler or I

  roll the deployment out that dropped the

  main feed item table accidentally that

  was that was an interesting test of how

  good my backup strategy was and took the

  service down for four hours but you know

  that's why you have backups and that's

  why you eventually learned to be very

  very very careful whenever you're

  touching your production databases yes

  yeah database backup is very very


  every VPS provider I know of supports

  snapshotting the machines backup usually

  you have to do something special to

  backup a database because you can't just

  like copy the file and have it be

  consistent but there are tools for every

  popular database to take like snapshot

  backups of them and dump them into files

  or whatever so so look into that for

  your database of choice for MySQL it's

  definitely a extra backup as the one I

  recommend from Percona Postgres probably

  you Postgres people you probably go oh

  just automatically does this for you you

  don't have to even think about it cuz

  Postgres is amazing in MySQL you know

  bad but right is that is that roughly

  correct not exactly I used a thing

  called Wally as my backup system which

  is yeah like the nice thing about a most

  good backup systems to with databases is

  they can often allow you to do point in

  time recovery mmm which is really

  helpful for that kind of a human error

  problem where you issued a particular

  command at a particular time and that

  destroyed all your data so like even if

  you had like your backup if you had a

  snapshot right afterwards like wouldn't

  you know it's like you need to recreate

  the database you need to go back in time

  and like undo queries and so if you have

  an a system where that uses point in

  time recovery which I don't like while

  he does and I'm sure there's

  sequal things to do the same thing like

  you can kind of replay what your

  database was doing and recreate it like

  just before you were an idiot yeah bitch

  binary logging in my school yeah which

  is awesome being able to do that is a

  tremendously helpful thing hopefully you

  never use it but you know I think given

  that we've both done it or done things

  like that it probably happened at some

  point and you just want to be prepared

  oh yeah one thing also uh you know

  earlier research about automating as

  much as possible with the set up as much

  as you can automate the the health of

  your servers like you know Pingdom

  monitoring for the overall uptime of

  your app is great one thing I do to

  prevent like your disk space issue is I

  have to Pingdom alerts for any surface I

  run one of them is is it up and that

  one's on high alert if anything goes

  down it that would like the main site is

  down then everything of mine beeps and

  vibrates like everything alerts me but I

  also have have a lower class one usually

  called stats or vitals where that hits a

  special URL which checks all the servers

  for things like low disk space and very

  high CPU load and things like that and

  it it gives Pingdom a down result if

  anything like that is out of whack and

  then it's logged in a place that I can

  see like what caused this so I can see

  Oh at you know at noon every day this

  server has a really high load or I can

  see oh this server right here has low

  disk space right now so I'm never

  surprised by things like low disk space

  or very high load because I have that

  automated and being automatically

  checked by another Pingdom check yeah

  that's a great idea like it's it's it's

  those types of things that you're just

  trying to think of ways to anticipate

  the problem before it happens yeah I

  mean even like a low-tech version of

  that which I've done on some servers is

  you just put a cron job that runs every

  five minutes that emails you of disk

  space is too low yeah worst case

  scenario you wake up to a pile of 40

  emails okay you go you go and fix it

  like so it's yeah it's great there's so

  many easy ways to do this kind of

  automated maintenance of servers now

  it's running a server is really not as

  hard as you might think if you've never

  done it before yeah and I really think

  that if you can't if the things that

  we're talking about this week sounds

  scary like that's an important thing to

  get over and get used to and comfortable


  because as a developer it's just like it

  feels like it's like in some ways like a

  very basic skill that if you can't do

  you're kind of holding both your your

  work back in terms of the things that

  the the level of complexity of other

  projects you want you can take on as

  well as your career in terms of like the

  things that you're able to do and the

  things that you know the jobs you could

  potentially get and those kinds of

  things like these are useful scales that

  don't go out of date if you know at a

  basic level like how to administer a

  server that's always going to be useful

  and interesting like I don't think

  that's going away like as much as new

  high level abstractions and things up

  here like that's cool and all but like

  UNIX has been around for a very long


  for very good reasons all right we're

  out of time this week we'll probably

  cover more of these topics in the future

  or expand on them so keep listening if

  you want more of this or anything else

  for that matter because we are great and

  you can keep listening to our show

  thanks a lot for listening everybody and

  we'll talk to you next week