Creator of Home Assistant: Web Components, Self Hosting and Home Hacking

Welcome to Syntax. And, I'm sorry. I didn't ask you how you pronounce it. How do you pronounce your name? Paulus Schoutzen.

Awesome. Are are you Dutch? Yeah.

Beautiful. I'm Dutch as well. I'm obviously Canadian, but, Dutch heritage. So always love love when you have it on. So Paulus is the creator of Home Assistant, which is the I don't know. How would you can you give us the pitch of what Home Assistant is and and who we are?

Yeah. So I'm Paulus Schoutze. Home Assistant is an open source smart home platform with a focus on local control and privacy.

You can run it fully at home the way you want it. At its core, it's a Python application, but we prefer you use a Docker container or, like, our operating system. And it's the brains of your smart Node. So it pulls in all the data from wherever you host it. If you have Wes thermostat, we pull it from the cloud. If you have Philips Hue light bulbs, we talk to it locally. We bring everything together.

And from there, you can do whatever you want. Right? We believe that users that buy Vercel, that collect data in your house, you own that data. You can do with it what you want. So from Node Assistant, you can have dashboarding, automation, history, Vercel, but we also allow it to send it to Apple Node, send it to Google Home, to Amazon. If you like those ecosystems, just go there. Our open API also like these things like Node RED, self hosted automation engine. It just fully plugs into Home Assistant, can do anything you can do inside Home Assistant automation, put a different approach. And it's great. Like, everything is just possible. It's just the foundation of your smart home, really. Yeah. It's it's honestly awesome because

like, I've I've run it for probably five or six years now.

I'm currently running it on the the Node Assistant operating system, which is, like, one step even further behind. Like, I'm running it as a VM, not just as a Docker container, as an actual VM. And it's awesome because, like, I have so much gear out there, and there I don't think there's one I have some stuff. I have Google Home and, like, not everything integrates with it, and, like, it's it's it doesn't I can't architect it as as what I want. You Node? And the Apple stuff, you gotta, like, be in Apple's good blessings to ESLint that environment, but, like, you can connect anything to anything on Node Assistant. And I just have, and it's just it's rock solid too, man. Like, I've been running it for forever, and it just works. I think

what happens there is that there's just because anyone can build anything. Right? We don't limit. Like, anyone can build any integration. We don't need partnerships. So so for example, you know, I have Elgato Lights. Now somebody had Elgato Lights and built an integration. And now I have a local Elgato integration to control my lights. So for example, I can have the lights turn on when I start streaming.

And these kind of things is like it's not a use case of Google Home to be like studio automation, but, like, it's kind of these things we end up doing all the time, like, as content creators, for example. Mhmm. Great. With Home Assistant, you can do it. I have dipped into it, like Node not dipped into it. I've been running it quite a bit. And even, like like, all the lights behind me, let me see if I can

go to my Home Assistant UI here and and turn on my recording scene.

Which one I or do what do I have here? It's so funny because I almost never use the Node assistant UI.

It's just automated.

I walk into my office at 9AM, and they're all on. And I walk out at 05:00, and they they turn themselves on and and lamp o'clock. And it's it's pretty cool. But and as much of, like, a home automation geek as as I am, I also today, we wanna talk about, like, a couple things. Specifically, Home Assistant runs on web components.

And I think that was a

yeah.

And, also, Home Assistant is a I I know CJ is gonna be excited about this as well. It's like Home Assistant JS probably one of the one of the largest self hosted projects everywhere, and and making something that is easy, but also people can host themselves is is certainly a challenge.

So the reason I chose web components initially, right, so not the reason why I stuck with it, but the reason I chose it is because at my work, while I was working, we were very early adopter of React. I went to the first React conference. This was back when you had React create class, right, before the components even, before the hooks. Right? Like, this is way back. Oh, man. Yeah. And I was like, I wanna do something else at home. Right? And I was like, homosysm was my playground.

And then Google released material design with Polymer. Right? Polymer Vercel, like, OG web components before they were part of browsers and web standards.

And I was like, oh, yeah. Let me try this. I like material design. And so I added that to Node Assistant, and this was this was rough. Like, this was with non standardized HTML imports. So they had to they had their own bundlers that didn't fully work. They whenever everybody was transitioning with Babel to, like, e s six or, like, five to six JS it was still called back then. And, like, started using classes in Polymer, couldn't do that yet, etcetera. And so it Wes kind of rough in the beginning, but, you know, it stuck around. And I think today, I wouldn't wanna pick any other technology for Node assistant because of what it enables us.

And I don't know. Do you want me to just go down what what it enables? Like

Yeah. Yeah. I I think that would would be a a good use case. Maybe we'll explain real quick, Lee, for for anyone listening JS, like, Node Assistant is a, like, a Python application, and the UI to interact with that is is built in the browser. Right? And that includes everything from simple light toggle on and off to to maps to graphs of where your electric car is to you can you can put a calendar in there. Like, literally anything you could possibly think of.

But then you also wanna allow people to skin it and theme it and have, like, a, like, a flexibility and control over making it. Like, there's this there's a a subreddit that's just dedicated to people who make, like, like, home screens, like a like a old TV they hang on the warp, and it shows what's coming up. And I I'm always curious about, like, how do you how do you do that in a way that you give people full control, but it still looks good at the same time?

Yeah. The the trick there really is Shadow DOM. With Node Assistant, we have made sure every layer is customizable so you can people can define their own cards from their dashboard, and we just push the state in as like a property, and then they can render whatever card they want.

And by leveraging Shadow DOM, Wes really allows us to encapsulate the CSS because there's a lot of people out there that they don't you know, they learn to program. Right? Like, they learn programming or web development as part of their Node assistant journey in building out a dashboard, And they do wild stuff. They do very wild stuff. But being able to just say, okay. Write whatever CSS you want. Like, make the background of every div red. Great. Because in Shadow DOM, it doesn't touch the rest of the application.

It also allows us to because there's two parts to it. One is it should work today, but, actually, it should work in a year from now. Right? Like, if your environment around it changes, like the CSS that's being used or the components that you rely on, then you end up, like, breaking a card. But if a card is fully encapsulated with its own stuff, which, yes, it brings in more CSS, it also guarantees kind of stability. And so the cards inside Node assistant that are built in, they share a lot of the stuff together.

Custom cards, they don't. They just have their own little world shadow DOM. They can mess with it. They can write all the CSS, the JavaScript.

The state, that one has to stay fixed. Right? So the way we pass around state and Home Assistant is actually, this because this is before, like, this is really old as well. I know we were using nuclear. Js.

Nuclear. Js was the the state management that was the inspiration for Redux.

Right? So that's old.

And instead of using objects for Redux ESLint like we're doing just objects, nuclear was using immutable JS. Okay. So you had to pull in immutable JS, but immutable JS Wes, like, maintained by this guy at Facebook. And he left Facebook, and immutable JS kind of died, and nuclear JS died. And then I kinda realized, like, it's too much magic. Like, if you do all this magic, if you wanna have people create custom cards, you can't have them pull in reducers or whatever, selectors, all that stuff. So instead, I basically created, like, the idea of Redux, right, where you just have one object with properties and the properties only change if, like, a value changes. Mhmm. And we just pass this object around. Wes just push it in. And if it doesn't change, like, the actual data you're interested, you don't rerender. Otherwise, you render.

And this piece of state management actually has survived, like, I don't know, eight years, which is kinda wild, but it works. And today, custom cards from back in the days actually still work. Like, if you had a card to render a card or a switch or whatever, as long as the things you're into like, the if the services, they still work as well. So all that stuff remains stable, those cards can still function.

I think this is great. Like, I I saw your blog post back from 2016

about how you chose Polymer and web components. And the fact that you made a technical decision, well, like, nine years ago at this point, and it's lasted, which is hard to say about like, because, I mean, if if you would have chosen React, you would have had, like Three or three. And you wanted to say relevant and update. You'd have to you'd have to, like, rewrite three times. So that's awesome to hear that it that it's working. That's something that I actually already realized when we were working at React, at a company I was working, is that a new version would come out. And back then, we were making a lot all these changes. Right? Because Wes twenty fifteen was just coined and, like, classes and everything was new and everything had to be old. Like, you have browser refine, all that stuff. It's way less settled back then than it is now. But then a new component which would require a new React, and you would have to upgrade everything and kind of go along. And I realized that with today, with web components, we don't have that problem. Like, we still have some Polymer in Node Assistant. Right? Like, we've been there's been a new material Wes component package which they also now put in maintenance mode. So I'm we're not gonna bet on Google UI toolkits anymore.

But we are, we still have some Polymer. Like, this stuff, even though it's not maintained, it still works. And it's still because it interacts at the browser level with DOM elements. So the sidebar element in Node system with gestures, that's all still Polymer.

But in it are just modern web components with the newer frameworks and all these things, and it just keeps working. The beautiful thing about

browser standards is you they're not gonna go away.

I feel that way even in my courses as Wes. So if I do a CSS I have a Flexbox course from, I don't know, eight six, eight years ago, something like that, and it's still a % valid.

It's amazing that, like, you build something in in browser tech, you know that it will it will stand the test of time. And that that's really important because, like Yeah. I I logged in to my Node assistant a couple of months ago, and I hadn't updated the thing in in two years. I took a backup. I installed a brand new version and reinstated it. Everything just worked. And I was like, oh, man. Like Nice. That's beautiful.

Yeah. I I think browser vendors, they are the people that make the standards. They take backwards compatibility very serious. Right? They even going as far as, like, I remember the whole discussion about, like, flattening an array.

Right? And where it Wes, like, I think MooTools had used one function that they wanted to use, and so they didn't wanna break it. And so they have now an array dot flatten function.

I mean, that's really, like you don't see that in other in other platforms. Right? Like, even Node Assistant is not that stable.

So what are you using now? If I inspect element on my Home Assistant dashboard, I see a whole bunch of, like, pretty much no HTML elements. You know? I see I see all web components all the way down. You know? Home assistant tag, home assistant main tag, home assistant drawer, partial panel resolver, and then there's shadow shadow roots all the way down, which anyone who's listening, web components a shadow DOM, like you said, is it it self contains the styles, meaning that there's no way for CSS that's written inside of a shadow DOM to leak outside and accidentally goof something up, or you share a class name between two elements.

And the other way around too. Right? The other way around too. So CSS from outside doesn't leak into the Shadow DOM except for CSS properties. Oh, yes. It That's Is that ever a problem? Actually, I'll I'll ask that question now. Is that ever a problem where

sometimes you do want CSS to make its way into the card?

So the way you do that is that you create a slot. Right? So you create a slotted element, and then you can transform an HTML from the parent element inside the child element.

It it feels a little wonky, to be honest. Like, there's definitely, like you know, we access this idea of portals. Right? Where, like, very deep in, you just create a dialogue and boom, like, somewhere else in the page it pops up. We've recreated that by just creating an element and adding it to the body. You know? We just that's our portal. And then we just render in there. And at the end of the day, we have at the end of the dialogue, we remove that from the from the element.

But we also use a lot of events to kind of hack around that. Mhmm.

Because events are very powerful as well. Oh, so you have like a like a pub sub or or what? So we have, for example, a start dialogue ESLint. So you can register like, every dialogue is really just a custom element. Right? Like a web component. So we have at the root, we have a bunch of listeners like a dialogue manager. And so you can Sanity, open this dialogue, and it will then see, okay, this dialogue is in this file. So we dynamically import the file, because we don't put everything in the main bundle, of course. Then we instantiate this element. We add it to the DOM. Web components are nice in that you can just add things to the DOM before they are loaded. And the moment they're loaded, they just pop up. Right? They instantiate like, they read the HTML. They parse it. And so that's how our dialogues work. That's why sometimes if it's like, if the stars don't line up and Homosuggest, you would click a button and you can wait, like, a second, it generally should be cached by the the service worker, of course. But

if it's not, then, like, it can wait a second, but then the dialogue pops up. And it's because of this system where we just add the dialogue and we import it at the same time. That's great. And I'm gonna ask you about the service worker in just a second, but I'm seeing little comments where it looks like there's lit in in here. Is that what you're using for the web components now?

Yeah. Yeah. Yeah. So we used to use Polymer, and then they came up with a Polymer had a lot of things that didn't make it into the web component standard, and it also relied a lot instead of where React would, like, batch changes, Polymer would not batch changes. So if you make set one property first, it would re render for that, and then you set the next property, it will re render for that, which is craziness. Like, it's all of a sudden, every race condition had to be caught because if this order was different so with lit, they changed this. They changed this to way to work more like you would do react. Like you said, it binds properties, and then at the end of the micro task, it rerenders.

And so we use LitElement.

The a lot of stuff from LitElement is now in the standards. Right? Like, so there's style what's it called? Style groups. We have reusable style templates, obviously, set of DOM custom element. And then the way lit kind of they have the lit renderer, and then they have the lit elements.

And lit elements actually just there's this thing called reactive element, which is literally just a custom element that can respond to properties and has some life cycles like did update, will update, update it. And we actually use that part in some of our elements or we use lit elements if we just actually have to render them. But if we do a lot of magic, like our we made our own page router, which is the partial panel resolver, for example, that you saw. It it has, like, a definition of, like, these are the panels that we know. We don't know the internal routing. Every every element that is can contain multiple pages is responsible for its own routing. But at the partial panel resolver, again, really like a dialogue. We just import the file. We create the custom elements. We can cache custom elements if you wanna maintain the state in the DOM. Right? Like, which page was selected or these kind of things. Yeah. Basically, the underlying layer of lit reactive element helps us just manage the life cycle of the elements, reroute, and then we are deeper in the stack. It's all lit element.

Node thing I wanna mention is for anybody listening, go read the source code. Like, I I just started diving in this morning, and I've never first of all, I've never seen so many web components.

And the the the Node is super readable. Like, the I I could I see it as almost like a learning resource. Like, how can you so many people talking about web components Sanity, like, well, it doesn't scale or it doesn't work when you have such a large application.

You've done it, and this is this is a testament to it. And it's open source, so you can go in and see how all of this is written and yeah. It's super cool. Yeah. I'm curious if it's, like, one of the the

biggest implementations of, like, like, open source project of using web components. You know? Everyone always points to, like, well, YouTube and Google uses it, but that's not that's not open source.

I mean, I think a lot of things that we do, we like, we are the most active open source project in the world, and we are huge. Right? Like, 21,000 people contributed in the last year on our core, not on our front end. That's a actually don't know the exact number there. But so anytime we do something, we're often the biggest. Yeah. So, like, the biggest cell phones and the biggest web component user. But it also means that the technologies we choose, they have to be rock solid. Right? It means that we are we can't afford a rewrite with the amount of people that work on the project. You can't, like, do something. So for example, we are considering adopting shoelace.

Shoelace got bought by the awesome Wes Awesome group of it's called I know the company behind Font Awesome. Web Awesome.

Yes. And they created Web Awesome, and it was like a Kickstarter. And so they are making open source components with, like, a paid component to it. We're we're gonna use the open source parts. We cannot use paid stuff because we're distributing everything open source.

But that's that's gonna be our next iteration that we are gonna rewrite a bunch of our stuff. That's great. Yeah. We had, it's Connor Rogers and Corey Levisco. We had them on the podcast, Syntax. Defam forward slash seven fifty eight.

Everybody's really excited about this project because it's going to be a whole big set of UI components that are built in in Wes components, meaning that meaning that you can use them in React. You can use them in Angular, but or you can also just use them straight away in in some straight up HTML.

So that's gonna be

I'm really excited for this project. And Scott's Scott's been working with it as well. Yeah. And it's it's gonna be neat. I think that we have been relying a lot on the Google web components. Right? But then Google just you know, they are kind of reorganizing. And let and Angular kind of I know what they're doing, but like the the material design components are lit, but then they they're betting a lot on Angular because they have this other internal web framework that's gotta be merged into Angular or something complicated. And they're just shuffling all these resource around. And I think it's time that we, you know, we bet on open source for these kind of things Wes with the company behind it, though, because I think that is important so that the longevity is guaranteed, like, just an open source project without corporate backing. Like, there needs to be a way to fund maintainers because maintainers need to live and eat as well. And if you don't pay them and somebody else is paying them for something else, they'll they're gonna do something else. Right? At some point, like, there's not enough attention or burnout or they get kids or they move to a different city and get different hobbies. Anything can happen. Right? So the fact that Shoelace or Web Awesome has, like, a company behind it that you can give money, right, to keep maintaining this stuff Mhmm. I think JS awesome.

The native app, is that a a web view? Is it native with bridges? Or how does that work? It's all web view. So we Wes

basically, we don't wanna build our dashboards in native because right now Wes have a big community building custom cards. And so if you have a cool custom card in your dashboard, you would open the app on your phone. It's like, it doesn't work. Like, that's lame.

So we did build a bridge. We call it the external bus. And basically, it's an abstraction of certain features are being offered by the Wes view. So if you launch the UI with a I think it's called external auth equals one, it will instead of looking locally in the local storage for the refresh token and the access token, it will reach out to the mobile app because the mobile app has its own access and refresh token, and that one will then take on the whole auth lifecycle. So whenever the access token expires, the native app will make the request to get a refresh token. And when we had this communication in place, we then now have the mobile app expose certain things in, like, a configuration object like, hey. What else are you capable of? Because people update their apps in different, like, frequencies. Right? So maybe somebody updates their Android apps once every six months or somebody updates every day. So we need to support kind of, like, incrementally all these features.

Plus the app is, of course, used on the desktop, so we have a fallback anyway. But for example, features that we have is open the app settings. So we didn't wanna have a toolbar to open the app settings. So if the app cannot connect to the UI, it will show a button like, hey. Do you wanna open the native app settings and do some I see. Checking with your Vercel? But we inject inside the if you go to your settings page at the bottom somewhere, there's like app companion. You click on it, a native page opens with settings.

We have the QR code scanner. So you wanna the QR code scanner just works a lot better natively, especially this is this is actually a big problem that we run into. If you locally go to a server, you don't have HPS.

And so that means you don't have microphone, you don't have camera.

All the cool stuff the web has to offer, at least part of it, is limited. Right? So we've been building those things natively. So QR code scanner, we have natively. We have, our voice assistant natively now as Wes. Because, yeah, that way it works. And look, I've been trying, like, talking I think there's a somewhere internal proposal at Mozilla about, like, how to make this possible. But, you know, Google and Apple, they never care about the secure context as it's called in browsers to allow that for local because it's just not a problem they have. Right? It's a problem that we have. It's a problem that Plex has been able to solve, but they do it in a weird way where they have a public DNS that with a certificate that you have locally, and it points at your local IP. So only internally you can go to it, but it still requires the Internet to exist and to be accessible.

I always wonder how they do that. There's there's a couple not Jellyfin. What is this? Streamio, I've been has something similar to that. But, yeah, I'm always curious how they are able to do that because that not secure bang is is kind of scary for people, and and it kinda sucks that like, I run mine at homeassistant.local JS it it has a MDNS on my local network,

but it's not a secure instance. Local host is, and you can run, like, a camera on all those limited APIs, but not not on the other ones. Are you doing anything for code sharing between Android and iOS? Because, like, typically, if I were building something like this, I'd use, like, capacitor, but I'm seeing there's, like, two separate repos for Android and iOS. Are you using anything? Node so we we decided not to do it

because the parts that are UI facing are all Wes view. Right? So the things that are Scott natively Android or iOS are the Android Auto, CarPlay, the watches, the ESLint. It's it's such small pieces that code sharing there doesn't really help that much. I see. And we try to push all the logic into Node Assistant. Right? That's where we you know, a QR code is just a bunch of bytes and let the back end figure out what to do with it, for example. That makes sense. Let's talk

about the self hosting aspect of of everything. Because if you want to install Home Assistant on your

local computer or you have, like, maybe a home server, I have a Synology that I run it on. What are my options, and what's the process for getting Home Assistant running? So there's different options. And, like, this is open source. Right? So there's always a limited options, but we had to actually scope down the options a little bit to just get maintenance because we get under control because we would get all these bugs and, like, oh, this is a bug broken if I run Node system Docker in, like, a jail on FreeBSD or something. We're like, yeah. But we don't really care about it.

So there's four options. The the easiest one, home assistant operating system that is using Buildroot. It JS a custom built operating system with the smallest amount of footprint just to run Node, really. That's the key. It has two partitions. So if you do an operating system upgrade and it fails, it will automatically roll back. We run everything in Docker. So we have the supervisor. The supervisor is in Python application that manages the system. So it talks to the operating system and say, hey. Can you update this? Can you start this? Can you stop that? There's a I know there's a microphone plugged in. It will forward the events to Node Assistant. Then Home Assistant runs as a container.

We have an audio container, a Wes container. So we have some bit of, like, helper functions around it, all running in separate Docker containers.

Now if you don't want any of that, you can just run Node Assistant container individually.

So that is just in if you run Docker.

Now the reason we always recommend to run Node Assistant operating system is because not everything in the world is written in Python, sadly. So the Z Wave driver Thankfully. The Z Wave driver is written in TypeScript, for example. So but it's the best Node, and it's the one that we're funding now. So if people wanna run Z Wave, which is a standard for the Scott home, you have like a radio dongle, you plug into your smart home hub, into Home Assistant, it detects it. It will actually install Docker container with Z Wave TS for you transparently right behind the scenes. You're not doing anything. It will install USB stick, connect Node Assistant to it over WebSocket connection, and now Home Assistant can speak Z Wave. Now this is great, but this in Node Assistant OS because we control the whole stack, because we control the docker host, we can do all this magic. If you run Node Assistant in a container, which is totally valid way to run it, you would have to set up Z Wave JS yourself in your own container, and then you have to update it yourself.

Yeah. I I hit that. I'll tell my example is I was running a I have all these WyzeCams, because WyzeCams are super cheap, but it's because they want you to subscribe to their their monthly plan. Right? That's that's the the thing. I'm I'm sure you've got opinions on that.

So I was like, well, screw that. I'm just gonna stream these suckers twenty four seven.

So I installed this, like, Wyze Bridge, which essentially just streams twenty four seven, I think, off of their Amazon server. I don't wanna see the bill. And then it it pipes it into, like, a RTSP stream, and then I can pipe those into my Node Assistant.

And instead of me running that Docker container manually, when you're running Home Assistant OS, which is like a like a VM, it can then go and and start many other Docker containers for you, that plug in being one of them. Yeah. No. Exactly. Well, I've I've I've tried setting that up too, Wes. And I think if you're on your local network, it actually sends the the video signal over local. But if you're accessing remotely, it goes through WISE servers. Oh, okay. Yeah. Oh, that's good. So I'm not not racking up their server bill.

No. But that that was one of the use cases that I had too because I have, like, all these devices from various things. And so I I set up Node Assistant on a Raspberry Pi. I use the the Home OS. Nice. I would recommend, like, if you're a dev that bought a Raspberry Pi and didn't know what to do with it, it's just sitting in a drawer, you can throw Home Assistant on it and start connecting devices

to it. And if you want to see all of the errors in your application, you'll want to check out Sentry at century.i0/syntax.

You don't want a production application out there that, well, you have no visibility into in case something is blowing up, and you might not even know it. So head on to century.i0/syntax.

Again, we've been using this tool for a long time, and it totally rules. Alright.

Actually, you know, fifty fifty one or 52% of our community runs on Raspberry Pi devices. So Really? Yeah. Because it's just a it's such a gateway drug. Right? Like, oh, you have this Raspberry Pi. A lot of people buy these computers because it's like, oh, this is 35, 40 5, 50 5. They're getting slightly more expensive nowadays. But you buy this device, they sit in a drawer, like, yeah. And then what are you gonna do with it? And at some point, you learn about home assistant, and then, oh, I'm gonna install it. To go back to our installation method, so we have the operating system, which you could run as a virtual machine, which is what the Wes is doing. If you installed on a Raspberry Pi like CJ did, then it's just that. Right? Like, it just up it can update itself everything to the UI. There's no command line.

There is this hacky in between path, which is called home assistant supervise, which I do not recommend. And we see the users also that are using this lowering and lowering. And this is where you basically run the supervisor on a Debian Linux installation, and then it kind of basically, you have full access to the host operating system to SSH to run other applications and all that stuff, and the home system does its thing. And eventually, things will always Scott conflict. Right? Like, people install Portainer to do kind of cool stuff with Docker, but then Node system expects exclusive Docker access and Portainer starts messing with Node system containers and everybody's angry at each other.

So supervised, We we always say, like, computers are so affordable today. Like, get, like, a Raspberry Pi. Get, like or get a place that runs VMs, run Node assistant. And then the last way you can run home assistant is just as Python application. You can even today, just pnpm install home Assistant. Boom.

And it just works. I've never successfully pip installed anything, but

Oh, you should so the the Python world is rust getting their rustification going, and there's this startup called Astral. They build UV.

And, like, the installation of Python packages got so fast that we save hundred eighty five days a month with Node Assistant CI just getting it. Like Really? Yeah.

Just because of that. Sanity environments, all that stuff. So that way, you can actually install Home Assistant successfully.

Oh, that's great. That's that's really impressive. Do you do you have to be really picky about what gets added, both in terms of, like, bundle size, but also in terms of, like like, memory consumption? Because you're running on these tiny little computers rather $70 computer.

Not really. I think the way because the way Node Assistant works is that it's set up very modular. So we can preinstall all these dependencies, but everything get imported on the fly only as you use it. So if you, for example, if you have a Node too many smart home systems, you could run Home Assistant on a Raspberry Pi with one gigabyte of RAM. But my colleague tried the other day, and he has, like, 15 Sonos devices and Wes cache Vercel, and then he blew through his memory. Right? So it's depending on how many device you have and how many different integrations get pulled in, you need more memory. We sell Node Assistant green, which is just a because people were like, oh, VM. I don't know what it is. Raspberry Pi. I don't wanna, like, do that. So we we started just selling a straight up Linux Bos, with Node Assistant preinstalled called Home Assistant green.

And it has four gigabytes of RAM. We're just like, we're just gonna put four gigs in. Nobody will ever have issues with RAM. CPU usage is like 5%. Like, your smart home, not that much is happening. Right? We all think we're super cool, but, like, you know, you get, like, a temperature report is, like, every five or ten seconds. You know? How how often do you turn on your light bulb? Right? Like, eight times a day maybe? Like, those events, they just don't really add up to a lot of CPU. Now if you start doing video streaming, video transcoding, these kind of things, it starts to add up, but the normal user cameras can be normal, but, like, if you don't have the transcode, which is the normal operation, then there's just not a lot of CPU. Oh, man.

And what about all the AI and and voice assistant stuff? You guys are in that space. Right? Oh, yes. I love it. It's so cool. Two years ago, and just really before the AI hype started, we realized that if you look at, like, the big smart home platforms like our competition, it will be Google Home, Apple Home, Amazon Alexa, and what they do better than us. Right? Because they they don't do such well on dashboards or integrations or history or automations, but they did voice. And our users were like, well, now I still need to go to Apple or Google to do voice.

And so we set off a journey in the beginning of 02/2023 saying we're gonna do our Yarn of the voice, which was a bit optimistic. It took two Yarn. But the goal was we want users to allow to control them to using their own language, control their smart home. And so not any of the AI stuff was just control your lights, really.

And as we were building this, all of a sudden, OpenAI drops whisper. Right? And all of a sudden, TPG 3.5 comes out, and all of a sudden, meta drops llama. And so now we're building this voice tech, which everything is always pluggable in Node Assistant. So different speech to text engine, text to speech, the the brains itself.

And so by the end of this year last year, we actually released this hardware. It's the the voice preview edition, which is just a tiny box that you can use to control your smart home. And inside Node assistant, you can either send it to AI, have, like, chat GPT or GPT four control your home, or you can do everything fully local. We're actually tomorrow, I guess I don't know sure when this airs, but tomorrow, Wednesday, February 12, we're doing a year of the voice chapter nine stream, which we've been keep doing them basically for two years now Wes we're launching a new speech to text engine called speech to phrase that runs on the Raspberry Pi four within one second, which was like speed was the most important thing that Whisper could not offer us.

And it's super accurate, but it's limited. So it knows turn on the lights. It's trained on your areas and device names, and we are able to do speech for home control fully local.

And then if you wanna do AI, you either need bigger hardware or you need, like, a cloud API to do, like, the speech to text part. Yeah. And then you can do all the AI stuff, and it's so much fun. It honestly

has it's crazy. Like, I've got Google Homes in our house for for the voice stuff for the exact reasons that you just said.

And it's amazing how it's gone from this is amazing to this sucks.

Like like, it's it's not good at all. And, like, you ask it a question, like and it Wes like, I don't know, but here's a website. It'll just, like, throw it up on you. And it's so funny how our my perception of it has changed, being like, man, if this just if I could just pipe this thing into my OpenAI API and spend 12¢ a day changing my commands, 12¢ probably even even a lot if you're considering.

So I'm I think I wanna get this. This seems awesome. This is I mean, my kids love it. So my I have one here that is, like, looks like Pokemon. Like, I just had a somebody three d printed this with, like, all the case files. Everything is open source about this. The hardware, the software stack, everything.

And my son, you know, the press the middle button, he can talk to it. And it has conversation support in that, like, if if you follow-up, it knows the context of the previous things you've said because we just Scott the whole history to the large language model.

And he has whole discussions with GPT before about, like, Pokemon or Minecraft or he has all these questions about games, and then he's reading the Mario encyclopedia, and then he's, like, questioning things. And, like, the AI has an answer to everything. If it's correct or not, it doesn't matter. It's like we're talking Minecraft. Right? So it's really good. That's great. For me, that's one of the killer apps too. Like, I've like, I'm very privacy

conscious, and just the idea that, like, every voice app you're interacting with is going to some cloud, to some company. So to be able to self host it and and have it in your home Wes pretty cool. I actually have, like, this this voice kit that I think Google released it a while back. Yeah. Yeah. Yeah. But it has, like, a Raspberry Pi in it. I think it's defunct now, but they had, like, this I I don't know. They branched off to another company. But this is one of the first of, like, trying to do, like, voice locally. So, yeah, that that sounds super exciting.

Now the so the the the reason, by the way, why this one is so much better is because it has this XMOS chip. And XMOS is this British company that's making chips for audio cleanup for speakerphones forever. Right? They're just really good in capturing voice.

And so you put that in this voice assistant, you basically get all the r and d where Google and Amazon has spent billions on. Well, you just get it for, like, a dollar and a half or something. You put it in your device, and we we're at the same level.

So my mind was blown last week when after we had booked this, I was just doing some hacking on some some LED stuff, and I've got so the anyone watching the video, I've got a Wes 32 here. And I was talking to Scott. I was like I was like, you gotta get one of these, man. I've I've got so many little projects running on ESP 30 twos. And I was like, and you can flash it in the freaking browser.

And then, like like, half an hour later, I was just, like, flashing it in the browser, and I was like I was like I was like, I'll look into this project a little bit more. And it was you. You made this project.

I didn't even realize.

So Yeah. You built the with Wes serial, the ability to flash a, what, $5 microcontroller?

Yeah. So this yeah. Yeah. I have I have a few here as Wes, ESP 30 twos. Yeah. So we don't just run Node Assistant. Right? So we are last year, I created a foundation, OpenHome Foundation.

I donated Node assistant to it. We donated ESP Node to it. These are the two user facing projects we have. And ESP Home is a project to use YAML to generate software to run on these microcontrollers.

And so for example, this is a microcontroller that has a microphone speaker. People Yarn listening. I'm holding up an, an m five Stack Atom Ego. It's a $13 device with a microphone and a speaker, which is crazy cheap, and it has a case around it, USB c powered. Wow.

And e s p Node just say, hey. This has a microphone and speaker. Make it a voice assistant. And now it's a voice assistant for home assistant. And it has wake word detection, and you can play music on it. It's not loud.

It's not the best voice capture, but it works. Right? And this is the same the way HP Node is just you write a YAML file, it generates software, it installs it on the device, and then it, configures the Wi Fi. It pops up on the Internet on your local network, I mean. Home Assistant finds it. ESLint integrated.

But what we realized when we were building this is that the first step to get once you have it up and running, Home Assistant Sanity ESPN can do over the air updates. It's awesome.

We realized that the first step is so hard because you need to install the software on this device. And so what people had to do was install Python.

And I Impossible.

Impossible. Yeah. You can't you can't go there.

So then I realized, like, wait. But this is Vercel, so we can just use Wes serial.

So I work with Espressif, which is the company behind the ESP chips. And they already had, like, kind of a prototype Wes we build it out, but they just had, like, a raw library. And so I built this tool called ESP Wes tools, which you basically define a manifest saying, hey. This is, for example, the project WLED, which is one of the projects that uses our tool.

And for these different chips, here are the builds. And so what it does is that it connects to a device over web serial. It asked them, okay, what kind of expressive device are you in? Wes 32 and ESP 32 JS three. There's this is different models with different architecture, different requirements.

Then we automatically select the right build. We install that on the microcontroller.

Then we restart it so it starts the software up. Then we have the second step feature because the second feat second step that you need once you have these devices installed is that it has to connect to your Wi Fi so you can control it. Right? Like, WLED has, like, an embedded web server, for example.

HP Node wants to talk to home assistant.

So the second step is that we created this open protocol called improv Wi Fi, which, by the way, it works over web Bluetooth or web Vercel, both. I have we build SDKs for both. And then we just say, which Wi Fi access ESLint do you see? It sends it back over web serial. We show a UI. Users enter the password.

It sends the password over to the device. Device connects to the network then sends back a URL to follow-up for the user to finish onboarding, which can be the IP address that it just received, for example. And then you're on the web server and you get started. And all of a sudden, we realized, like, okay. From Python installation and super difficult, now it's just two minutes. Right? You plug it in, installation,

Wi Fi. You're on a web server running on a $5 device. Like, you don't you don't even have to, like, download and install, like, an installer and click through it. It's you literally visit a website, and it pops up, and you can flash your thing. I've I've got lots of WLED projects, and, I was doing I was doing another one. I forgot what it was, but it was, like, this Wi Fi improv. And I was like, that was a project. I was like, oh, I was like, oh, this is a a standard project, which makes putting these things online much easier.

So that we build that one too. That's that's amazing.

Yeah. And so, actually, improv Wi Fi was interesting because we initially started doing it with Bluetooth. Yeah. Because the old way used to be that you have an access ESLint, and then your phone connects to the access point. And through DNS hijacking, they serve you a place to enter the Wi Fi username and password.

But the for the device to test out the Wi Fi username and password, they have to tear down their access points so you don't get feedback if it fails. And we're like, Bluetooth can just do this. Like, there's all every native application of every product out there has it. It's just no open standard.

So Wes just build one.

And then we're like, oh, we can do normal Vercel, and then we did that.

That's that's amazing. Yeah. I, I recently did a project where I controlled my Roomba over, Wes 32, and I was I was flashing it. And I was amazed at how far and how accessible the whole, like, robotics microcontroller space has become. Both Node assistant as well as this whole, like, microcontroller is filled with enthusiasts. You know? It's not it's not a lot of full time developers and and and nerds. It's just enthusiasts clicking stuff together and making it work, and it's amazing.

Yeah. I mean, this is for us the reason, like, why do we build ESP Wes tools? Why did we, like, make it as a standalone project and not build it into ESPHome? Is that exactly this. There's a lot of enthusiasts, and they they build stuff for other enthusiasts. Right? And we wanna make sure that we unlock all these cool features like WLED. Right? How easy can we make it? So how can these web based tools, Scott everybody can use WLED. Node everybody can install this Node or all these other crazy projects because that just that last bit, we're like, okay. We have to create a new standard, create a bunch of new libraries.

Somebody had to do the work. That's what we do. Right? To bring all this stuff to Scott just our community, but any anyone that is just tinkering with this. You ever talk to anybody from either Safari or

Firefox about the Wes serial and web Bluetooth APIs? Because I know they're still being developed for Bluetooth LE in Google.

So I have been talking to somebody at Firefox, and, like, they are he's like, well, internally, the road map, like, maybe it's an option. There's actually I don't know. Can I just read out a URL? Because he just asked me literally today. He's like, if you all just go to this page and tell Firefox that you like Okay. Wanting to have this web serial, I I it is too long of a URL. But I'm gonna try Put it in the show notes? Can we put it in the show we'll put it in the show notes. Yeah. We'll we'll put it in the show notes. But, anyway, it's just like place where you can tell Mozilla called Mozilla Connect, where they connect with the community and where you can tell them, but they haven't done so far.

Safari, I think, is a hard no.

They don't like fun stuff that they cannot control. Mhmm. I think well, I mean, the thing that they both said, which I can I can see their perspective, but I also don't like it JS that by the way, I don't know if we said, but maybe two steps back, only Chromium implements Wes serial and web Bluetooth? Right? Safari and Firefox have said Wes don't wanna do this. And their concern is that if you give this permission by accident, which can happen, you don't know what has happened to you. If you give away your location permission by accident, okay, I've accidentally shared my location with this website. If you give notification permission, you get annoying notification, you disable it. If you give away web USB, web Bluetooth, web serial, things can happen. Right? Because these are real world APIs that can do cool stuff, but you don't know what. Like, yeah, you can maybe see the log of what was written, but it's just bits and bytes. Right? Like, there's no you need to be a programmer to understand that. And that's, like, their angle saying that's why we don't want it. I come from the world of, like, just give people treat them like adults. Right? Like, we always try to treat everybody as, like, with these as babies. Like, just don't press these buttons, and that's why you end up with, like, tablets where you cannot program on. Right? Like, you can only go to the App Store.

That's ridiculous.

I would love for it because you think about, like, a new generation of, like, makers and stuff.

They are growing up on on iPads.

And if they're not able to do, like, a low level connect to to something like this, that's that's a missed opportunity, I think.

But I can also see Yeah. Not accidentally like like, I can I can connect to my headphones from web Bluetooth? You know? And and you could probably load a custom firmware on there that would do something bad.

There's this, I cannot find, but there's this Bluetooth, temperature humidity sensors from Xiaomi. They cost, like, $56 per per device, and you can actually over the air, through web Bluetooth, install custom firmware on them. So just you fake an over the air update, and they're like, oh, yeah. A new firmware. And then it actually speaks, BT Node, which is a custom Bluetooth protocol that we build to share IoT data.

And so then, you know, it gets into Home Assistant through our Bluetooth proxies, and then boom, you have your own data.

That's crazy. What? Can I ask your thoughts on this whole matter as well? Because there's there's been, like, Wes? Like, two standards. There's ZigBee and Z Wave.

Z Wave. Yeah.

So disclosure, just I'm on the Z Wave alliance board.

Just I don't I don't think it matters, but, like, just disclosure. So there's the matter standard in a way is great because it's the first smart home standard that works over Wi Fi. So ZigBee and Z Wave both connected their radio standard and their application layer of, like, what is a light together. So you couldn't use it separately. That meant that, for example, your washing machine using Wi Fi couldn't be integrated with a standard.

Matter actually solves this by focusing on IP. Right? So anything that has an IP, they based on IP six, it can work into Matter. So your washing machine can now go into Matter. It's also the first standard that is basically developed where all the big tech got together. They were like, we need a standard that needs to be good. Here's all the things we learned, and they contributed everything. Right? So there's parts of HomeKit in there. Apple just said you can use anything from HomeKit. We don't care. Google said you can use anything from Google Node, and Amazon said you can use anything from Amazon Alexa. And so there's a lot of really good ideas that are built into Matter. For example, every device can be connected up to five smart homes at the same time with their own encrypted connection. So if you use Apple Home and you're like, I wanna try Node Assistant, you don't actually have to tear down your Apple Home. You can use Home Assistant in parallel to it with your meta devices, and they will work with both. Now this is, by the way, theory. It's quite of unstable.

I think the the rollout of Meta is a little bit overhyped, I would say. I, I think that people expect everything from it on day one, but they are slowly adding more device types. And Wes the specification is released, then, of course, smart home platforms have to implement it. And then devices have to be built. Device have to be certified. This is hardware. This moves very slow. Right? Where I think in the JavaScript world, the way we work is that while we're tinkering on a standard, somebody builds a transpiler into, like, into our tooling so we can play with the standard and everybody is already using it, then it gets standardized. And once it's in browsers, in three years, we put it in the modern build of our application.

So Matter, the ideas are great. The execution is not so great yet.

I think we should see the benefits of Matter in, like, five years, which is I know this is forever. Right? But for example, it it relies on IPv6 and Wes heavily.

And guess what? Most Wi Fi routers just suck at that even though these standards have been around forever. Right? They just suck. Now try convincing, like, everyone in the world to buy a new Wi Fi router because of your fancy new smart home standard.

Scott gonna happen. It's not gonna work. No. Long term, though. It's the same thing with browser standards as Wes. You know? Like, we've been talking about the customizable select drop down for for many, many years, and we're almost there. Yeah. And, like like, part of me wants it right now, but part of me wants them to get it right so that Yeah. The the next twenty years of web development have have amazing primitive.

I want my combo boxes. Like, that's been the biggest pain point. Like, a combo box on mobile Wes you can type, the keyboard pops up, then you start dragging the results, the keyboard goes down, but then your touch point hits outside of the list item and your list closes again.

So true. So true. That's I actually did we had, the folks on that were working on the on the customizable Vercel spec. And I was like, what about combo box? And they're like, it's coming after this one.

I mean, it's this stuff is very hard to get it in a way that it can be customizable for every use case yet remains accessible yet I know. I think what browsers on mobile phones do great is that you hit a a native combo Bos. You just have a native UI. It's just not the drop down. It's just like, hey. Here's just all the results like you're used to from any list on your phone,

and it just works so well. My last question I have for you is, what does your smart home look like? Like, are you are you nuts? Like, do you have everything?

No. So the thing is every time I, you know, I I get a lot of Vercel. Like, either people Bos send me devices to try and these kind of things. And every time I'm trying to automate my house or put things in my house, I see kind of bugs or things that can be improved, and then I just start coding and fixing that stuff. Right? I need to, like, scratch that itch. I do have a lot of stuff, though. I have I have a Reolink doorbell. I have Yield Logs. I have, ZigBee lights. I have a Z Wave remote.

I have some matter oh, I have matter over threat blinds, which thanks to my Apple Node GitHub that I use a strap border router works 50% of the time. That's part of the whole threat matter stuff that isn't fully working yet. Oh, and the one thing that I like a lot is that, you know, we've been doing a lot of stuff with voice, with home assistant.

And so we build this thing we call the world's most private voice assistant, and it's actually an analog phone. So an analog phone, you connect it to a voice over IP box, which then the voice over IP box that we use, Grandstream, has this feature called off hook auto dial. It means that the moment you pick it up, it dials a number. Now this number can be an IP address because it's voice over IP. So it calls Node assistant. Home assistant picks up, and now you're talking to a large language model. So you pick up, and then, like, my kids have these whole conversations again Node about, like, whatever.

And now in the last release, we did, like, last week, Node assistant can call back. So for example, sometimes the laundry is done, but you don't wanna send notifications to phones because you don't know who's there.

What if you could make a sound in a room and somebody has to pick up to, you know, show that they've received a notification to get it?

We got that. So Oh. That one is like the the cherry on the top of my smart home. That's great. I love that. Alright. So the last two sections of the podcast we have are a sick pick and a shameless plug. Sick pick is where you you pick a product or an item or a service or a movie or anything that you you love. Doesn't have to be programming related or not. And then a shameless plug is, of course, anything you'd like to plug. Do you have a

a sick pick for us today? I have the sick pick, but it's kind of not cool anymore because of, like, some stuff they recently did. But, like, I got this Bamboo three d printer, and it just it's so easy to three d print. Like, I used to have, like, other three d printers, and it is really a hobby.

The with the latest bamboo that I have, it's like an x one carbon.

I just have this app on my phone, and I see something cool, and I hit ESLint. And then two, three hours later, I go to my office, and there's just a thing. Right? Like, I'd make this, like, Mario and this Node awesome. You know? Node. And it's like it just it just exists. And I just hit print. Like, I'm not I tried some tree modeling. I just don't have the patience for it, the measurement, all that stuff, the iterations.

But just browsing a website and hitting print, amazing. I have two young kids, and they go to birthday parties. And I'm like, the coolest dad. Right? Like, I have this dragon skeletons and, like, whatever. I've it's my birthday next week, and I my wife is like, you gotta get a three d printer. And I've been looking at I've got the tab open right now for that. I I think I'm gonna go with that one. It's very nice. If you get the they have this thing called AMS, which I didn't know I wanted, but it's really cool. It's like an automatic material system, and so it can change between the different plastics that it use. So you could do four colors, but you can actually chain them. So now you can go up to 16 colors that it can automatically mix into your prints.

That's amazing.

Yeah. That's cool. Do you have a a printer, CJ?

No. I'm in the same boat. Like, I've I'd for like, I'm being a hobbyist and maker, like, this three printers are the one thing I haven't explored.

But recently, I wanted to start building custom cases for old screens.

Like, make so I want I have, like, old iPads. I don't use anymore, old tablets, but I wanna make it look like like a CRT monitor. So I wanna three d print, like, the bezel.

And so I was looking into, like, you can there are services online. You can oh, nice.

But you can send them three d models, and they'll print them for you. But I was like, man, I should probably just get a three d printer. Yeah. That's great.

The the the I just I've been holding up a picture frame, for the people that are listening only. This is a picture frame that fits, Nintendo Switch games. Oh, wow. And so I use this for my son. He puts in his games, and we hang him on his wall so that he has, like, a whole, like, Mario, Minecraft like, wall. Yeah. That's great. Oh, I were really quickly, I wanna shout out Andrew I don't know how to say it. Schlemnullen?

I'll I'll link link up his, his Twitter, but he does a lot of hardware hacking as well.

And he also has been trying to figure out how to communicate with a old VoIP box so you can you can connect to a like, a old school telephone, and I thought that was really interesting. Alright. So shameless plug. What would you like to plug away to our audience?

Yeah. So Home Assistant is world's largest open source project. It is a lot of work, and we don't do that all in our spare time. In fact, six years ago, I started a company, Nabu Casa, which supports the development of Home Assistant. And by subscribing to Home Assistant Cloud by Nabu Casa, you support, you know, we raise money. That money, you get a remote connection, remote access into Node Assistant, better voice assistant, voices, better speech recognition, better camera support through WebRTC routing. Now last year, I actually took Node Assistant, created a foundation. I donated Home Assistant to it. I donated ESPHome to it. It means that these projects, they can never be bought. They can never be sold.

Nabokaze never had any investors. It was always fully funded by the Sanity, so that made this possible.

And so now having these projects secured, we still Namukast is the commercial partner and still funding development of the 52 people that work today on Node Assistant and the whole broader ecosystem.

Wow. So, I mean, if you Yarn using Home Assistant or you are going to use Home Assistant, support us. Right? So you can do that by subscribe to Home Assistant cloud or getting, like, a Home Assistant green on Amazon, getting a voice PE. That's the the voice preview boxes that we have. All these sales of those things support the development of Home Assistant, ESP Node, ESP Wes tools,

all that stuff. That's awesome.

I'm very glad that you will not one day wake up and turns out Xiaomi or someone has bought the all of Node Assistant, you know, or goo even worse, Google. And they eventually will just shut her down Yeah. In six months from Node. So I appreciate that. Cool. Alright. Well, thank you so much for coming on. This was, I couldn't believe when we met you at the conference, and I was like, biggest homosysic fan ever. And, and then I found out you're using web components, and I was like, my whole worlds are overlapping here. So I appreciate you giving us a hour of your time to to come chat. That was really fun.

Yeah. Happy to be here. It was great.

Awesome. Alright. Peace.