You see - my first and longest passion in life has been programming. I remember, in vivid detail, my dad sitting on the carpet with 6 year old me and a big, clunky, mold-injected, dark-grey laptop. He wanted to show me qBasic - something he didn’t know himself but he had a feeling it was going to be a big deal. My dad was an electrical engineer and was versed in hardware but he could see the writing on the wall for software.

I spent hundreds of hours on that laptop. I had no idea what I was doing but I knew my dad did “stuff like this” (he didn’t) and I knew I wanted to do the things he did. I asked him tons of questions and he pointed me to the very good and rich help documentation that came with that version of qBasic. I can’t claim to have made anything amazing those first few years - but I learned a lot and I knew that was what I wanted to spend my life doing. I don’t know how or why but I just knew.

I got into C and C++ years later so I could work on MUDs. If you’re not familiar, they’re text based games generally played through a telnet client. I spent a larger part of my youth scouring around textual worlds and I wanted to be able to forge new worlds, spells, and skills as the immortals did. I remember getting CircleMUD to compile for the very first time Christmas morning. That moment eclipsed the entire Christmas.

Fast forward to a few years ago when ChatGPT was finally starting to be useful. This was post “copilot” auto-complete but pre-vibe coding as we know it. The writing was, once again, on the wall. I was working on some decentralized clustering stuff - sending custom UDP packets around to save on latency and copilot was just shoveling really good code around. This wasn’t stuff that models would have been trained on because it was completely new protocols for nodes to discover one another. I sat stunned thinking - wow - this is…crazy.

Now fast forward to about a year ago. I was working on building a web based MUD (can get away from your past) and was bouncing between Grok for writing out massive amounts of code and VS Code. I built something playable with tons of features in just a few days. I was still hand managing the code because of course I should be doing that. I had, in fact, written a few things that allowed me to work with agents from the CLI (pre-stage harnesses) but it felt wrong to just…use that for writing code. I needed to see it, read it, make sure it wasn’t garbage code.

Then a few months later I vibe coded around 40 personal projects for home automation. Straight up vibes, no VS Code, never looking at the code, never reading it, just knowing what I prompted and that the stuff was working as expected. I remember telling my wife “this is crazy, I’m just building all these things I’ve wanted to build for years and…we just get them.” And I remember telling her “my career is done.”

My wife has no problem telling me when I’m wrong and she did then. She said “Austin, you’re still doing the same things - you’re just faster. Can AI just make these things?”. I thought about it “Well, not really, I mean…I still have to explain what I want and get it to actually work. It’ll do silly, impractical things so I have to steer it the right way…” And I started feeling a little better about AI.

Not better about programming though. That which I had spent over 30 years learning, practicing, loving. No - that is largely behind me now. Not because I want it to be but because it’s no longer practical. I could sit down and hand write some Java or C#. I could spend a few days working on a project I’ve been meaning to get to. Or I could have it done this afternoon. Maybe when I retire and am looking at ways to keep the mind busy…I’ll dust off VS Code tickle the keys. But…my wife and kids take priority over my time right now.

Fast forward more months, in the tech world about three decades, and I’m confident in the tidal shifts of software development and the continued necessity of the developer. AI is a wonderful auto-complete but you have to give it the context to get what you want. That’s not just describing the problem, that’s making sure it’s solving the problem in a good way, and that it’s actually solving the problem rather than hallucinating.

These are non-trivial tasks which rely on understanding computer science fundamentals, system design and distributed systems. Further - avoiding hallucinations and validating the work is non-trivial.

Sure - you can do this all by hand. You can look at the code, review it, validate the results manually. But this is the old way - and it’s much, much slower.

Even before AI - I was a firm believer in tests. Tests gave a developer confidence that their code did what it needed to and that changes couldn’t break that. After AI - I’m a firm believer that tests are the linchpin in a system built on non-determinism and unknowns.

Ok - but how do I know my tests actually test real things? This is part of the non-trivial work we as “post-programming developers” or maybe “agent wranglers” have to do. Human-in-the-loop verification of tests, the product, and getting agents to manually verify behaviors enough that the tests and product must be good.

It’s not perfect and it’s not what my original passion is. But it’s the world now and it means we can build things in days that would have taken months. We can build systems in weeks that would have taken years. We have the powers of creation at our fingertips and need only to think to create whole new worlds.

That is a goofy sentence and I know it. But it is also the closest metaphor I have for what this has felt like.

Sometime around August or September of last year I remember pushing back on my skip-level manager about AI. Not in a "this is all nonsense" kind of way. More in the way engineers tend to push back when the demo looks great but the last mile still looks like it has teeth.

The models were impressive. They could explain things, generate code, summarize documents, and generally do a convincing impression of something useful. But that was also sort of the problem. A convincing impression of useful is not the same thing as useful. It felt like the industry was asking us to trust a very enthusiastic junior developer who could type at impossible speed and occasionally forget what repository it was in.

So my position was pretty simple: AI was interesting, but it wasn't ready. Not evil. Not salvation. Just not ready.

I don't think that was a bad position at the time. Honestly, I still think skepticism was the right posture. There are few things more expensive than pretending a tool is production-ready because everyone around you is tired of nuance.

Then December happened.

We got direction to go all in on AI. This is the kind of sentence that can either be exciting or deeply concerning depending on how many production systems you've been paged for in your life. I expected to be frustrated. I expected to spend a lot of time proving that the edges were still too sharp. Instead, after spending real time with the newer models, I had the slightly uncomfortable realization that the ground had moved.

Not a little bit either.

It wasn't that AI suddenly became magic. It still made mistakes. It still needed review. It still needed someone with taste, context, and a willingness to say "no, that's not what I meant." But it had crossed some threshold where the failure mode changed. The old failure mode was "this cannot really help me build the thing." The new failure mode was "this can help me build the thing, but I need to steer it like I mean it."

That is a very different world.

The window on the wall (pre-AI)

Our house doesn't really have a window to the front yard. There is a long, vertical window near our front door but it's not very useful to look through. After some convincing and pleading, my wife let me replace some artwork with a TV to act like a "window" outside. We have a number of Wyze (or did) cameras around the outside of the house so I just assumed it would be easy to show the feed.

Wyze used to be a very friendly, open platform but over the past 5-10 years that's really changed. They removed their RTSP firmware that let you do something like this along the way and have really built up their walled garden.

I used Grok to find alternatives and found Tapo cameras offered a very open and local friendly camera - so I replaced some of the cameras with those and got the TV showing the feed.

The problem is - it wasn't great. At that time I hadn't really dug into video (yet) and spent a long time trying to improve the quality and latency of the video. Really to not much avail. So I called it "good enough" but I was disappointed and my wife was a bit perturbed having not really gotten the "window" I was talking about.

The window on the wall (post-AI)

After December, I came back to the window with the full power of the dark side... erm... Force.

The first version technically worked, if you were willing to be extremely generous with the word "worked." It was a sad, overly slow and jittery video feed pretending to be a window. The TV was on the wall. The camera was outside. Photons were making some kind of journey between them. And yet the whole thing felt less like a window and more like watching the front yard through a gas station security monitor from 2008.

So I went back to it, but the question had changed. It was no longer, "Can I get a camera feed onto an Android TV?" It was, "Can I make this feel like an actual window on the wall, with a dashboard and a live-streaming, high quality, 4K view outside?"

That meant learning video in anger. I went deep on go2rtc, WebRTC, RTSP, codec profiles, hardware acceleration, keyframes, packet loss, and all the little ways a video pipeline can technically be alive while spiritually being dead. I tried direct streams. I tried transcoded streams. I tried TCP paths that looked reasonable until a tiny stall turned the whole thing into a slideshow. I added UDP ingest aliases because RTSP-over-TCP head-of-line blocking is apparently something you should care about.

But. I didn't really do any of that. An agent did it in roughly a total combined iteration time of 1-2 hours.

Then the single-window version became an all-camera version. I stitched the cameras into one go2rtc stream: front_window_camera_grid. A native 4K 2x2 canvas, four 1920x1080 tiles, all in one easily consumed format.

And then, because the window was never really just about cameras, the house state went on top of it. Weather data from...somewhere with watches and warning alerts. The current outside conditions. Family member locations...yes like Harry Potter. Automated day and night camera switching for better video quality. A front-window status endpoint that could compose all of this into one watchable page. The TV stopped being a screen showing a camera and started becoming what I had wanted in the first place: a house operations dashboard that happened to have a live view of the world outside.

The funniest part is that this eventually pulled Wyze back into the story. I had moved toward Tapo because Wyze had spent years making this harder, but apparently spite is a renewable energy source. I ended up digging into Wyze streams across generations: the older TUTK/Hualai cameras, the newer GWell/MARS/IoTVideo world, doorbells, camera models that needed different adapters, relay processes, raw H.264 frames, cloud metadata, and enough translation machinery to make them visible to go2rtc. The walled garden still had walls, but now I had a ladder, a map, and a lot of very determined code.

This is the part of the AI story that still feels wild to me. I did not become a video engineer first and then build the system. I built the system while becoming just enough of a video engineer to keep going. Every time I hit the next weird failure mode, I had a collaborator that could help me reason through it, generate another probe, explain another protocol edge, or turn a half-formed theory into an experiment.

That is how my little "window" went from a wall-window dashboard into the place where more and more of my things operated - called Conclave. Weather, cameras, locations, smart-home state, status pages, monitors, voice, notifications, the whole increasingly unreasonable pile. And everything exploded into too many parts to keep in my head, which is how I ended up needing PLS.

Nothing can stop the power of the Force. Nothing.

The shipyard

So I built PLS.

PLS started as the answer to a very practical problem: Conclave and my agent had exploded into too many projects, services, repos, and deployments for me to keep in my head. It turned into a Rust project lifecycle service with a server, CLI, client library, REST API, MCP surface, Keycloak/OIDC auth, SQLite/Postgres persistence, an operations dashboard, deployment records, release packaging, installers, and enough tests that it feels like a real thing instead of a weekend notebook that learned HTTP.

Of course, a project lifecycle service needs deployments to track. So Coxswain grew up alongside it. Coxswain is the Proxmox/Docker deployment packaging tool (with full Helm CLI parity) that lets me package services into sextants (aka charts), deploy them into LXCs, manage runtime values, stamp versions, check drift, and sync deployment events back into PLS.

I built the window, then the place where the house operated, then enough services around that place that I needed a registry for the fleet, and then a deployer that could tell the registry what happened to the fleet.

That would have been a reasonable place to stop. Naturally, it was not where I stopped.

The assistant substrate

The next layer was the stuff that makes an assistant useful over time.

Memory MCP became the place for durable memory: not just "remember this sentence", but embeddings, background indexing, separate deployments, and enough structure that future work can find the shape of old work. Secrets MCP became the sane place to put credentials so agents do not spray API keys into logs, docs, shell scripts, or whatever other cursed place a token might end up if left unsupervised.

Todo became more than a CLI. It grew tags, an MCP service, true-user auth, a REST API, a simple web frontend, API keys, OAuth login, a Tauri tray app, a SwiftBar plugin, and Logseq sync work. Fastmail MCP turned email and calendar access into something Guppy could reason about and act on without me living in an inbox.

At some point this stopped feeling like "AI helped me write some code" and started feeling like building a nervous system.

The pieces were not glamorous by themselves. Auth migrations are not glamorous. Secret catalogs are not glamorous. Deployment metadata is not glamorous unless you've been burned enough times to develop strong feelings about knowing what version is running where.

But those pieces changed everything. Once they existed, the next project started with memory, secrets, todos, docs, deployment, and observability already in the room.

The product layer

Macro360 is probably the cleanest example of the shift.

It started as a food, exercise, and calorie tracking app because I wanted something to replace MyFitnessPal. It grew into auth, diary screens, food and exercise catalogs, MyFitnessPal import paths, Apple Health sync, passkeys, mobile UX passes, observability, deployment work, and a pile of small fixes that are the difference between "technically works" and "I might actually use this."

That last part is important. AI did not make the need for product taste go away. If anything, it made taste more important. When the cost of generating code drops, the value of knowing what should exist goes up.

Then there is Jagu.

Jagu started from a pretty mundane desire: I wanted a better way to run scheduled jobs for my agents and Conclave. Cron works, in the same way a folding chair works. Perfectly fine until you start asking it to be furniture.

Jagu became an OIDC-authenticated Docker job engine with an API, UI, scheduled runs, encrypted task secrets, Docker env-var injection, SQLite or bring-your-own persistence, Keycloak integration, frontend screens for tasks/runs/secrets/API keys/users/audit/system health, and deployment shape through Coxswain. It has tests for auth boundaries and real browser coverage because, again, the point was not to make a demo. The point was to make something I could trust enough to give it chores.

That is the thing I keep coming back to. The useful output of this AI wave has not been a pile of snippets. It has been chores getting promoted into systems.

The house started participating

The funniest proof that this was different is that the window kept asking for more of the real world.

Conclave and my agents started reaching into the physical world. Ambient Weather ingestion came online. Rain Bird irrigation got an integration. MQTT got cleaned up with a broker, tests, and rebroadcasting. Smart-home work grew a local/offline-first provider layer, an agent, Alexa MCP, a private Alexa skill, voice-control services, and an M5Stack Atom Echo bridge to replace Alexa/Amazon voice integration (Alex+ just kept changing how I needed to phrase things to still do what I wanted. Plus the "ads" are extremely annoying).

That is a lot of words to say: I can talk to my house, my house can publish events, and there are now enough small services involved that a normal person would probably back away slowly.

But it works. Or, more accurately, it works, breaks in interesting ways, gets monitored, gets fixed, and works better the next time. Which is what most real systems do.

The camera work is the best example of the "breaks in interesting ways" part.

There is a Generic Camera MCP, a Tapo Camera MCP, Wyze probing, go2rtc work, and the wall-window Android TV display. It can show weather, location state, and camera feeds on the TV. There are stitched camera grids. There are RTSP streams, WebRTC paths, TCP aliases, UDP ingest experiments, and periodic monitor jobs that check whether the whole thing has quietly turned into a blank mosaic.

That sounds ridiculous because it is ridiculous. But it is also exactly the kind of thing that used to die at the "I should really do that someday" stage. With AI in the loop, "someday" started becoming Tuesday night.

The recursive part

Eventually I started using AI to build the local AI stack that I use to build the things.

There is a local LLM host on the Mac Studio running Ollama, LiteLLM, a turbo model server, local embeddings, and routing for services that need model access. Open WebUI moved to Proxmox while the heavy GPU-backed pieces stayed on the Mac Studio where they belong. There is deployment work around that too, because apparently I am no longer allowed to just run a thing and remember where it lives.

Then came the performance rabbit hole.

I spent time on MLX and TurboQuant benchmarks, prompt-cache behavior, KV cache tradeoffs, MTP scheduling, and whether there were real source-level wins hiding below the obvious configuration knobs. Some attempts helped. Some were flat. Some were rejected after measurement. This is, in other words, still engineering.

That is probably the healthiest way to think about all of this. AI did not remove the part where evidence matters. It made it possible to run more experiments, keep better notes, and move through more of the search space before running out of evening.

There were also weirder explorations: Agents and Angels, the desktop/RPG UI, Entity Overlay, audio capture, voice cloning workbenches, and Riprok. Some of these are products. Some are research. Some are me following an idea just far enough to find out whether it has legs.

I have always liked that part of programming. The difference now is that the distance between curiosity and artifact is much shorter.

The lows are still lows

None of this means AI makes software easy.

It makes some parts faster. It makes some parts much faster. It also makes it very easy to create more surface area than you can responsibly operate if you are not careful.

Auth is still auth. Secrets still need discipline. CI still fails. DNS still finds ways to be DNS. Cameras still wedge. Linux hosts still need updates. Ghost still needs security patches. Firewalls still need to be more restrictive than "sure, why not." A model can help you diagnose all of those things, but it does not make them stop existing.

The human role changes, but it does not disappear. I spend less time staring at a blank file and more time deciding what should be true. I spend less time remembering the exact syntax for everything and more time judging whether the generated plan is sane. I spend less time typing boilerplate and more time saying "no, that abstraction is not buying us anything."

Which, if I'm being honest, is a pretty good trade.

So what is AI?

In August or September of 2025, I do not think I was wrong to be skeptical. The demos were ahead of the reliability. The hype was ahead of the habits. The industry was doing that thing where everyone says "transformational" enough times that the word starts to lose structural integrity.

But I also do not think AI is usefully described as good or bad on its own. It is not the light side. It is not the dark side. It is leverage. It amplifies intent, judgment, impatience, taste, sloppiness, discipline, curiosity, and all the other extremely human ingredients that get poured into it.

But by December, or at least by the time I seriously started building with the newer models, something had changed.

Not because the models were perfect. They were not. They are not. But they had become useful enough that the limiting factor moved. The limiting factor was no longer "can AI help with this?" It was "can I describe the system clearly enough, review it carefully enough, and keep the operational discipline around it strong enough that the output compounds instead of collapsing?"

That is a much more interesting problem.

Today PLS says there are dozens of projects in the fleet. Some are tools. Some are services. Some are apps. Some are half-research, half-confession. There are deployments, dashboards, MCP servers, voice bridges, weather ingesters, camera monitors, job schedulers, local LLM routes, and enough documentation that future me has at least a fighting chance of understanding what present me was thinking.

That's the part that feels amazing.

AI is scary, frustrating, amazing, and...it really does feel like the Force.

tl;dr - I had a TikTok collection full of book recommendations, collected over roughly two years, and no sane way to read it. So I pointed agents at the mess, made them OCR the videos and image slides, and ended up with 3,217 book mentions across 412 TikToks.

This started in the deeply respectable way most personal data projects start: I had been tapping the little bookmark button on TikTok for years and pretending that counted as organization.

To be fair, it sort of did. I had a collection for books. When BookTok threw some new grimdark fantasy list, sci-fi ranking, or suspiciously enthusiastic recommendation for Red Rising at me, I saved it. The problem is that TikTok book recommendations are not really data. They are videos, captions, subtitles, image carousels, cover shots, creator patter, stitched audio, and sometimes a single blurry slide with twelve titles on it.

So after a couple of years of this, I had something that looked like a reading list but behaved more like a junk drawer.

I wanted a boring table:

• TikTok URL
• books recommended in it
• evidence for each recommendation
• total count by book

Simple enough, except the books were not in one place. Some were in the description. Some were in the auto-generated subtitles. Some were in the video itself. A lot were in TikTok's photo-mode slides, which still have URLs that look like /video/{id} because why make this easy?

Anyway, I had agents now. This seemed like exactly the sort of fiddly task they should be good at.

Getting The Pile

The first step was getting the collection out of TikTok. I used the logged-in web API call behind the collection page and paged through it until I had the whole thing locally.

That gave me 412 TikToks.

At this point I had metadata: IDs, URLs, authors, descriptions, subtitle links where TikTok exposed them, and for photo-mode posts the imagePost.images array. That last bit ended up mattering a lot.

I did the obvious first pass:

• parse descriptions for explicit "Books mentioned" lists
• fetch subtitle files where available
• pull structured metadata out of the API response
• aggregate title counts conservatively

That got a decent chunk of the way there, but not nearly all the way. BookTok is not a well-formed CSV file, despite my personal wishes.

The First Wrong Pass

The next assumption was that the leftovers were video-only.

So I downloaded the remaining TikToks with yt-dlp, sampled every fifth frame with ffmpeg, OCR'd those frames locally with Apple's Vision framework, pulled audio to WAV, and ran local Whisper STT on it. Then I batched the OCR/STT text to subagents and had them extract book titles instead of trying to hand-roll a parser for "bookish human speaking over screenshots."

This was useful, but it also produced a weird result. A bunch of the "videos" had no video stream. They had an audio track and a cover image. The OCR looked like:

TOP 10 SCIENCE FICTION NOVELS OF THE 2000S

Which is not, strictly speaking, the list.

This is where Austin from twenty minutes later was smarter than Austin from twenty minutes earlier: those were not normal videos. They were TikTok photo-mode posts. TikTok shows them at /video/..., but the API has an imagePost object with every slide in the carousel.

Back to the scraper.

OCR, But Actually The Slides This Time

The corrected pass used imagePost.images directly. For each unresolved TikTok, I downloaded every slide image and OCR'd each slide independently. This was the missing step.

The final unresolved batch was:

• 74 photo-mode slide decks
• 683 individual slide images
• 136,788 characters of fresh OCR
• 0 download/OCR errors

Those slide decks added 609 more recommendation mentions by themselves.

The extraction workflow was pretty simple:

1. Save the raw API pages locally.
2. Fetch descriptions, subtitles, and image slides.
3. Run local OCR and STT.
4. Split the resulting text into small batches.
5. Send each batch to a subagent with a strict JSON schema.
6. Merge the results into one TikTok-to-recommendations map.
7. Rebuild aggregate counts from that map.

The subagent part is worth calling out. I did not want to pretend that "OCR text from a TikTok slide" is easy to parse with regex. Sometimes the title is next to the author. Sometimes the author is half missing. Sometimes the slide has blurbs, rankings, UI text, and a book cover all smashed together. The useful unit of work was not "extract all capitalized words." It was "look at this messy evidence and decide which titles are clearly recommendations."

That is a good agent task. Bounded, checkable, annoying, and mostly judgment rather than architecture.

The Result

After the corrected slide pass, the catalog covered all 412 TikToks.

The final numbers:

• 412 TikToks cataloged
• 3,217 TikTok-level recommendation mentions
• 1,634 unique normalized recommendations
• 683 image slides OCR'd in the final correction pass
• 0 items left in the review queue

The top recommendations were not shocking, but they were satisfying:

Red Rising being number one by a mile is maybe the least surprising thing that has ever happened on BookTok. I am not complaining. It just feels like the algorithm has a quota.

The Weirdly Useful Part

The useful part of this was not that I got a list of books. Though, yes, I did get a list of books, and my TBR is now less a list and more a zoning problem.

The useful part was that this was a genuinely personal dataset that I could not reasonably process by hand. It was too big to casually inspect, too messy for a simple script, and too small/weird to justify building some polished product around it.

That is where agents fit nicely.

Not in a "replace all software" way. More in a "take this pile of semi-structured personal exhaust and turn it into something I can actually use" way.

The workflow was not magic. It still needed me to notice when the first pass was wrong. It still needed local OCR. It still needed API spelunking. It still needed schema checks, merge scripts, and a final sanity pass. But the agents made the middle part tractable: hundreds of small judgment calls over messy captions, slides, subtitles, and OCR fragments.

That feels like the shape of a lot of useful agent work right now. Not a shiny demo. Not a chatbot pretending to be a database. Just a patient little assembly line for all the annoying glue work between "I have this somewhere" and "I can use this now."

At the end of the day, I still have too many books to read.

But now at least I know which ones TikTok has been yelling at me about for two years.

I've managed some semblance of a smart home for the past 7 years. This began as most things like these do - a toe in the water, then a foot, and then suddenly you've fallen in and are trying to stay afloat.

The toe was smart light bulb. I got it from a random company giving them away at San Diego's Comic Con (I think around 2017). It may have only been bluetooth... I don't remember. But it piqued my interest enough to say to myself "Some day, my home is going to be smart. Like that Disney movie but without the house trying to kill me."

The foot was a batch of Sengled light bulbs that connected to Alexa through a proprietary hub. We had a few Amazon devices already so it was a logical step. This was around the time FAANG were dipping their own toes in the metaphorical water so the integrations were there but not great. The Alexa app left a lot to be desired (note: it still does - they never really went beyond the toe) and it became quickly apparent trying to set up rules and schedules that this wasn't the smart home I envisioned. Not only that, but the hub seemed to be quite limited in range. I could reach bulbs in my house but it struggled to communicate with bulbs outside.

Fortunately for me, Sengled had done things right. That hub was simply a zigbee hub and the light bulbs were all run over the zigbee protocol. I found all of this out while deep diving what I needed to do to set up a "proper" smart home.

The first improper version of this proper smart home ended up using Home Assistant. This was still around the 2018 time, I'm not sure where that would fall in Home Assistant versions, but it was relatively easy to set up. I picked up a Raspberry Pi and a random zigbee/zwave dongle from Amazon and got cracking. A few hours later and I was able to control the light bulbs in my house. A few days later, and I can't quite remember if I had to set up a custom Alexa skill and/or an app, but I then was back at sqaure one with the Sengled bulbs - Alexa controlled. But this time with a better zigbee hub.

Some time passed, I added more bulbs [1, 2, 3], some smart light switches [1, 2], and some scenes to better manage daily/nightly tasks. To this day, one of those scenes, still exists - "master movie scene on/off" which turns on our lamps and turns our bedroom lights off. But not all things were right in the Hanson household - many zigbee devices would occasionally just not work. The Alexa integration itself would occasionally stop working.

This takes us to version two of the smart home - where I moved from Home Assistant over to OpenHab, version 1, if I recall correctly. Perhaps a little hasty, this move was driven by a number of posts I found advocating for its better integrations and better zigbee support.

So I wiped away the Raspberry Pi SD card, burned in OpenHab's raspbian custom image, and got started starting over.

I will say, OpenHab was way less polished than Home Assistant. Years later, now on v3, I'd still say it's probably less polished out of the gate. You can make it look pretty - but its very much on you to do.

It took a little doing, mostly learning OpenHab's peculiarities in setting up devices. I used there old, yet seemingly better, UI that allowed adding devices dynamically. More on this later... A few hours later, or maybe days, and our smart home was back up and running. Alexa integration was a few days later and used their app so I didn't have to manage an Alexa developer account anymore.

This ran fine for probably 6 months and then, growing in frequency, Alexa would simply not be able to interact with certain devices. The fix was restarting the Raspberry Pi and everything would work. This would last for a few months until a restart was needed to "fix things" again. Then it would only last a few weeks. Then it was down to a few days. Obviously...things weren't going in the right direction. Not only that, we actually used a few things frequently ("Alexa, turn master movie scene on" and "Alexa, goodnight") and having it not work was really cramping our style!

So, on a cold, miserable night in Janua... I don't actually know when I did this - I started digging into the problem. It turns out, many people online were complaining about this and, seemingly, OpenHab was saying it was a zigbee issue. That tooks me down a few paths that led to "you just need to use an MQTT broker and zigbee2MQTT". As a developer...I'm not sure why the zigbee integration couldn't be "fixed" but I also wasn't jumping in to do the fixing... so I installed the MQTT broker and zigbee2MQTT on the Raspberry Pi while moving from OpenHab v1 to v2. As a sidenote for the reader, and if I remember correctly, this was yet another destructive upgrade that forced me to rebuild everything from the ground up. I also picked up a new zigbee dongle just in case the issue was with the old dongle.

A few hours later, or days, and things were working again. Now my memory is particularly hazy at this point (it actually always is) but this might have been when OpenHab ditched their old UI and forced you to hard code the zigbee devices in YAML files... Or that might just be because the MQTT broker doesn't support any kind of dynamic device adding... I'm not really sure - but suffice it to say, at some point, I was no longer able to just casually add new devices anymore and I had to manually add them in OpenHab's YAML configurations. Not a huge deal for me but no casual user would ever do this... Ok, they'd never do anything above either...

Things were one again working in the world. The annoying Raspberry Pi restarting issue that plagued us was gone and fixed and everything was great. Or so I thought.

Maybe a year after that, I think somewhere around 2022 or 2023, things would occasionally just not work. Alexa would say it couldn't reach the hub, same old, same old. Seeeeeeemingly, the Raspberry Pi was overheating? Or for some reason rebooting?

Now, this was also around the time I was chatting with some folks about setting up a "home lab" and/or local kubernetes clusters to run things. Rather than digging into the issue with the Raspberry Pi I figured this was an opportunity to invest in some infrastructure. I picked up a couple little micro form factor PCs (Beelink something) and installed Proxmox on it. I quickly realized this was entirely too few resources and returned the one I still could and kept the other. Around the same time I buddy gave me a few extra mff PCs he'd had that were a little punchier. I set those up as proxmox nodes and had a nice little "home lab". I then added a few VMs for OpenHab v3, MQTT, and zigbee2MQTT, figured out how to pass the zwave/zigbee USB dongle through to the VM, and we were back in business!

Fast forward a couple years and I ended up buying a Dell PowerEdge R730xd Server to expand my my home lab. I started shuffling VMs over to it from the mff PCs but couldn't move the zigbee2MQTT VM as it had to access the USB dongle. This wasn't really a problem as the little PC was tucked away but it was an annoyance - I didn't really want the little PC anymore.

Then, at some point, the Bee-something mff PC started randomly not being able to talk to VMs on the network. Other things could talk to it, and it to some things on the network, but not everything. With only minimal digging, an ifreload -a would fix the problem and make things right. Occam's razor would suggest something to do with the VM shuffling, maybe a conflicting IP address, or something - but I also started seeing random warnings in the syslog for the networking stack... All I knew was, things were looking a lot like they had before - bad.

I ended up running across this SLZB-06 zigbee coordinator/hub/thing a week or so ago. This essentially turns the USB dongle into a standalone device. A few hours later and the mff PC is gone, the SLZB-06 in its place, and everything is right again! ¯\_(ツ)_/¯

Philipp Oppermann's guide to Writing an OS in Rust is an absolutely fantastic place to start and I encourage anyone interesting in programming to go through it.

As indicated above, I've been hankering for something really low level but, truth be told, I didn't really care for Rust. I understand that its borrow-checker is its secret sauce and what makes it a better suited language for writing reliable programs. But I really don't like the added complexity. Consider the fact that there are an overly unhealthy number of StackOverflow questions asking how to implement linked lists and other trivial data structures in Rust. I digress, I don't want to make this a bash Rust post - this post is not that.

What I do want to go over is getting something in Zig running on "bare metal".

In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system.

There seems to be a fair amount of interest in doing this but mostly centered around writing something for the Raspberry Pi [1] or other micro-controllers [1, 2]. That might be something I look in to later but for now I'd really like to mostly follow along with Oppermann's blog guide but in Zig.

To do that, seemingly I need to compile a freestanding Zig binary and then extend it into a Zig kernel capable of being ran by some bootloader.

Unsurprisingly, I'm not the first person to try this [1, 2, 3] so there was a bit of a path to follow.

First things first, let's work on compiling a free-standing Zig binary. We can actually accomplish that directly by passing some arguments to zig.

Zig is very much a growing language right now with a fair number of breaking changes happening with every version. I'm going to be using 0.8.1 and I'm not going to tell you why - just yet anyway.

Ok, fine, it's because this is a work in progress and that's the latest version I've gotten things to work with. Seemingly, everything here should work up until 0.10.0 as Async functions and support for overriding the stack pointer is missing in 0.11.0 and beyond until...it's readded. ¯\_(ツ)_/¯

zig build-exe freestanding.zig -target i386-freestanding

A Simple Freestanding Binary

Let's try that with a simple Zig program.

freestanding.zig

const std = @import("std");

pub fn main() void {
    std.debug.print("Hello, world!\n", .{});
}

And it compiles but with a linker warning...

➜  zig build-exe src/freestanding.zig -target i386-freestanding
warning(link): unexpected LLD stderr:
ld.lld: warning: cannot find entry symbol _start; not setting start address

There's not a lot of documentation for the freestanding targets but we can surmise that a freestanding targeted binary requires a _start method rather than relying on main. This is probably because of the lack of an initial runtime (crt0). ¯\_(ツ)_/¯

freestanding.zig

const std = @import("std");

export fn _start() void {
    @call(.{ }, main, .{});
}

pub fn main() void {
    std.debug.print("Hello, world!\n", .{});
}

Which compiles with

➜  zig build-exe src/freestanding.zig -target i386-freestanding
/usr/local/etc/zig/0.8.1/lib/std/fs/file.zig:20:15: error: container 'std.os' has no member called 'fd_t'
    handle: os.fd_t,
              ^
/usr/local/etc/zig/0.8.1/lib/std/debug.zig:67:22: note: referenced here
    const stderr = io.getStdErr().writer();
                     ^

This is even less obvious but it's because we can't use the standard library with a freestanding binary!

freestanding.zig

const std = @import("std");

export fn _start() callconv(.Naked) noreturn {
    @call(.{ }, main, .{});
}

pub fn main() void { }

If you read through the Rust variant then you know that we mark the _start method with export ensuring it uses the C calling convention. You're also aware that the _start method is called by the system and isn't expected to return anything so we loop instead and mark the method as noreturn. Finally, we also mark the method as having a Naked calling convention which "makes a function not have any function prologue or epilogue".

If you didn't do the export, noreturn, and callconv(.Naked) stuff above compilation still succeeds. You'll get an ld.lld linker warning saying it couldn't find _start and defaults to 0x100000. Then when you eventually get a kernel image nothing happens.

Coincidentally, if you only use export but skip the rest, things seemingly work. But we'll blinding follow along the path laid out before use [1, 2, 3] making sure we're tidy and abiding system call etiquette.

And this finally compiles successfully!

Ok, that wasn't so bad...but we can't run it on MacOS?

➜  ./main
zsh: exec format error: ./main

Not surprising actually, given the binary we built targeted an i386 system and is an ELF file. ELF is a common executable file format used by a number of types of systems but, if you're running MacOS, you'd need a Mach-O formatted file.

You can experiment a little with this by specifying different targets and inspecting the type of file.

➜  zig build-exe src/freestanding.zig -target i386-freestanding
➜  file kernel
kernel-mb1: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, with debug_info, not stripped
➜  zig build-exe src/freestanding.zig -target x86_64-macos
➜  file kernel
kernel: Mach-O 64-bit executable x86_64

A Runnable Kernel

We have a freestanding, ELF-32 formatted binary but we can't do anything with it.

That's not true! We can turn it into a "kernel" by making it multiboot compliant so we can run it with QEMU directly via -kernel which supports loading mutliboot1 compliant kernels.

Note: QEMU's -kernel doesn't support loading multiboot2 compliant kernels.

linker.ld

ENTRY(_start)
 
SECTIONS {
    . = 1M;
 
    .text : ALIGN(4K) {
        KEEP(*(.multiboot))
        *(.text)
    }
 
    .rodata : ALIGN(4K) {
        *(.rodata)
    }
 
    .data : ALIGN(4K) {
        *(.data)
    }
}

This linker script forces the first section to be the multiboot1 header followed by executable program code. Next we specify sections for read-only data and mutable data. See this most thoroughly commented linker script for an example script with great documentation.

And we need to update freestanding.zig to a mutliboot compliant kernel.

kernel.zig

const ALIGN = 1 << 0;
const MEMINFO = 1 << 1;
const MB1_MAGIC: u32 = 0x1BADB002;
const FLAGS: u32 = ALIGN | MEMINFO;
 
const MultibootHeader = packed struct {
    magic: u32 = MB1_MAGIC,
    flags: u32,
    checksum: u32,
};
 
export var multiboot align(4) linksection(".multiboot") = MultibootHeader {
    .flags = FLAGS,
    .checksum = @intCast(u32, ((-(@intCast(i64, MB1_MAGIC) + @intCast(i64, FLAGS))) & 0xFFFFFFFF)),
};
 
export fn _start() callconv(.Naked) noreturn {
    @call(.{ }, main, .{});
    while (true) {}
}

pub fn main() void { }

There's a lot going on here but it's all specific to making the "kernel" (come on, it's not doing anything) multiboot1 compliant.

We create some constants, a packed struct to define the layout of the data, and then an exported variable, multiboot, containing our actual multiboot1 data.

We annotate the exported variable with align(4) and linksection(".multiboot") which lets the linker know we want the section 4-byte aligned and associated with the .multiboot section in the linker script.

And that's that! We can then compile the freestanding, i386 targeted, multiboot1 compliant binary and run it with QEMU!

➜  zig build-exe src/kernel.zig -target i386-freestanding -T src/kernel.ld
➜  qemu-system-x86_64 -kernel kernel

Erm... Did it actually do anything? Yes! Yes it did! It didn't fail to launch! Ok... but let's make it at least print hello world like all of the other guides do.

A Useful (?), Runnable Kernel

Making our "kernel" print to the screen is a lot harder than just using std.debug.print because we don't have access to the standard library. So...we'll have to roll our own. Surprisingly, it's not too terribly difficult because we can get direct access to the VGA text buffer [1, 2].

Here's a heavily inspired (mostly copied) console.zig which lets you change colors and print text.

console.zig

const fmt = @import("std").fmt;
const mem = @import("std").mem;
const Writer = @import("std").io.Writer;
 
const VGA_WIDTH = 80;
const VGA_HEIGHT = 25;
const VGA_SIZE = VGA_WIDTH * VGA_HEIGHT;

/// Enumeration of VGA Text Mode supported colors.
pub const Colors = enum(u8) {
    Black = 0,
    Blue = 1,
    Green = 2,
    Cyan = 3,
    Red = 4,
    Magenta = 5,
    Brown = 6,
    LightGray = 7,
    DarkGray = 8,
    LightBlue = 9,
    LightGreen = 10,
    LightCyan = 11,
    LightRed = 12,
    LightMagenta = 13,
    LightBrown = 14,
    White = 15,
};

/// The current cursor row position.
var row: usize = 0;

/// The current cursor column position.
var column: usize = 0;

/// The current color active foreground and background colors.
var color = vgaEntryColor(Colors.LightGray, Colors.Black);

/// Direct memory access to the VGA Text buffer.
var buffer = @intToPtr([*]volatile u16, 0xB8000);

/// Create a VGA color from a foreground and background Colors enum.
fn vgaEntryColor(fg: Colors, bg: Colors) u8 {
    return @enumToInt(fg) | (@enumToInt(bg) << 4);
}

/// Create a VGA character entry from a character and a color
fn vgaEntry(uc: u8, newColor: u8) u16 {
    var c: u16 = newColor;
    return uc | (c << 8);
}

/// Set the active colors.
pub fn setColors(fg: Colors, bg: Colors) void {
    color = vgaEntryColor(fg, bg);
}

/// Set the active foreground color.
pub fn setForegroundColor(fg: Colors) void {
    color = (0xF0 & color) | @enumToInt(fg);
}

/// Set the active background color.
pub fn setBackgroundColor(bg: Colors) void {
    color = (0x0F & color) | (@enumToInt(bg) << 4);
}

/// Clear the screen using the active background color as the color to be painted.
pub fn clear() void {
    mem.set(u16, buffer[0..VGA_SIZE], vgaEntry(' ', color));
}

/// Sets the current cursor location.
pub fn setLocation(x: u8, y: u8) void {
  row = x % VGA_WIDTH;
  column = y & VGA_HEIGHT;
}

/// Puts a character at the specific coordinates using the specified color.
fn putCharAt(c: u8, newColor: u8, x: usize, y: usize) void {
    const index = y * VGA_WIDTH + x;
    buffer[index] = vgaEntry(c, newColor);
}

/// Prints a single character
pub fn putChar(c: u8) void {
    putCharAt(c, color, column, row);
    column += 1;
    if (column == VGA_WIDTH) {
        column = 0;
        row += 1;
        if (row == VGA_HEIGHT)
            row = 0;
    }
}
 
pub fn putString(data: []const u8) void {
    for (data) |c| {
        putChar(c);
    }
}
 
pub const writer = Writer(void, error{}, callback) {
    .context = {}
};
 
fn callback(_: void, string: []const u8) error{}!usize {
    putString(string);
    return string.len;
}
 
pub fn printf(comptime format: []const u8, args: anytype) void {
    fmt.format(writer, format, args) catch unreachable;
}

Then let's update kernel.zig to display "Hello, world!".

kernel.zig

const console = @import("console.zig");

const ALIGN = 1 << 0;
const MEMINFO = 1 << 1;
const MB1_MAGIC: u32 = 0x1BADB002;
const FLAGS: u32 = ALIGN | MEMINFO;
 
const MultibootHeader = packed struct {
    magic: u32 = MB1_MAGIC,
    flags: u32,
    checksum: u32,
};
 
export var multiboot align(4) linksection(".multiboot") = MultibootHeader {
    .flags = FLAGS,
    .checksum = @intCast(u32, ((-(@intCast(i64, MB1_MAGIC) + @intCast(i64, FLAGS))) & 0xFFFFFFFF)),
};
 
export fn _start() callconv(.Naked) noreturn {
    @call(.{ }, main, .{});
    while (true) {}
}

pub fn main() void {
    console.setColors(.White, .Blue);
    console.clear();
    console.putString("Hello, world");
    console.setForegroundColor(.LightRed);
    console.putChar('!');
}

Viola! We have a runnable "kernel" (it's not really a kernel, right? It doesn't do anything useful). But we should probably make it bootable.

A Useful (?), Bootable Kernel

The last (lol) thing we need to do is make this baby bootable!

We can use grub-mkrescue to build an ISO that we can then use to boot QEMU from.

If you're using MacOS there are some dependencies you'll need to get set up. Nothing too crazy, building and "installing" objconv and GRUB2 from source, and installing QEMU.

objconv is surprisingly straight forward:

cd ~/github
git clone git@github.com:vertis/objconv.git
g++ -o objconv -O2 src/*.cpp
sudo ln -s ~/github/objconv/objconv /usr/local/bin/objconv

GRUB2 is less so but their documentation is fine.

cd ~/github
git clone git://git.savannah.gnu.org/grub.git
cd grub
./boostrap.sh
./autogen.sh

# Need to be able to build i386 ELF targeted C things
brew tap homebrew/core
brew install i386-elf-binutils i386-elf-gcc
mkdir build
cd build
../configure --disable-werror TARGET_CC=x86_64-elf-gcc TARGET_OBJCOPY=x86_64-elf-objcopy \
	TARGET_STRIP=x86_64-elf-strip TARGET_NM=x86_64-elf-nm TARGET_RANLIB=x86_64-elf-ranlib --target=x86_64-elf

# MacOS has a super old version of awk
brew install gawk
sudo ln -s /opt/homebrew/bin/gawk /usr/local/bin/gawk
export PATH="/usr/local/bin/gawk:$PATH"

make
sudo make install

# GRUB2 needs xorriso (now called mkisofs)
brew install xorriso

Phew, with that out of the way let's install QEMU.

brew install qemu

Alright, now we need a GRUB2 configuration file:

grub.cfg

menuentry "A Useful (?), Bootable Kernel" {
	multiboot /boot/kernel
}

Ok, now we can finally create our ISO file manually (ick).

mkdir -p iso_dir
mkdir -p iso_dir/boot/grub/

zig build-exe src/kernel.zig -target i386-freestanding -T src/kernel.ld
cp kernel iso_dir/boot/
cp grub.cfg iso_dir/boot/grub/

grub-mkrescue -o kernel.iso iso_dir

And then all we have to do is launch QEMU!

qemu-system-x86_64 -cdrom kernel.iso -debugcon stdio -vga virtio -m 4G -machine "q35" -no-reboot -no-shutdown

There's more to do here - specifically around automating the entire build process using Zig's build tooling. But I'll leave that for next time.

First, let's install docker and virtualbox.

brew install docker docker-machine
brew install --cask virtualbox

Now let's create a "docker machine" using docker-machine. This wraps around virtualbox to spin up a vm (and probably does other stuff).

docker-machine create --driver virtualbox \
               --virtualbox-cpu-count "4" \
               --virtualbox-memory "8192" \
               default

Now we'll set up a credential store for docker.

curl -O https://github.com/docker/docker-credential-helpers/releases/download/v0.6.4/docker-credential-osxkeychain-v0.6.4-amd64.tar.gz
tar -xvf docker-credential-osxkeychain-v0.6.4-amd64.tar.gz
sudo mv docker-credential-osxkeychain /usr/local/bin/
sudo chmod +x docker-credential-osxkeychain
sed -i 's/desktop/osxkeychain/g' ~/.docker/config.json

Let's install Kubernetes.

brew install kubectl minikube
minikube start

And then, if you use zsh / oh-my-zsh, you might like this. (Ok, yes, you can use this with bash too, just put it in the right place)

cat <<EOF >> ~/.zshrc
docker-start() {
    docker-machine start default
    eval $(docker-machine env default)
}

docker-stop() {
    docker-machine stop default
}

_docker() {
	if [ -z "${DOCKER_HOST}" ]; then
		eval $(docker-machine env)
	fi

	docker "$@"
}
alias docker=_docker
alias d=_docker
alias k=kubectl
EOF

And, finally, if you really, really, want to go all out and have docker start on your machine start (which sounds nice but your computer might not like it):

~/Library/LaunchAgent/com.docker.startup.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>EnvironmentVariables</key>
        <dict>
            <key>PATH</key>
            <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:</string>
        </dict>
        <key>Label</key>
        <string>com.docker.startup</string>
        <key>Program</key>
        <string>/Users/YOURUSERNAME/Library/Application Support/Docker/startup.sh</string>
        <key>RunAtLoad</key>
        <true />
        <key>KeepAlive</key>
        <false />
        <key>Disabled</key>
        <false />
        <key>LaunchOnlyOnce</key>
        <true />
        <key>StandardOutPath</key>
        <string>/tmp/docker.startup.stdout.log</string>
        <key>StandardErrorPath</key>
        <string>/tmp/docker.startup.stderr.log</string>
    </dict>
</plist>

~/Library/Appication Support/Docker/startup.sh

#!/bin/bash
docker_tag="Docker"

/usr/local/bin/docker-machine ls --filter "Name=default" --filter "State=Running" -f "{{ .Name }}"
machine_name=`/usr/local/bin/docker-machine ls --filter "Name=default" --filter "State=Running" -f "{{ .Name }}"`

if [[ ( -z "${machine_name}" || "${machine_name}" == "machine does not exist" ) ]]; then
    # Launch the machine
    echo "Starting \`default\` machine" | logger -t "${docker_tag}" -p "user.notice"

    # Redirect stderr to logger user.error
    # Redirect stdout to logger user.notice
    /usr/local/bin/docker-machine start default 2> >(logger -t "${docker_tag}" -p "user.error") 1> >(logger -t "${docker_tag}" -p "user.notice")
else
    echo "\`default\` machine is running" | logger -t "${docker_tag}" -p "user.notice" 
fi

Then,

sudo chmod +x ~/Library/Application Support/Docker/startup.sh
launchctl load -w ~/Library/LaunchAgent/com.docker.startup.plist

You can watch it via log stream | grep -i Docker.

• NodeMcu Mini D1 (WeMos D1 Mini) x5
• OLED 128x64 SSD1306 I2C Self-Luminous Display
• ELEGOO Electronic Fun Kit Bundle

The only thing that has come in the mail thus far was the immediately useful thing, the Mini D1 board. Getting that to work on my MacOS (running BigSur) wasn't uneventful but as far as these things go it was relatively easy.

First, download and install the Arduino IDE for MacOS.

So far, so good.

Second, Download and install the WeMos D1 Mini Drivers

MacOS is going to complain that it blocked an extension from being installed. I went down a long rabbit trail here worried that something wasn't installed. If you do some digging you'll find a ReadMe.PDF with the driver download that mentions usbserial.kext in /Library/Extensions and /System/Library/Extensions.

It seems like the extension was installed in Library/Extensions and wasn't in /System/Library/Extensions. Don't worry though, it turns out that it doesn't matter and things will just work. Don't waste time, like I did, trying to get the .kext into the System directory. I didn't have much luck but I think I probably needed to disable the System Integrity Protection. Meh.

Third, setup your Arduino IDE to work with the ESP8266:

Add the following URL to Arduino -> Preferences -> Additional Board Manager URLs:
http://arduino.esp8266.com/stable/package_esp8266com_index.json

Fourth, install the ESP8266 "board" using Tools -> Board: -> Board Manager. Search for "ESP8266" (you might need to do this a couple times) and then a single entry should pop up called "esp8266" by "ESP8266 Community".

Fifth, we need to do some patching to some random ESP8266 python code.

vi ~/Library/Arduino15/packages/esp8266/hardware/esp8266/2.7.4/tools/pyserial/serial/tools/list_ports_osx.py

Comment out lines 29 and 30 and add the following afterwards:

iokit = ctypes.cdll.LoadLibrary('/System/Library/Frameworks/IOKit.framework/IOKit')
cf = ctypes.cdll.LoadLibrary('/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation')

Finally, you can test your board by loading the Blink example sketch via File -> Examples -> ESP8266 -> Blink. Click the right arrow near the top left and your ESP8266 board should start blinking after a few moments.

Special thanks to AverageMaker's WeMos D1 Mini guide and the Arduino forums for making this setup take ~15 minutes.