2024: LLMS can (sometimes!) transform "turn left" -> bot.turn(90)

It's 2024. Elon just endorsed Trump. Illya just left OpenAI. And if you spend a month of your free time hacking on Llama3, you can get AI turn natural language commands into API calls to get a robot to do things (kinda!).

hard parts include:

  • robot API only ran on linux, Llama3 needed the beefiness of my Macbook, so had to painstakingly connect the computers over local wifi, repeatedly typing one local IP address into the terminal of the other computer

  • couldn't even use the "smartest" models of 2024 because there's no MCP or anything like it

  • Llama3 hallucinates like crazy. Tell it "make the robot twerk" and it'll spit out something like "bot.twerk("As a large language model, I cannot twerk", 55)

I learned:

  • What a system prompt was.

  • And that you could basically just toss API docs into the system prompt and get an LLM to ~understand~ the API (more or less).

2025: MCP is magical

It's 2025. DOGE is DOGE-ing. Thinking Machines has raised billions of dollars but they haven't even written a blog post yet.

In a week of free time, a mediocre software engineer can transform a toy robot API into a toy robot MCP and it kinda just works?

hard parts include:

  • Yn these earliest days, if you asked Claude to help you debug MCP and it says "I have no idea what MCP even is" (!!!)

  • You had to debug MCP a TON. The new claude desktop app always crashed on me in inscrutable ways

  • See the code above (here's a link)...these were still the days of hacking on existing starter repos made by a dev rel. This project in particular had a lot of uv related pain, because this was the first time I ever used that for dependencies.

  • There were bugs in the robot API that I just had to live with

I learned:

  • Other people were thinking similarly to what I had been up to in 2024 (as in, MCP just totally solved what I was trying to do with API in the system prompt)

2026: You can do anything (and everyone (including you) will shrug)

It's 2026. The singularity has maybe started according to people you respect? Last time you checked github you (claude) had 26,643 commits in the last year, so something is certainly happening.

You can hook the robot up to a MIDI keyboard and 'play' the xylophone with the keys, and then get it to play little loops:

You can tape a dry-erase marker to the robot's butt and get it to draw things:


When you get frustrated with how slow this is, you can you can try Jev(which doesn't work as well, but is a lot faster):

You can even delicately perk your computer in the rafters in an attic with the webcam facing down and the robot connected, giving the instruction of 'knock over the box', and it won't knock over the box, but it'll push it out of frame:

(time lapse! Each "step" is ~5-10 seconds)

hard parts include:

  • perch your computer in the rafters of your attic in the summer with the webcam recording and you will overheat your computer

  • definitely felt some LLM-melancholy doing all this.

  • the real world is higher friction than the world of bits. Hard to anticipate where the sim-to-real gap will be felt.

I learned:

  • robotics is hard! The real world is out there, and it's full of friction. I didn't think we lived in a simulation before, but if you do think we live in a simulation you should buy a Wonder Dash robot on ebay and hook it up to claude.

  • I felt sometimes like Claude's real-world liason. Making sure the robot was charged. Turning it on and off again to reset a bluetooth misconnection. Adjusting and readjusting the camera. A surprising amount of fussy work.

2027?

I may keep doing this. Honestly I'm at the place now where probably buying a new robot with finer-grained controls would be where the lowest hanging fruit would live.

I think taking a step back, and reflecting on it all as I write up this blog post, what I've gotten from all this is a pretty visceral sense of just how fast LLMs are improving.

2024 was not that long ago.