Robot Manipulation
Driving a robot arm with AI, from plain-language instructions.
| My role | Builder |
|---|---|
| Started | 2026 |
| Status | Public repository |
| Field | AI & robotics |
| Website | github.com/renan-devil/robot-manipulation |
Robot Manipulation explores how to drive a robot arm with AI: you give an instruction in plain words, such as “Put the red cube on the blue tray”, a model looks at a camera image and decides what to do, and the arm does it. Today it runs entirely in simulation, in MuJoCo, on a laptop and for free.
The design splits a brain from a body. The brain is a vision-language model that only points at things in the image — where to pick, where to place — and is swappable between an offline colour finder, Google DeepMind’s Gemini Robotics-ER and Mistral. The body turns those two points into motion with classic robotics: a ray from the camera to find the 3D point, inverse kinematics to find the joint angles, and a smooth trajectory. The brain never sees joints or metres and the body never sees words, so either side can be replaced without touching the other.
The repository includes a phased build plan, each phase spending a little more only once the previous one has proven its point — from simulation to cloud models, a remote real robot, a robot arm on a desk, and finally end-to-end vision-language-action models — and a survey of what Google DeepMind, Mistral, NVIDIA and others have released.
Milestones
- 2026 — Phase 0 published: simulated arm, swappable AI brains and a build plan.