Prototype a hand-tracking Mac app with Codex and Apple Vision
Codex + SwiftUI + Apple Vision + Computer use
I wanted to try controlling my Mac with hand gestures through the camera. I described the idea to Codex, and it built a native app using SwiftUI for the interface and Apple Vision to recognize hand positions. It included a practice area so I could test gestures before using them to control the computer.
Then I tried it. Nothing moved. I told Codex what I was seeing, and it used computer use to inspect the running app and macOS permissions, then adjusted the controls. By the end, the prototype was recognizing my pointing gesture.
The useful part was that I could try the interaction and have the same agent look at the app with me. That got the idea far enough along to test what a hand-controlled interface could feel like.