One thought I can’t shake after watching Apple’s VisionOS stuff is how much I want the eye tracking functionality in their headset. From what I’ve heard from the few people who have actually tried the device, it works amazingly well and some described it as the most magical part of the whole experience. For anyone who hasn’t seen it there are four infrared cameras on the inside of the headset that are pointed at your eyes. They can very precisely track exactly the point your eyes are looking at in the VR/AR world. In VisionOS to click on something you look at it and then touch your thumb and pointer fingers together to “click”.
I don’t want the whole headset, or a whole new OS. Just give me some eyeglasses with the infrared cameras and let me replace my mouse with them. Imagine a world in which your hands never have to leave the keyboard because left and right click are just keys on your keyboard and your computer knows which element you are looking at to click on. You can sit with better posture because you don’t have to move your arms between the mouse and keyboard. You can fly around your operating system with incredible speed, because your eyes move way faster than your arm and fingers. You don’t need to learn as many keyboard shortcuts because looking at something and pressing one key is much faster than contorting your hands to get that arcane shortcut. It feels like a desktop power-user’s dream.
Apple made a good point that what your eyes are looking at is traditionally private information, and they took pains to conceal what your are looking at from the applications your are interacting with (of course their closed-source OS has access to that data). In my ideal device I think I’d be ok with forgoing the “hover effect” on the element or button you are looking at in exchange for my eye position only being transmitted to the operating system when I click.
I did some web searching for people working on this and found a few products, but the consumer priced ones like the Tobii Eye Tracker 5 are apparently not accurate enough to replace a mouse. There is some medical-grade hardware aimed at making computers accessible for those with motor disabilities that looks better but it costs around $15k. I also found that Apple has a head tracking mouse built into macOS as an accessibility feature, but it wasn’t precise enough to be quicker than a mouse in my testing.
I think it will take a couple years but surely this is the future of pointer devices for general computing. The unfortunate casualty will be FPS games, but it’s a tradeoff I’m willing to make for the greater good.
mplorentz / Matt Lorentz
npub16z…l92l7
2023-07-13 21:50:30
Author Public Key
npub16zsllwrkrwt5emz2805vhjewj6nsjrw0ge0latyrn2jv5gxf5k0q5l92l7Published at
2023-07-13 21:50:30Event JSON
{
"id": "e46c91fef2bd75044cc157f676c73a931d54061a840fd2e35e1889cbd6d2d22d",
"pubkey": "d0a1ffb8761b974cec4a3be8cbcb2e96a7090dcf465ffeac839aa4ca20c9a59e",
"created_at": 1689285030,
"kind": 1,
"tags": [],
"content": "One thought I can’t shake after watching Apple’s VisionOS stuff is how much I want the eye tracking functionality in their headset. From what I’ve heard from the few people who have actually tried the device, it works amazingly well and some described it as the most magical part of the whole experience. For anyone who hasn’t seen it there are four infrared cameras on the inside of the headset that are pointed at your eyes. They can very precisely track exactly the point your eyes are looking at in the VR/AR world. In VisionOS to click on something you look at it and then touch your thumb and pointer fingers together to “click”.\n\nI don’t want the whole headset, or a whole new OS. Just give me some eyeglasses with the infrared cameras and let me replace my mouse with them. Imagine a world in which your hands never have to leave the keyboard because left and right click are just keys on your keyboard and your computer knows which element you are looking at to click on. You can sit with better posture because you don’t have to move your arms between the mouse and keyboard. You can fly around your operating system with incredible speed, because your eyes move way faster than your arm and fingers. You don’t need to learn as many keyboard shortcuts because looking at something and pressing one key is much faster than contorting your hands to get that arcane shortcut. It feels like a desktop power-user’s dream.\n\nApple made a good point that what your eyes are looking at is traditionally private information, and they took pains to conceal what your are looking at from the applications your are interacting with (of course their closed-source OS has access to that data). In my ideal device I think I’d be ok with forgoing the “hover effect” on the element or button you are looking at in exchange for my eye position only being transmitted to the operating system when I click.\n\nI did some web searching for people working on this and found a few products, but the consumer priced ones like the Tobii Eye Tracker 5 are apparently not accurate enough to replace a mouse. There is some medical-grade hardware aimed at making computers accessible for those with motor disabilities that looks better but it costs around $15k. I also found that Apple has a head tracking mouse built into macOS as an accessibility feature, but it wasn’t precise enough to be quicker than a mouse in my testing.\n\nI think it will take a couple years but surely this is the future of pointer devices for general computing. The unfortunate casualty will be FPS games, but it’s a tradeoff I’m willing to make for the greater good.",
"sig": "90e8522aad087f01340dab065ab0f99720e693feff71070e13a91ce987e949e42b3dace14d0ba9ab0ed34d03e94a75c753e45afdae5a77d236625962da4fcc10"
}