So the title basically says it all. But I’ll explain a little chrther—

There are some models that were previously behind closed doors which would allow the user to gain some level of information about an image. You could feed any photo to the ai and it could describe in detail anything about the image to the user. I’m thinking this could have huge implications in the real world with live video feedback, and possibly with screen readers as well. Imagine screen recognition from apple, but the most perfect form of it. Imagine opening up a program with no accessibility, and simply having an ai walk you through where to place your cursor to access each element, and describe in detail any question you could have. Or better yet, speaking with the ai and asking it to perform actions for you.

I can imagine this being incredible in smart glasses, and really cool for video games. Asking the glasses to keep an eye out for bathrooms, asking it to look for sandwiches on the menu, or playing a video game and asking the ai to guide you around, etc.

I think the implications for this tool is going to be far reaching for us, but I was curious what you guys thought.