Key Highlights:
- Meta has announced its next big update in vision AI with the launch of SAM 3 and SAM 3D, pushing its Segment Anything project into a new generation.
- SAM 3 improves 2D understanding by making it much better at detecting, segmenting, and editing objects in images and videos using natural language prompts.
- SAM 3D goes a step further by turning flat images into fully rebuilt 3D versions, and it comes in two separate models: SAM 3D Objects and SAM 3D Body.
Meta has been one of the major names in the image segmentation industry. To catch you up, the AI giant launched the first Segment Anything Model (SAM) back in 2023. That model helped to cut out objects from an image in just a few clicks with simple clicks.
Well, Meta is now taking things one notch up with its new SAM 3 and SAM 3D models, which it launched today. In the announcement post, the company mentions that these new models can upgrade what AI can do in photos, videos, and even 3D spaces.
SAM 3 massively improves how AI understands and edits objects in 2D photos
First, the SAM 3 model builds on everything the original SAM model used to do. But this one promises far more accuracy and flexibility. The new model can detect, separate, and track individual objects in both photos and videos, and it does this more precisely than earlier versions. The biggest improvement, however, is how well SAM 3 works with natural language when it comes to specific objects in an image.
To give you an example, let’s say you can upload any picture and command the SAM 3 model to “change the blue shirt to a red shirt.” The model will be able to then figure out which shirt did you exactly mean. To make results precise, you can even give longer and detailed prompts, like “select the people sitting down who are not wearing a red hat.” Meta says that the SAM 3 works well with longer prompts and more creative tasks.
Meta is already using SAM 3 inside some of its own apps. In its AI video tool called Edits, Meta is experimenting with SAM 3 to help create new visual effects. The company also plans to use the model in Vibes, its TikTok-like short video platform, to make better editing and object-tracking tools for creators.
SAM 3D turns flat images into detailed, realistic 3D models from a single photo
In contrast to what SAM 3, which improves 2D understanding, SAM 3D is all about turning flat images into full 3D reconstructions. In simple words, the model can look at a single picture and generate the objects, animals, or even people in that image as detailed 3D models. Not to mention, this model has immense possibilities in the robotics, VR/AR, gaming, and digital design industry.
It’s worth noting that SAM 3D has two different models. The first, SAM 3D Objects, is built to reconstruct everyday items. You can think of 3D models of furniture, tools, gadgets, and even full indoor scenes. Even when you input a single image, it can predict depth, shape, and structure well enough to output a proper 3D design. The second model, SAM 3D Body, is trained specifically for rebuilding human bodies. It can estimate someone’s body shape, pose, and proportions from just one photo and generate a complete 3D model.
Meaning, you can hand it a casual selfie or a random image, and it will still attempts to output a full 3D reconstruction that is usable for animation, motion analysis, avatar creation, and other digital applications. Not to forget, SAM 3D also allows you to shift the camera angle or change the perspective of an entire scene.
As far as availability is concerned, both SAM 3 and SAM 3D can be used inside Meta’s new Segment Anything Playground, which is a simple web-based tool that anyone can try.
