CHANNELS

Aerospace

Automotive

Automated & Autonomous Vehicles

Battery & Electrification Technology

Defense

Drone TV

Electronics

Manufacturing & Prototyping

Materials

Medical

Motion Control

RF & Microwave Technology

Robotics & Automation

Sensors & IoT

Test & Measurement

Aerospace
Automotive
Automated & Autonomous Vehicles
Battery & Electrification Technology
Defense
Drone TV
Electronics
Manufacturing & Prototyping
Materials
Medical
Motion Control
RF & Microwave Technology
Robotics & Automation
Sensors & IoT
Test & Measurement
Aerospace & Defense
Search Results

Meet FRIDA — An AI-Powered Robot Creating Art

Carnegie Mellon University’s Robotics Institute has a new artist-in-residence: FRIDA, a robotic arm with a paintbrush taped to it. FRIDA — named after Frida Kahlo, stands for Framework and Robotics Initiative for Developing Arts — uses AI to collaborate with humans on art. Specifically, it uses AI models akin to those in ChatGPT and DALL-E 2.

"FRIDA is a project exploring the intersection of human and robotic creativity," said project member Jim McCann  . "FRIDA is using the kind of AI models that have been developed to do things like caption images and understand scene content and applying it to this artistic generative problem."

Topics:
AR/AI Electronics & Computers Robotics

More From SAE Media Group

    Transcript

    00:00:05 >> Jim McCann: These days, a lot of what we automate is really, really precise, but painting is not. FRIDA is a project exploring the intersection of human and robotic creativity. This is a platform to look at how robots can use imprecise tools. And especially the way FRIDA plans and works is imprecise.

    00:00:27 >> Peter Schaldenbrand: We've got so many tools that are new like DALL-E, but they work in the pixel space. So you can't really directly apply a lot of that to the real world. But FRIDA is an instance where you're creating an image in the real-world using machine learning.

    00:00:41 >> Jean Oh: FRIDA stands for Framework and Robotics Initiative for Developing Arts. People have very high-level idea about what kind of things that they want to paint or express in art. Very few people actually have a very concrete idea what the final artifact would be,

    00:01:01 but they start with some semantic goals. >. Peter: You could describe what you want to see with language. You can give it an image that you like for its style. You could have paint just a photograph that you like, or any combination of those. The system will tell you which colors to mix. So the human still has to mix the paints right now.

    00:01:17 It'll show you pictures of the colors, you mix it and then you just provide it to the robot. So it has an understanding of language and it has an understanding of images. It can compare those in a latent space. >> Jim: FRIDA is doing something that most robots don't. FRIDA is using the kind

    00:01:32 of AI models that have been developed to do things like caption images and understand scene content and applying it to this artistic generative problem. And FRIDA is doing that by planning in a semantic space, a space of meaning instead of a space of outputs. >> Peter: FRIDA builds the simulation by testing a bunch of brushstrokes.

    00:01:49 So it'll grab a little paint, make a brushstroke. And it measures and tries to model the interaction between its trajectory with the brush and how that brushstroke appears. So with that relationship, it can predict how future brushstrokes will look. >> Jean: And it starts actual painting.

    00:02:08 And then uses the camera sensing to observe the canvas. And then take the current canvas and think about the semantic again and then try to optimize. >> Peter: Currently you specify the number of brushstrokes and if you're unsatisfied, you can run it again and just keep adding more brushstrokes.

    00:02:26 >> Jim: It's going off and doing its own thing and it's trying to get to the meaning you ask it to get to. >> Peter: I think the coolest thing about FRIDA is that its imprecision contributes to its content creation. So because you're constantly taking pictures and reconsidering your goals,

    00:02:42 If you mess up really bad in the beginning, like the robot really screws up, it can riff off of that. And it'll work with its failures and it will alter its goals. And I think that's just how a person paints. >> Jean: We wanted to develop FRIDA to mimic that process, which is a very iterative, continuous process to get to the final stage.

    00:03:05 >> Jim: Is FRIDA using artistic expression? On one hand, we could be very reductionist and say, again, artistic expression is this mysterious thing we don't understand. We understand how FRIDA works. It's juggling numbers in tables. And those numbers were derived from large collections of images and words. So, therefore, there can't

    00:03:24 be any room for artistic expression. On the other hand, we could turn this around and say, well, what does an artist to do but distill some subset of the zeitgeist, distill some subset of what people around them are saying and doing, and turn that into an expression. In which case that's

    00:03:40 exactly what FRIDA is doing. >> Jean: Personally I wanted to be an artist. Now I can actually collaborate with FRIDA to express my ideas in painting. >> Peter: The goal of the system is really make it feel like when the painting is done, the person who was using the system can feel creative ownership. >> Jim: I've really been surprised by how

    00:04:00 compelling some of the images are that FRIDA creates, maybe just because of their very physicality. >> Jean: So there is one painting that was about the women's rights. It's like the woman looking far ahead. That's one of my favorites. >> Jim: I like FRIDA's canonical downtown Pittsburgh view. That is vibrant,

    00:04:24 it's exciting, and it's Pittsburgh, and I like all those things together. >> Peter: There's this one of a frog ballerina, that I think turned out really nicely. It's subtle, but once you get it, it's really silly and fun. And I think the surprise of what FRIDA generated based on my inputs was really fun to see.

    00:04:41 >> Jim: Painting is interesting, but we don't really have a lot of interesting things that we make with painting other than paintings. So we want to move to sculpting. Because sculpting has a lot more actual real-world applications. >> Peter: So I see this as contributing to fabrication. I think it'd be really fascinating

    00:05:00 to just make a lot of the stuff that we fabricate, like, more artistic and interesting, less robotic. >> Jim: The far term is more about trying to address something I think that we do wrong with robots, which is, we get really obsessed with precision and we forget that

    00:05:15 one of the great powers of planning is the ability to deal with variability. I see this just in the cost of robotic systems because we invest so much in making them precise. I see this in the way that there's a disconnect between sophisticated planning and sophisticated hardware. And I think that trying to

    00:05:30 build this succession of systems which work with variation that work around their own imprecision is pretty cool.