As it turns out, a rabbit with an AI-generated top hat was just the tip of the iceberg.

Google is the latest phone vendor to announce AI-based photo editing tools this year. Samsung’s slightly troubling but mostly delightful Sketch-to-Image feature and Apple’s much more benign-seeming Image Playground, due out this fall, are the first to come from Google. The Pixel 9’s answer is a new tool called “Reimagine,” and after using it for a week with some of my colleagues, I’m more convinced than ever that none of us are prepared for what’s coming.

Reimagine is a logical extension of last year’s Magic Editor tools, which let you select and delete parts of a scene or change the sky to look like a sunset. It was nothing surprising. But Reimagine doesn’t just go one step further—it kicks the whole door down. You can select any nonhuman object or part of a scene and enter a text prompt to generate something in that space. The results are often very convincing and even scary. The lighting, shadows and perspective are usually true to the original photo. You can of course add fun things like wildflowers or rainbows or whatever. But that’s not the problem.

A few of my colleagues helped me test the limits of Reimagine with their Pixel 9 and 9 Pro review units, and we managed to create some very disturbing things. Some of these required some creative inspiration to get around the obvious guardrails; if you choose your words carefully, you can get it to create a reasonably convincing body under a blood-stained sheet.

During our week of testing, we added wrecked cars, smoking bombs in public places, sheets that appear to cover bloody corpses, and drug paraphernalia to the images. It looks bad. As a reminder, this isn’t any special software we used on purpose – it’s all built into a phone my dad was able to buy from Verizon.

When we asked Google for comment on this issue, company spokesperson Alex Moriconi responded with the following statement:

Pixel Studio and Magic Editor are helpful tools that let you unleash your creativity with text generation into images and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts. This means they may create content that may be offensive if the user asks them to. That said, not everything is allowed. We have clear policies and terms of service about what types of content we do and don’t allow, and put safeguards in place to prevent abuse. Sometimes some prompts can challenge the safeguards of these tools, and we remain committed to continually improving and refining the safeguards we put in place.

Certainly, our creative requests to bypass filters are a clear violation of these guidelines. It’s also a violation of Safeway’s policy to label organic peaches as conventionally grown at the self-checkout, although I don’t know anyone who would do that. And someone with the worst intentions doesn’t care about Google’s terms of service either. What’s most troubling about all of this is the lack of robust tools to identify this kind of content on the internet. Our ability to create problematic images is far ahead of our ability to identify them.

When you edit an image with Reimagine, there’s no watermark or other obvious way to tell that the image is AI-generated—there’s just a tag in the metadata. That’s all well and good, but standard metadata is easy to remove from an image by simply taking a screenshot. Moriconi tells us that Google uses a more robust tagging system called SynthID for images created by Pixel Studio because they’re 100 percent synthetic. Images edited with Magic Editor, however, don’t get those tags.

Of course, manipulating photos is nothing new. People have been adding weird and misleading things to images for as long as photography has existed. But the difference is that it’s never been easier to add those things realistically to your photos. Just a year or two ago, adding a convincing car crash to an image would have required time, expertise, knowledge of Photoshop layers, and access to expensive software. Those obstacles have disappeared; now all you need is a bit of text, a few Moments, and a new Pixel phone.

Plus, it’s never been easier to spread misleading photos quickly. The tools to manipulate your photos convincingly are right on the same device you use to take them and post them for the world to see. We uploaded one of our “reimagined” images to an Instagram story as a test (and quickly removed it). Meta didn’t automatically label it as AI-generated, and I’m sure no one would have noticed if they had seen it.

Who knows, maybe everyone will read and follow Google’s AI guidelines and use Reimagine to add wildflowers and rainbows to their photos. That would be nice! But just in case they notit may be best to be a little more skeptical about the photos you see online.