Filter by tag

An art forger’s guide to image processing

Let’s talk about art forgery. Millions of dollars and the reputations of experts and auction houses ride on being able to tell an authentic piece from a ersatz one. For a forger, a perfect facsimile (or just one that passes muster) makes them tons of money. Getting caught means jail time.

But how do forgers paint a plausible Leonardo or Vermeer or Picasso? The answer has to do with misdirection: the patterns your attention is drawn to and away from. Meanwhile, the connoisseur needs to combat their natural biases and learn to find tells in the forgery.

This back and forth just so happens to mimic the interplay in a powerful approach to Deep Learning known as Generative Adversarial Networks. And we’ll get to those in a bit. First, though, it’s important to understand how a professional discerns a genuine article from a phony.

What your brain lets you see and what it doesn’t

When experts examine a work of art, they see not just the piece as you or I might at a museum, but the brushstrokes, composition, iconography, pigments, shading, style, and how the piece “fits” into the oeuvre of the artist him- or herself.

Looking at a Leonardo, they notice if all the brushstrokes came from a left-handed artist and whether a face is “too sweet” (Leonardo made his subjects tough). When they examine a purported Jackson Pollock, they know that he didn’t use some colors (like gold) and that while he explored synthetic paints, he didn’t use acrylics.

Jackson Pollack’s Autumn Rhythm

No one is born knowing how to look at art this way. It develops out of experience—reading, talking, sketching, and most of all looking and looking and looking. Human brains are terrific at pattern matching, so good that most art experts looking at a potentially forged piece talk about how they feel something right or wrong in their gut and then go about trying to find which features they can use to explain this to someone else.

To a certain extent, then, the connoisseur looks at a new piece and compares it to their internal training data: a history of examination and expertise. Machines copy this approach. If you can give computers enough training data, they will uncover patterns and, depending on the machine learning algorithms, even tell you how confident they are and which features they’re using. An artificial intelligence system will have biases based on the training data—basically, what’s overrepresented, what’s underrepresented.

But unlike people, they don’t seem to run into two types of biases that core to human cognition: a computer won’t hope that it’s discovered a work of great value. They’re not praying to find an newly unearthed Gauguin.

Another advantage is that they readily change their opinion based on new facts. If on Monday it says it’s real and Tuesday you give it more data or a different perspective, it will update its conclusion immediately.

To put it another way, expert humans sometimes see what they want to see and have trouble upending that conclusion.

Beyond the Now, until the last few decades, all verification of an artwork’s authenticity was done by connoisseurs. The best forgers know that experts will peer so closely at the art that they will smell any scent of coffee they use to create brown spots; the experts will smell linseed oil and know that should have disappeared long ago from a truly old painting. Lesser forgers may or may not ignore these details, but master forgers get even the smallest details right. They match materials—choosing old canvases, beetle-ridden frames, and historically correct pigment so that they can pass common dating tests.

Authenticators also check what’s called “provenance”, the history of the work. This is the most common way to authenticate, but it can also be exploited in what’s called “the provenance trap”. Forgers have found, time after time, that if they can make the history of an object look solid, people won’t look that closely at the art itself.

If you can’t trust an expert, who can you trust?

Can art experts use technology to find artists’ non-metaphorical fingerprints on known works and use those to validate proposed works? As we get access to more processing power, better data, and smarter algorithms, the answer, increasingly, is yes.

That doesn’t mean we should trust these solutions implicitly though. Take the story of Peter Paul Biro. He recently invented new forensic techniques to identify frauds and real pieces and became famous in the art world for the way he often upended expert expectations. His story is told in a documentary called Who the #$&% Is Jackson Pollock?

But the documentary ends too early.

Since it was released, a growing body of evidence has emerged that the Biro family haven’t been merely restoring and consulting. What they sold as technology may have actually involved falsifying chemical tests and copying artists’ true fingerprints into new works that they didn’t make.

Sliding from restoration to forgery seems to happen isn’t uncommon. After all, these are people have spent hours and hours looking at art. They know the patterns to emulate and have the skill and practice to do so. For example, after World War II, Lothar Malskat was hired to restore the medieval frescos in a north German cathedral that had been damaged by fire. When they were unveiled, they were so wonderful that the German government put them on stamps.

Malskat then announced they weren’t restorations but original works by him. And since no one believed him, he sued himself.

His proof included a North American bird in the frescos that couldn’t have been in Germany in 1300: a turkey. That’s called a timebomb—a way for a forger to prove a work is their own if they ever want to. It would be an undertaking to detect that sort of thing by computers, though one could probably use facial recognition to have found Marlene Dietrich who Malskat also hid in his fresco.

But it’s not as if only restorers go for forgery. Before Michelangelo was Michelangelo, his first work was a fake. He paired up with a crooked art dealer to make a statue. The dealer buried it in his garden and then unearthed it as an ancient piece. And Bernard Berenson may be among the best connoisseurs of history—he also seems to have used his expertise to authenticate inauthentic works that made him money.

So who can you trust? Whose eyes? And if everyone from connoisseurs to Michelangelo has slipped into forgery, can you ever really know? What if, instead, we trusted a computer?

Teaching a computer to see

In the intro, I talked about one flavor of deep learning is called Generative Adversarial Networks. What’s interesting about this technique is that it involves a race of a “judge” against an “artist” or in our terms, it’s something like building (a) a connoisseur who tries to detect which images are real (that is, like their training data) and (b) a forger who tries to create new images that can fool the connoisseur. Putting these in an adversarial relationship helps the overall system figure out which features matter: a feature that is known by only the forger will result in the connoisseur losing; a feature known only by the connoisseur will help them beat the forger.

This interplay makes both the “judge” and the “artist” more apedt in their roles. It makes the “artist” more creative and the “judge” more observant. And the more times a computer judges correctly (or updates after judging incorrectly), the better chance it has at getting it right the next time.

Art draws our attention to the content in an image, but to its style and form, too. Recognizing objects and subtle traits is at the heart of most commercial applications of computer vision. But when you see an image or a video, you don’t reduce it to a bunch of nouns. You take this unstructured information and put it into some framework based on the context you’re seeing it in.

And while many of the business uses of computer vision have to do with rapid understanding of vast amounts of data, there is another piece here: overcoming cognitive biases in human brains. As story-after-story shows, our brains are amazing at pattern-matching, but confirmation bias is a clear blindspot. Forgery works because of this trait, not in spite of it. The best way to spot forgery (or to do any fine grained, complex visual task) just might be combining experts and machines.

Specialization and seeing more clearly

Artificial intelligence is not ready to handle all of the diversity of the world, but it does very well in closed domains. That is to say, we’re not yet good at general AI but we’re getting quite good at specific-use AI.

This limitation is in some ways a blessing since it forces us to define what specific user goals the AI is going to help with. How you categorize something depends on what you are looking to do with it.

For example, imagine a picture of a beach. That could image could be a cherished memory of a great vacation, used to so help an end-user them connect with others. It could inform a visual algorithm by helping a user shop for a vacation or a swimsuit. It could be used to measure environmental degradation like erosion or a whole host of other uses. And it would be the same picture for each.

Getting back to the crux of our piece though: for most people, forgers are non-threatening, appealingly rakish criminals. There’s something egalitarian in the way their work can potentially deflate puffed-up experts. But don’t write off the connoisseurs so quickly. They often deeply love what they see and while they usually report an immediate gut instinct, there are also stories of them sitting, staring at a work and suddenly sensing a puzzle: something is not quite right.

The thing is, forgers and connoisseurs both teach us how to look. The promise of deep learning and other approaches to computer vision is to map all of the rich, complicated things happening in an image to a set of labels. To find the features that are behind the choices a forger makes with each brushstroke and the features behind the intuitions that guide an expert viewer.

For a person to distinguish a real Leonardo or Picasso from a fake, they need to see lots of examples. Computers can process examples fast but they have to enough training data, annotated accurately. Computers do not pick out patterns for their loveliness or in order to resolve a discomfort in the pit of their stomachs. Embedded in choices of categories, training data, and algorithms, computers must have the right guidance for what they should be learning and how they should see. And if we combine the best of both approaches (and learn how to subjugate our biases), we’ll learn to see a whole lot better.

Tyler Schnoebelen

Tyler Schnoebelen

Tyler Schnoebelen is the former Founder and Chief Analyst at Idibon, a company specializing in cloud-based natural language processing. Tyler has ten years of experience in UX design and research in Silicon Valley and holds a Ph.D. from Stanford, where he studied endangered languages and emoticons. He’s been featured in The New York Times Magazine, The Boston Globe, The Atlantic, and NPR.