Meet Sébastien: Rocket Scientist turned Retouching AI Developer

And frankly, much of it is bullshit. Pure marketing.

Everything autonomous is not AI, and AI isn't always better. Writing a script to automate a process doesn't make it intelligent, artificial or otherwise.

Even where AI is possible, using neural networks isn't always the right approach. It's costly, needs to be applied narrowly, and often is less efficient than using a less buzz friendly—but more effective—method.

So why is "AI" such a powerful label?

Because properly applied, AI really is transformative. We know: we've seen incredible gains using AI to automate product image editing.

So when and how should AI be used in order to be effective? How are those decisions made? Who's making them?

To find out, I interviewed Sébastien Eskenazi, Pixelz' in-house R&D Computer Vision Scientist.

Sébastien's a bit of a renaissance man, and his background reflects it. Consider his education for a quick example: in France, he completed a Masters with a double major in Aeronautical Engineering and Mechanical Engineering, then went on to get a PhD in Information Technology.

Before Pixelz, Sébastien was a project manager for automotive software: making tools for self-driving cars, basically. Before that he was algorithmically comparing digital documents to uncover forgeries.

At Pixelz, he's been hard at work developing computer vision algorithms and neural networks to push the boundaries of retouching automation for tasks like drawing layer masks, smoothing garment shapes, and cleaning up photo backgrounds.

But enough from me. Let's hear from Sébastien himself about technology, AI, and why retouching is such a fascinating field for computer vision scientists.

Full Interview (Sébastien on AI in Retouching)

BACKGROUND

What's your tech background?

I'm a computer science engineer. I had a PhD in France for that, and after that I went to Vietnam to work for a big company called FPT software. In the U.S. I guess it's the equivalent of AT&T, maybe something like that. I was working for a customer who was making electronic chips, system on chips, for cars. I did a bit of work for computer vision, basically, and neural networks.

Did I see that you actually got a double degree in aeronautical engineering and mechanical engineering?

Yeah that's true too, that was a while ago. I finished high school and then I did three years of pure math and physics, and then I moved on. That's the standard course in France for engineers. And then I moved on to what's called an engineering school, equivalent to some university I guess. There I learnt a lot about aeronautical engineering, aerospace engineering, and mechanical engineering. I got two degrees like that, a master's and an engineering degree.

So for a while you were interested in flying, and then cars, is that right?

What happened is I went looking for jobs outside of France, but any jobs related to aeronautics or space outside of France is defense-related, so you have to be a national to be allowed to work on this. So it's impossible to find jobs outside of France. Then I moved on to PhD, kind of career reboot, because I was already a bit knowledgeable about IT. The thesis itself was related to security, IT security. Some people can abuse digitization services. For instance, they submit fake documents, and when they're digitized no one really checks them. So finding a way to check these documents, that's what I worked on.

AUTONOMOUS VEHICLE MODULES

Did you actually work on the autonomous driving aspect with cars?

Not really. I worked on tools for tools for people who would make autonomous cars.

Little modules that are abstracted out?

Exactly. Well, people make neural networks but they don't make neural networks out of nothing, they need a framework. So I was working on these frameworks basically, on these libraries that make neural networks. And I did neural networks so I was working on that. I also wrote a white paper on a proposed architecture for autonomous cars.

Every time you want to implement a functionality that is automated, you need to think about the big picture and all the aspects of it. When you're talking about autonomous cars, it's so complex, you really need to think about the framework and how you're gonna put it together. And it's really the same thing here at Pixelz actually, every time we automate something it's not just a piece of software on its own. You've got humans, you've got photo editors, you've got customers, you've got customer accounts, you've got lots of people involved. We need to make sure that the software we make works well with all these actors around it. So that's the challenge, you know?

PIXELZ TEAM

How do you approach AI development at Pixelz?

I am in the research and development team. We kind of work on whichever topic we want, but we do have priorities based on which tasks require the most time from the photo editors, and which tasks we can automate, in order to optimize the company of course. This gives us priorities. We have skills ["skills" are specific tasks photo editors perform on our assembly line], and so for each skill we know how long it takes, and based on what the skill is about we know if we can automate it or not.

We start from there, and each one of us picks a topic and says, "Okay I'm gonna automate this one," or, "I'm gonna try to improve this one." So we start like that, and then each one of us takes care of the whole thing. Depending on how you approach the system, some will just focus on their algorithm. For me I prefer to focus on the whole process, because if I make a tool but it's not used by the photo editor, it's useless.

Then I'm in charge of actually looking at the whole process in which my tool will be involved. I discuss with several actors, like Janus [Pixelz CTO] and the platform team. I also sometimes go to check with the photo editors themselves, observe how they use the tool or not. Also with the data science team to check the tool performance. We really look at the whole picture.

DEV PROCESS

Are your tools always neural networks?

No, definitely not. So you've got a functionality you want to implement. There are two issues with any functionality: one is, how easily you can express it in a way that's understandable for a machine. If it's very difficult to express, then a neural network will be more suited than a pretty fine algorithm. But sometimes algorithms are more efficient than neural networks, because neural networks need a lot of training data and they can be biased with the data set that we use for training. Sometimes plain computer vision algorithms are better suited. They're still not very easy to implement, we're still talking really high tech stuff. Not many people could do that.

Especially like when you do photo editing you deal with such a variety of images that that's the second topic. One is, okay, can you express your functionality? And the second is how many corner cases are you gonna have? The number of these corner cases, these things you see rarely, that also makes it difficult. Based on these two things, how easy is it to express and how many corner cases you have, you can decide which approach is best like should I use a neural network, or should I use a specific algorithm, or a mix of both? For instance we can use an AI to detect some parts of the image, and then you're gonna use computer vision algorithms like some kind of very advanced filtering (filtering is very basic), but some kind of very advanced filter to process the regions that have been detected by the AI. So sometimes it's a mix of both also.

Some other things we can look into is you use computer vision to provide more elaborate and more useful input to the AI. You can do a combination of those also.

AI & COMPUTER VISION DEFINITIONS

What exactly do you mean by "computer vision?"

You know there is marketing and then there is marketing, it's not polite, but it's called bullshit. There is a lot of that unfortunately. Oh, okay, so we need to clear two things. One, AI is nowhere near intelligent. It's definitely not intelligent.

What is AI? It's you take a bag of grains of sand, and you throw it on the ground. At first what you get is a big mess. And then you take a training data set, and that training data set is like a small finger that's gonna arrange the grains of sand one by one. And if you do that long enough then at the end you end up with a pattern that's meaningful. This is AI, and this is what you train, and then with this pattern it means something.

So an AI is just a bunch of parameters that we have optimized, and it's not clever at all. It has no ability to learn after it's been trained, except in some cases that have been made specifically for that. It will not create anything by itself, although some networks have been kind of customized to do similar things to that. So it's not that clever.

Then, we have many other algorithms that are also not clever but still very, very useful. We've got, what's called classifiers. And AI will take an image and tell you, "This is a dog," right? But before AI, we could also do that actually. We don't need a neural network to do that. It's just not as good.

So what we do is we take the image, and then we will apply let's say a set of filters. A filter like we're looking for patterns basically, certain patterns like lines, like circles, other stuff like that. So we apply these filters on it, and we get from the output of the filters what's called "features." Then we do some kind of mathematical processing on these, and in the end we can obtain a value that will tell us, "Oh, this is also a dog." So you don't really need a neural network to do that, it's just neural networks perform better.

"Computer vision" is image processing. It includes all these branch of algorithms that are not neural networks, as well as some neural networks. It's more generic. So it includes all kinds of filtering, it can be color filtering, it can be filtering like you know blur filter? This kind of image processing is computer vision. In photography, you know that lens introduces a distortion. We can remove this distortion with computer vision, we can do that.

For instance, when you have several images of the same object, we can make a 3-D model of this. You don't need AI for that, you need computer vision. Computer vision is all about this, basically. So we have many algorithms. For instance, it's capable of identifying patches of uniform color, which is called super-pixels. This is computer vision, it's a whole set of algorithms, not just AI.

So AI is kind of just a catch-all for anything where you're using a neural network to do your filtering. And computer vision is the rest of it.