In the fall of 2015, celebrated visual effects whiz Pablo Helman was in Taiwan celebrating Thanksgiving with Martin Scorsese. The 24-year veteran of Industrial Light & Magic, the company founded by George Lucas at the onset of the Star Wars franchise, was midway through production on the director’s Jesuit missionary saga, Silence, for which Helman had to digitally re-create the enormity of St. Paul’s College of Macau. But over holiday dinner, Scorsese began pitching Helman on a different film entirely.
It was another adaption, this one based on I Heard You Paint Houses, Charles Brandt’s biography of mob hit man and supposed Jimmy Hoffa murderer Frank Sheeran. Much like Silence, the story was expansive, though instead of spanning geography (Portugal to Japan), the movie would stretch across years (approximately seven decades). Here, Helman — whose credits as a VFX supervisor include The Mummy, The Bourne Supremacy, and Star Wars: Attack of the Clones — wouldn’t simply be manifesting background imagery, he would working on the very face of the story: Robert De Niro, playing a man expected to appear onscreen as young as 24 in one scene and as old as 80 in another.
Helman read the 170-page script for The Irishman in one sitting soon after. He was hooked. Little did the Argentine effects artist know he would spend the next four years creating and perfecting an untried method of turning back the cinematic hands of time, expending the sweat equity of hundreds of technicians across Europe, Asia, and North America in the process. “We were all very nervous about the undertaking,†Helman says. “It was huge. And we all knew what the stakes were.â€
How exactly do you make old men young again? The question has bedeviled Hollywood since aging actors’ sagging features first began flickering across 30-foot theater screens. When it came to de-aging De Niro, Joe Pesci, and Al Pacino for The Irishman, the $140 million Netflix production opted for a specific kind of fountain of youth, created from artificial-intelligence software, first-of-its-kind motion-capture technology, and an experimental three-camera rigging system that rendered the Oscar-winning trio of septuagenarian actors as eerily smooth-skinned incarnations of their younger selves.
The Irishman is now one of the presumed front-runners for a Best Picture Oscar nomination, having already taken the awards for Best Film from both the New York Film Critics Circle and the National Board of Review. Earlier this month, it landed on the American Film Institute’s year-end best-films list in addition to scoring a Screen Actors Guild nod for Best Cast in a Motion Picture and a slew of Golden Globe nominations. It’s also been included on the Oscars short list for Best Visual Effects. Ahead of the official nominations, Helman walked Vulture through his years-long journey to awards season — and all the second-guessing that came with it.
Step 1: Proof of Concept
The Irishman had officially been in development at Paramount since 2007, long before Helman and Scorsese broke bread in Taiwan. In the early going, it seemed like no one could crack the script’s old-to-young-and-back-again timeline issues. Scorsese knew De Niro, who had been firmly attached to the project for years, would never tolerate having his face plastered with dozens of motion-capture markers or wearing a helmet affixed with a mo-cap camera, the prevailing methods for turning actorly emoting into computer-generated imagery.
Instead, Helman seized upon the idea of using a three-camera rig, with the primary camera in the center flanked by two Alexa mini-cameras, to gather the gigabytes of digital information motion-capture dots might have otherwise harvested. He would then run the footage from the three cameras through a rudimentary version of ILM’s proprietary de-aging software (called Flux or Face Lux) to create a new three-dimensional, virtually rendered actor that could be digitally manipulated — or what the supervisor calls a “performance in 3-D space.†The eventual de-aged rendering would be informed by both light sampling on set and scanning sessions with the actors, but the process involved no actual facial markers.
To test the method in 2015, De Niro, then 74, traveled to New York and, under Scorsese’s direction, filmed a beat-by-beat re-creation of the Christmas-party scene from Scorsese’s Oscar-nominated 1989 crime drama GoodFellas. “After ten weeks, we computed all the information we had and put it together,†Helman recalls. The results were conclusive: De Niro had been miraculously de-aged to a credible facsimile of his 46-year-old self. “We made him look like he does in GoodFellas — and that got the movie green-lit,†Helman says.
Step 2: Hurry up and Wait
While Scorsese wrapped postproduction on Silence, Helman and his ILM team continued to refine Flux and the three-camera motion-capture shooting method for another year, eventually opting for infrared tech on the two secondary cameras. However, in a surprise move in February 2017, Paramount sold North American distribution rights to The Irishman to Netflix. The studio had become reluctant to shoulder the film’s relatively large budget in an era when tentpole and superhero movies have overwhelmed adult dramas at the multiplex. Thankfully, a streaming platform with deep pockets was willing to take the risk.
Over that year and change, Helman and his crew became increasingly convinced they were onto something groundbreaking. “Because we didn’t have markers. We had a different way of capturing the performance that was actually more faithful to the movements of the face,†he says. “It usually depends on how many markers you have, but at some point, you’re going to run out of space on your face to put those markers. Even if you have 200 markers on your face, you’re not going to be able to capture all the nuances that are under the eyes and next to the nose and in the eyebrows. Now, the quality is much better. You are not tied to the resolution of 200 points. Now, you have millions of pixels that you can go into.â€
By the time filming on The Irishman commenced in New York City in September 2017, Helman and his team were confident in their no-marker strategy. But in an unusual move, Scorsese permitted Helman and his crew to continue perfecting the setup over the 108-day shoot, running rough cuts of the film through Flux along the way.
Step 3: The Adoration of the Ball
Scorsese responded to subsequent shifts in the new mo-cap rigging system with what can fairly be described as grandpa-like, “How does this iPhone work?†befuddlement. “[There] was this ritual after each time we did a take,†the 77-year-old director said at a press conference following the movie’s Hollywood premiere. “Everything would stop and a young man would come out and he’d have a board with some kind of design on it. And he’d move it a certain way. And I’d be looking: What the hell is this? Then there was dead quiet, and he goes back and then comes back with a silver ball, and the ball moved, and I would call it ‘the adoration of the ball.’ I realized it was like a benediction of some kind.â€
According to Helman, the mirrored sphere was trotted out for 30 seconds in between takes to collect lighting data but insists it did not significantly slow or interfere with Scorsese’s shooting style nor with Pacino and De Niro’s improvisation-intensive acting methods. Weird as it may have seemed to an old movie salt like Scorsese, the ball played a crucial role in measuring what Helman calls the “color temperature of the lighting for the scene†required to calculate 3-D models of the actors.
“It’s something that becomes a ritual,†Helman says. “After a regular, non-visual-effects movie, after a setup is done, you move on really quickly. But we couldn’t move on; we had to stay put for 30 seconds. And there was a silence. Everybody was ritualistically quiet.â€
Step 4: Computer Learning
Before production on The Irishman began in earnest, the ILM team spent two years cataloguing a digital library of images and sequences from Pesci, Pacino, and De Niro’s filmographies, homing in on the ages they would portray in The Irishman. Upon completion of principal photography, ILM computers compared the de-aged sequences produced in Flux against the actors’ younger onscreen selves to ensure continuity. “We took a look at Casino, GoodFellas, Home Alone, and Godfather II, Godfather III, hundreds of titles,†says Helman. “After we rendered their performances as younger actors, we would run these images through our database, through an artificial-intelligence program that would look for alike angles and lighting conditions. And the computer program would spew out frames that were similar to the ones we rendered.â€
“[The frames] didn’t end up in the movie,†he explains. “But it was a sanity check for us. Because there are some times an actor doesn’t look like himself or herself … It depends on the lighting, the location, the camera angle. So we will use a program to look for what we call ‘behavioral likeness.’ The actors look the way and behave the way that we recognize them as such.â€
Step 5: A Long, But Not Too Long, Postproduction
Helman’s team eventually produced a whopping 1,750 visual-effects shots, which compose almost two and a half hours of The Irishman’s 210-minute run time — the equivalent of a finished movie in the footage’s own right. Some 500 VFX technicians in postproduction facilities in Vancouver, Ireland, China, and India, as well as in Industrial Light & Magic’s San Francisco headquarters, labored for eight months to finish the footage, giving each sequence in which the actors appear de-aged a thorough once-over to remove as much uncanny-valley artificiality as possible.
But contrary to early press reports that vagaries of the digital de-aging process resulted in release delays for The Irishman, Helman would like you to know the film’s lengthy postproduction time frame — which is typical of blockbuster-event films like Avengers: Endgame or Avatar but had never been applied so unsparingly to a talky period drama — was always baked into the schedule. “We were always going to deliver when we delivered,†he says. “It’s not that we were slowing down anything.â€
The two-time Academy Award nominee would also have you know Scorsese’s specific vision dictated that de-aged versions of the characters remain more faithful to the plot of the film than the actors’ actual younger physiques. “Marty’s design of the characters was tailor made for the movie,†Helman says. “Meaning that he didn’t want to rewind 30 years and find Jimmy Conway from GoodFellas. If he’s going to show Frank Sheeran in World War II, he doesn’t want to see Taxi Driver. Same thing with Pesci. Pesci at 53 years old was never that thin as an actor. He had a little more weight on him, but in the movie he’s really thin. That was by design.â€
Step 6: Look to the Future
Partway through The Irishman’s opening act, De Niro’s Frank Sheeran character is shown conversing in passable Italian during a high-stakes, get-to-know-you meeting with Pennsylvania mobster capo Russell Bufalino (Joe Pesci) — the dawn of a professional camaraderie that will span decades and result in the pileup of untold corpses. Although the sequence was filmed when both actors were squarely in the winter of their lives, each man appears convincingly middle-aged: their frown lines, crows’ feet, and chin wattles buffed into submission through Flux technological innovations. The scene suddenly flashes back to World War II, when we see an even-more radically de-aged 20-something De Niro unceremoniously assassinate a pair of Nazi soldiers whom he has just ordered to dig their own graves.
Even at a cultural moment in which it has become standard operating procedure to smooth away movie stars’ laugh lines and under-eye wrinkles through discreet digital legerdemain — and when the announcement of a movie featuring the CGI re-creation of James Dean has been met with an outpouring of fan opprobrium (and some legitimate hesitation, too) — Helman feels his Flux technology has the potential to affect lasting change in Hollywood. In addition to eliminating the hours in makeup an actor would otherwise endure for the time-consuming application of mo-cap dots, ILM’s technological breakthroughs ultimately amount to a kind of hidden carpentry. They are, by design, tools to enable a focus on actors’ performances, rather than cloaking them beneath flashy FX flourishes.
“Like any achievement, it’ll be measured against what it will do for the industry in the future,†Helman says. “I think actors are going to take a look at this and say, ‘Does that mean that I don’t have to wear 138 markers on my face? That I don’t have to spend two hours in makeup? That I don’t have to sit through a bunch of calibration cameras? And I can act on set with the lights, in front of the acting partner I need to perform against?’ Those are the kind of things that are important. And that’s where we’re going.â€
More From This Series
- Cheese: An Annotated History of the Oscar Class Photo
- The 2024 Oscars Shortlists Say Hi, Barbie
- The 2024 Golden Globe Nominations Are Here