As part of our weeklong exploration of SFX on film, we’re looking back at the genesis of a character who helped change the course of how computer-generated effects are used on film: Gollum, the Tolkien creation played by Andy Serkis and brought to life by Weta Digital in the Lord of the Rings and Hobbit films. To better understand the marriage of technology and artistry that brought Gollum to life, and the effect that collaboration had on future CG storytelling, Vulture spoke to Weta visual effects supervisor Eric Saindon about the process of creating and evolving the character over the course of the three Lord of the Rings films.
Eric Saindon: When we did Gollum, no one had really done a CG creature mixed with live action that you were supposed to believe was just a creature in the scene. Up to that point, you’d seen a lot of CG creatures, but none to the level of Gollum, where the detail [made] him an actor on screen, rather than just a scary monster, which is what a lot of the CG creatures up to that point had been. And because we were inventing technology as we were doing it, it was a huge challenge. Like, we were writing muscle systems. Because he was so skinny, you saw all of the muscles moving under his skin.
I remember reading that the reflection of light on and under Gollum’s skin and the way you could see his veins was one of the biggest innovations that you guys pioneered. That was something that had never been done before.
Yeah, it’s called Subsurface [​Scattering​] and we actually got an Oscar for that technology. It’s basically like, if you light a wax candle, you see the diffusion of the light through it. That’s sort of the idea of Subsurface. It spreads the light as it goes through the surface so you get that skin-like feel. And we still use Subsurface to this day. It’s different implementations of it, but it’s the same basic concept that was [developed] back on Gollum.
What was the process of creating that technology like?
Well, the Subsurface technology itself, [it] was actually Ken ​McGaugh​ and Joe Letteri who did the majority of the work for that, and they basically took a concept paper that had been written and implemented the math and the information in the paper into a shader-based system, so that we could put it onto the skin of Gollum. So it’s lots of different areas of technology within Weta that put all that together.
When we did Gollum, I was head of the creatures department, and we were writing our own volume-preserving muscle system that allowed us to basically take a skeleton, where we actually built the bones and everything for Gollum, and then put the muscles onto the skeleton and attached them at the proper points. So as Gollum bends and moves around, the muscles would flex and move like real muscles would, and allow Gollum to get skinned through this muscle system, which was a completely different way of doing things at the time.
How had people done it previously?
Everyone has something slightly different, but it all is based on skinning. A basic skinning is, you take points on the skin of the character, or the geometry of the character, and attach it to the bones underneath. So the bones underneath drive the skin, and they’re given different weights so they slide around based on how much that bone underneath is moving … So it’s driven by a point inside of the skin. The difference with this system is, the muscles underneath were actually moving and pushing the skin around. So instead of the skin getting pulled around by the skeleton underneath, they were actually getting pushed around by the muscles performing and moving underneath the skin.
How did you develop that?
We broke up the ideas, the different bits of code, into several different people, so one person was working on volume-preserving parts of the muscle, another was working on the attachment, another was working on the connections of the skeleton, all the different implementation parts. So as parts were finished, we were able to slowly put them all back together and build this.
I mean, the good thing was — the great thing is — Gollum was in film one [2001’s The Fellowship of the Ring​], like what, four times? Just little hints of him in the background.
Right, and often in dark settings.
Yeah, so you didn’t really need to have that much detail in him for that one. But we had to start it then. So we started working on all these systems in 1999, and we really didn’t need them working 100 percent until 2002, really, when we really had to get ​The Two Towers ​out and you really saw Gollum for the first time.
The real refinements came in movie three [2003’s The Return of the King], where all the extra skin detail, the wrinkling and everything, sort of just culminated at that point. And by that time, we were actually capturing Andy on set, too — so we did something else we had never done before, we did motion capture on set for Gollum for film three. The previous films we shot him on a mo-cap [motion capture] stage. So basically Andy had to watch playback [footage] and then interact like he was on set and try to recreate what was happening on set. So it was much less interactive for him and the performance just was not as driven by Andy, because it had to be edited and fit [into the footage]. By film three, it was Andy on set, so we were able to really get Andy as Gollum, rather than an interpretation of what was shot on the day.
It’s interesting to know that he was on set just for the third film. I’d be curious to go back and watch, because I’m sure it must have been really difficult to act and perform just watching playback and not being on set and interacting with the other actors and the environment.
I mean, Andy’s just such an interactive character. Like when he’s on set, he’s jumping off walls, he’s doing such crazy stuff. The mo-cap setup, even to this day — the cameras, the rig, everything like that, they’re built pretty tough, they can hold up to a lot of stuff. And on ​[2017’s War for the Planet of the Apes],​ ​we lost a couple, right? A couple setups, the actors did something that sort of broke it. Andy went through, like, 12 sets of mo-cap rigs on Return of the King ​alone​. He is so active, and so hyper, he’s like a kid bouncing off the walls.
I’m thinking about the scene in Shelob’s lair, where he’s jumping off the walls and running and jumping on top of Frodo and around him —
Yep. And if you watch the footage of Andy on set, when he’s with Elijah [Wood], that’s what he was doing [laughs].
I think that’s what has made performance capture such an interesting way of doing films now. Like on Alita [Battle Angel], now, we had Rosa Salazar, who plays Alita — she was on set every single shot, interacting with Hugo [Keann Johnson], interacting with the other characters — we never needed to shoot anything separate. That’s how far all of this, the performance capture, has come. We never worry about it, we just know the stuff we’re going to get on set is exactly what the performance was and when you watch it side-by-side with Alita and Rosa now, other than the characters obviously looking different, the performance is exactly the same.
It’s crazy for us just how far it’s come. I mean, on Lord of the Rings, the performance was definitely Andy, but it was a lot more hand-work then. We got the basic idea from Andy, but then character animators and facial animators had to go in and actually make sure that the lip corners were the same, the eyes were the same,  that all those basic shapes got to be what Andy had.
The Lord of the Rings maintained such a sense of realism and verisimilitude despite being a fantasy film, set in a fantasy period, because there was just so much attention to detail. All of the weapons were made by actual blacksmiths and there was such a historical basis there, but I think with the CGI also, that attention to detail with replicating patterns of light and muscle movements really helps the performance come through strongly and it really grounds the films.
And imperfections, too. I don’t mean mistakes in the work, but imperfections like — you were saying, blacksmiths made the swords; blacksmiths don’t make perfect swords, there’s going to be hammering that happens that makes imperfections in a sword. It’s the same in CG. Like Gollum has scars and cuts and weird little —
And bruises, right?
Yes! And eye twitches! Like, Andy had a little eye twitch. You’ve seen the film all those times, but, like, you probably didn’t even notice, the eye twitches Andy had, like little flutters of his eye —
I actually know exactly what you mean, the fluttering of his eyes and the dilation of his pupils, as well.
Yeah, yeah, and I mean all those little things, we took the time to put that stuff in and I think it’s what makes Lord of the Rings hold up. Like, you go back and watch the CG in it and I mean, eh, there’s some of the comps [3-D compositing, the process of editing and blending CG into footage] that could probably be a little bit better nowadays, but when you watch the films, the CG doesn’t take you out of the films, the visual effects still don’t take you out of the film, where if you go back and watch some old films, they don’t quite hold up. Jurassic Park, that one still holds up today, and that’s because they did the same thing — they had some of those imperfections, they spent the time on getting all those little details right, and that’s still a film you can go back and watch and think, “Wow, this still looks amazing.†And that’s what we were aiming for with Lord of the Rings, too.
You talked about the light and the muscles with Gollum, are there any other pieces of that puzzle that brought the character together?
I think another one is his eyes. It’s something we’ve always strived for here — the eyes need to bring the life into the character, and without that, you never get quite the level you need to get to. Gollum was one of the first characters [for whom] we really broke down how eyes are made, and the layers of the eyes and the crazy amount of detail you had to put into them.
On Gollum, the eyes we built were 5,000 polygons [units of 3-D composition], and that number was insane for the eyes. The amount of geometry that we had put in there was a quarter of the geometry for all of Gollum, so it was crazy that we were doing that. And I was just doing numbers and for Alita, the eyes are 9 million polygons. For each eye. So that’s how much the technology changes, right? Like Gollum, we were insane for putting 5,000 polygons in, and now we’re at 9 million. So the amount of detail you put into the eyes to get the depth to them, to see the different layers, the fibers and all those things, it’s something we spent a lot of time [with] on Gollum to get to look right, because every filmmaker has a shot close-up on the eye or going into the mouth. It seems like every movie I’ve ever done has had that happen [laughs]. So you have to spend the time on it and you have to get the meniscus, you have to get all these things we were creating on Gollum, that we’re still working on today, trying to improve on every movie, because it’s what brings those characters to life.