Handling Long Sequences in After Effects and Photoshop
by Ben Bardens
In the last issue of this publication, I explored some strategies for rotoscoping directly in After Effects using the Pen tool. In this column, I will introduce a way of using Photoshop, along with After Effects, to handle longer sequences. Since Photoshop has several tools for making selections––and an Actions panel to automate repetitive steps––it is ideal for handling lots of frames. In this tutorial I’ll use the paintbrush to create layer masks to rotoscope my footage. This is a good technique because it is non-destructive, and background areas can be painted back in if the edges need adjustment. I want to stress, however, that this is not the only way to rotoscope frames in Photoshop, and for some situations it might not even be the best. It is just one of many possible approaches to handling the frames once they are in Photoshop.
Figure 1: Check the technical specs for your source clip and make sure After Effects is interpreting it correctly. It’s especially important to remove interlacing before rotoscoping.
Download this archive containing a Photoshop actions (.atn) file and some additional notes: http://seco.glendale. edu/~bbardens/roto.zip. Feel free to use or modify these actions for your own use beyond this tutorial. If you don’t have a clip handy to rotoscope, go to www.stockfootageforfree. com, sign up and download a free clip. In the example, I am using the skateboarder clip from their Sports category, but you can use any clip that has a subject you wish to rotoscope.
1. Import your clip into AE. Check and make sure the footage is being interpreted properly by looking at the top of the Project panel. If the clip isn’t being interpreted correctly, choose File->Interpret Footage->Main and modify settings as necessary (see Figure 1).
Note: It is especially important to correctly de-interlace footage before rotoscoping. See the notes included in the archive for information on this.
2. Add the clip to a composition by dragging the footage to the timeline.
3. Set the work area for the range of frames you wish to rotoscope. If you wish to adjust your comp duration to match your work area, choose Composition->Trim Comp to Work Area (see Figure 2).
4. Choose Composition->Make Movie.
5. In the Render Queue panel, click on the pop-up next to Output Module and choose Photoshop. Then click on the word Photoshop to open the settings and change Post Render Action to Import. Click OK (see Figure 3).
Figure 2: Adjust the work area on your timeline before outputting the frames you wish to rotoscope.
6. Back in the Render Queue panel, click on the file name next to Output to and create a New Folder to save your frames in. Make sure and leave the .PSD suffix in the file name (see Figure 3).
Note: The # signs in the file name will control how many leading digits are in the sequence numbering. (For example, for the first frame, #### will result in 0001.)
7. Click Render.
Once the render is complete, the frame sequence will appear in your Project panel. You’ll also have a folder with all the frames located wherever you specified on your hard drive. By setting the post-render action to import, AE interprets the footage with the same frame rate as the original clip. This is especially important if you plan on compositing the rotoscoped frames against the original.
8. Drag the frame sequence to the timeline and place it above the original clip.
9. Hit the Home key on your keyboard to make sure you’re at Time = 0, and then choose Edit->Edit Original.
If file handling is configured correctly on your computer, Photoshop should open up automatically and display the frame. If for some reason the file opens in another program, close it, switch back to the desktop and locate the first frame. Right-click on the frame and choose Get Info (Mac) or Show Properties (PC). Change the Open with option to Photoshop and click Change All.
10. If presented with any warnings about Pixel Aspect Ratio Correction in Photoshop, click Yes and check the box for Don’t Show Again so that it doesn’t ask for every single frame.
Figure 3: Choose Composition->Make Movie and then modify the settings as pictured.
11. You could now simply cut out the subject and then save the frame. However, since we have a large number of frames to process, we’ll set up a workflow to limit the number of steps we’ll have to take for each remaining frame in the sequence. Start by creating a custom workspace for the task. Open up the Actions, Layers and Channels panels to keep them on your screen and close all the rest. If one of the panels isn’t showing, choose it from the Window menu. To save the workspace for later use, choose Window -> Workspace -> Save Workspace and name it “Roto with masks” or something like that. To reset your workspace back to the defaults later, you can choose Window->Workspace->Essentials (Default).
12. Load the Actions file you downloaded earlier (roto actions.atn) by choosing Load Actions from the pop-up menu in the top right of the Actions panel (see Figure 4).
13. Click the little arrow next to roto actions to expand the folder. You’ll see four actions: step 1, step 3, step 5 and save & close. Steps 2 and 4 in this technique are the user-defined steps and are not recorded as part of the actions.
14. You can run an action by selecting it and then clicking the play button at the bottom of the panel. To cut down on the amount of clicking, switch the Actions panel to Button Mode from the pop-up menu in the top right corner of the panel. Scroll down so you can see all four actions. Then click on step 1 to run that action.
This action un-flattens the background and adds a layer mask.
15. Choose the Brush tool by pushing the B key on the keyboard. Click on the Brush drop-down in the Tool Options Bar at the top of the screen, and choose a brush size. In this example, I’m using a 9-pixel brush.
Figure 4: The pop-up menu at the top of the Actions panel.
16. Set the brush Hardness to 100% (see Figure 5). It’s easier to soften a hard mask back in AE than it is to sharpen a soft one.
17. Make sure your Foreground and Background colors are set to black-and-white (Push the D key on the keyboard to reset them, then push the X key on the keyboard to flip the colors). In Adobe programs, when painting on masks/alpha channels, black is transparent and white is opaque.
18. Paint an outline around your foreground subject. Make sure to include all parts of the subject. If anything intersects or overlaps the subject, do not include it. Also, make sure to outline around inner areas, such as in between arms and legs (see Figure 6).
Note: It’s better to err on the side of making the outline a little loose around the foreground subject rather than too tight. You can use the Simple Choker (Effect -> Matte -> Simple Choker) in AE to expand/contract your edge.
19. Click on the button for step 3 back in the Actions panel.
This action turns off the composite RGB info in the channels, and selects the appropriate alpha channel so you can edit the black-and-white matte.
20. Switch to the wand tool by pushing the W key on the keyboard. The Quick Selection tool is the active tool in that group at first, so type Shift W to make the magic wand active instead (you’ll only need to do this once).
21. Back in the image, click anywhere in the white area outside of your outline to identify what area of the frame is the background. If there is more than one area that is part of the background (like the area between his legs in the example), hold the shift key to select multiple areas.
22. Click the button for step 5 in the actions panel.
This action expands the selection, fills it with black to create transparency and switches back to the composite RGB mode in the channels.
The background should now be transparent. Verify this. If necessary, you can switch back to the Brush tool to add or remove areas by painting with black or white.
Figure 5: The Brush drop-down menu in the Tool Options bar.
23. Click the action save & close.
24. Switch back to After Effects. In the Project panel, right-click on the frame sequence and choose Reload Footage (you only have to do this once for the first frame).
25. In the timeline panel, turn off the visibility for the original background layer so you can see the cut-out foreground subject.
26. Push the Page Down key on your keyboard to advance to the next frame and type Command E (Mac) or Control E (PC) to load the next frame into Photoshop. (Note that the frame sequence layer must still be selected in the timeline for the Edit Original function to work.)
At this point, the pipeline between your After Effects project and Photoshop is set. You can load each frame directly into Photoshop, use the Actions to do the roto, then switch back to AE and it will automatically update in the timeline.
The Basic Outline of the Steps for Each Frame:
- Frame opens in PS, hit F on keyboard to go full screen.
- Run action step 1.
- Hit B for Brush and paint outline. Use [ and ] keys to adjust brush size while painting. Use Command + or – to zoom in and out on frame as needed. Push and hold the Spacebar to toggle the Hand tool to scoot the image around while painting.
- Run action step 3.
- Hit W for Wand then click in BG areas (shift-click for multiple areas).
- Run action step 5.
- Run action save & close.
- Switch back to After Effects.
- Page Down to next frame and type Command E to open frame in Photoshop, and repeat.
Once you get into a flow, you can power through each frame in as little as two-to-three minutes.
Figure 6: Paint an outline completely around the edge of the foreground subject, then run action step 3.
Tip: It’s a good idea to decide on your effects treatment before you process all the frames. Some treatments, like a soft vignette, don’t require a very precise outline. Other treatments, like outer glows, require a much tighter edge. If you’re not sure what quality you need, roto just one frame, load it back into AE and then decide on your effects treatment before doing the rest of the frames.
This just scratches the surface of the many ways Photoshop and After Effects can be used together for rotoscoping footage. Further suggestions: For higher-resolution footage, use the Pen tool instead of the Brush. In a production environment where there are multiple artists available, put the frames on a high-speed server that everyone can access. A single compositor can have them loaded in their project while other artists are doing the roto on separate workstations. By periodically choosing File->Reload Footage, the compositor can update the project as the new rotoscoped frames become available. Set Function keys for actions instead using Button mode (choose Action Options from the Action panel pop-out menu). Lastly, if painting, use a drawing tablet instead of a mouse.
Ben Bardens teaches After Effects at Glendale Community College, Video Symphony and Studio Arts. He can be reached at firstname.lastname@example.org.