A Vertex PaintinG Primer

by Kyle Youngblom

A while back I was super fed up with the complexity of creating 3D assets for games. Normal maps, occlusion baking, retopologizing, UV layout, lighting; I wanted to toss it all out the window.

Then I saw what artists like Chelsea Saunders were doing with solid blocks of color on low-poly models, and I saw how ugly (in my opinion) the WiiU Wind Waker remake was, and I started getting into flat-shaded vertex painting. It's changed pretty much everything about how I work in 3D; letting me make art spontaneously and creatively without being weighed down by a complicated pipeline. I rarely even need to open Photoshop. Also it helped me get a sweet job, check out Guildlings.com.

This document is my attempt at a quick intro to help people get started. It's written for Autodesk Maya, though I'm sure the basic concepts would apply to other 3D software as well, and assumes a basic familiarity with the program. Let me know if you have comments and/or suggestions, and thanks for reading.

What even are vertex colors anyway?

A 3d mesh is comprised of a bunch of points (vertices) in space, with flat surfaces drawn between them. Each vertex can store an array of numbers that define its position in space, normal vector, and texture coordinates. A vertex can also store numbers to represent color information (Red, Green, Blue, Alpha). These numbers can be used to color a model instead of applying textures. By using Maya's Paint Vertex Colors tool we can add color directly to our models instead of applying textures.

This polygon is constructed of three vertices. Vertex A has an RGB value of 255, 0, 0 (red), B has a value of 0, 255, 0, and C a value of 0, 0, 255. The colors blend smoothly into each other.
​If we want hard lines between our colors, we can fill an entire polygon with one color and an adjacent polygon with another color.
​These two techniques can also be combined to allow hard lines and smooth gradients. Here we're blending between blue and red, with a hard border around the green area.

Vertex colors are typically put to use storing additional information to be used in a game engine for blending materials together. A game artist might paint in a value where they want snow to build up, or bake in ambient occlusion to multiply on top of your diffuse material.

​Why use vertex colors instead of textures?

Efficiency

​Textures take up a lot of memory and storage. Keeping texture use to a minimum drastically reduces the download size of a game, (especially useful on mobile) and allows a game to have more tame hardware requirements. Unlit shaders work really well with vertex colors, and also let you get away with a lot of visual tricks and cheats.

Fidelity

Textures are limited by their resolution and compression settings, but geometry scales and looks good at any screen resolution. It's future-proof. Boundaries between colors can be super sharp in a way that'd normally require very high-resolution textures.

Style

​More isn't always better. Working in vertex colors forces you to accept limitations and improvise. Flat-shaded 3D often has a timeless graphic or illustrative look, and I think the result can be really vibrant and fun.

Speed

It’s much faster to not have to worry about UV layouts, texture files, and switching between programs to texture and model. A single artist can create way more content in a low-poly flat-shaded style than if they have to worry about spending weeks in Zbrush modeling out the pores on a dog's nostril.

What are some drawbacks to vertex colors?

​Complex textures or patterns can be difficult or impossible to pull off (unless you become a boss at modeling, which you should).

Poly count suffers; something that could have been accomplished with a texture now has to be created by slicing up a mesh.

It’s not a common workflow. There isn't as much information on it out there, and software isn’t necessarily designed for it so requires some customization.

​OK I'm sold! Tell me how to vertex paint please

​Start by creating a cube.

Select your cube and activate the Paint Vertex Color Tool (Modeling: Mesh Display>Paint Vertex Color Tool). You’ll probably want to set this to a hotkey (I use shift+v).

You should now see a paintbrush cursor with a red circle representing your brush size. Hold 'b' and left-click in the viewport to change the brush size.

Make sure your Tool Settings are open (Windows>Settings/Preferences>Tool Settings, alternatively: double-click the Paint Vertex Colors icon in the Tool Box).

In tool settings, make sure the paint mode is set to “vertex”. Double-click the color swatch to the right of “Color value” and select a bright color.

Click on a vertex of your cube and it’ll turn the color you chose. Try again with different colors, paying attention to how Maya interpolates between vertices.

In tool settings, set the paint mode to “face”. This restricts the color you’re applying to the inside of a face, creating hard lines along edges. Test this out a bit.

Once you get the hang of that, try “vertex face” mode. This restricts color to the inside of a face, but only applies to a single vertex rather than filling the face.

Taking it Further

Ok, great, so we can color a cube. It's a start! The tricky part of vertex painting is using the topology of a model to help define the vertex colors. This means you've got to understand how colors blend and how the geometry of a mesh can support what you want to create.

Let's say we want to create a square of blue that fades into an outer border of green. We might start with this 3x3 plane, thinking we can paint the inner four vertices blue and the outer ring green.

But whoops, there's some weird business going on with the left and right faces. They're not blending how we might expect.

This is because secretly all these 4-sided polygons are actually two 3-sided polygons. Maya's automatically splitting them behind the scenes. Use "toggle hidden triangles" to view them.

If we use Maya's Multi-Cut tool to split the faces in the direction we want them to blend, we get a proper gradient in the direction we want.

Ok, this is the worst part of any tutorial: you gotta practice. Start small, pick a simple self-contained prop or some familiar object and try to model it out. See how much character you can cram in there with as few vertices as possible. See how much you can push simple geometry to create appealing, complex shapes.

Become amazing at using Maya's modeling toolkit. A big part of vertex painting is learning when and where to make small cuts in your geometry. This workflow requires a lot of experimentation and knowledge of Maya's modeling tools, but it's also a great way to learn them. Have fun!

And please, if you have any questions or comments feel free to contact me on Twitter at @KyleYoungblom, or through my site at KyleYoungblom.com.

Shortcuts, Tips, and Scripts

Once you get started with this stuff, you'll probably realize that Maya's not really set up for vertex painting in this way. You're going to want to do a bit of customizing to make things work for you.

Paint Vertex Tool
If you're vertex painting you'll be bringing up the tool a lot. I have mine set to shift+v. The command to bring it up:
artAttrColorPerVertexToolScript 4
Color Picker
Picks a vertex color from somewhere on the currently selected model. Note: if you're in 'vertex' mode in the paint tool it will pick a color value based on an average of the nearest vertices. Assign this script to a hotkey:
artAttrPaintVertexCtx -e -pickValue `currentCtx`
Wireframe on Shaded
One of the best things about unlit flat-shaded 3D is that it looks... well, flat. This can make it hard to see the topology of your model though, so displaying a wireframe on even deselected models is often helpful. Assign this script to a hotkey:
global proc toggleWireframeOnShaded() {    string $currentPanel = `getPanel -underPointer`;    if ("" == $currentPanel)        {        $currentPanel = `getPanel -withFocus`;        }    if ("" != $currentPanel)       {        string $panelType = `getPanel -typeOf $currentPanel`;        if ($panelType ==  "modelPanel")            {            int $state = `modelEditor -q -wireframeOnShaded $currentPanel`;            modelEditor -e -wireframeOnShaded (!$state) $currentPanel;            }       } }
Use All Lights
If you're working in a flat-shaded style, you won't want to be using Maya's default shading mode ("5"). Hit "7" instead to enter "use all lights" mode. As long as you haven't added any lights to your scene this'll display your vertex colors without blowing them out.
Display Face Centers
Puts a dot in the middle of each face of the selected model. This is handy because when vertex painting in 'face' mode, a face is only filled when you paint over the center point. You can also hold 'V' when using the move tool to snap to face centers. Face centers stay visible even when deselected while in wireframe on shaded mode.
setPolygonDisplaySettings("faceCenter")
Toggle Reflection
When vertex painting, hit alt+r. This'll mirror your stroke across the center axis when vertex painting. Kind of dodgy but sometimes useful.
Toggle Wireframe
When vertex painting, hit alt+a. This will toggle the wireframe on your selected model if you want to paint without it getting in the way.
Flood Surfaces
Flood model with currently selected color (while using the Paint Vertex tool) Command:
artFloodSurfaces