Friday, November 16, 2012

Skeletal Animation and GPU Skinning - The Crux

Recently, for one of my projects, I had to learn about skeletal animation and GPU skinning. As always, I first went to google which gave me a wealth of information. As always, I try to get the real crux of things rather than reading a whole document only to find out in the end that the details are not there. For my case, I could not find a simple explanation of how the skeletal animation works including the whole series of transforms involved. This blog entry will try to answer this.

The transformation matrices
I will start with the different transforms that are used in this process. Typically in a simulation or game, a transform is represented as a 4x4 matrix. For skeletal animation, we have a collection of bones. Each bone has a local transform (also called relative transform) which tells how the bone is positioned and oriented with respect to its parent bone. If the bone's local transform is multiplied to its parent's global transform, we get the global transform (also called absolute transform) of the bone. Typically, the animation formats store the local transforms of the bones in the file. The user application uses this information to generate the global transforms. I define my bone structure as follows,

struct Bone {
glm::mat4 relativeXForm, absoluteXForm;
int parent; //index of parent
};

As you can see, I only include the fields that I need. You can populate whatever you want in this structure. Final word about the parent variable. This stores the index of the parent bone of the current bone. For the root bone, the parent is -1. For all of the other bones, it will be a number starting from 0 to N-1 where N is the total number of bones in the skeleton.

OK lets say I have a collection of bones stored in a vector called skeleton.  We can get the absolute transforms for all of the bones in the skeleton using the following code.

for(size_t i=0;i < skeleton.size(); i++) {
Bone& b = skeleton[i];
if(b.parent==-1)
b.absoluteXForm = b.relativeXForm;
else
b.absoluteXForm = skeleton[b.parent].absoluteXForm * b.relativeXForm;

}

As you can see, it is very simple ans straight forward. Now there is another big word that I see scattered all over the place in the skeletal animation tutorials and codes available online. This word is bindpose. Simply put bind pose is the absolute transforms of the bones in the non animated state (i.e. when no animation is applied. This is usually when the skeleton is attached (skinned) to a geometry. We can also say, it is the default pose of the skeletal animated mesh. Typically, bones can be in any bindpose (usually for humanoid characters, the character may be in A pose, T pose etc. based on the convention used). Typically, the inverse bindpose is stored at the time of initialization. So continuing to the previous skeleton example, we can get the bindpose and inverse bindpose matrices as follows,

for(size_t i=0;i < skeleton.size(); i++) {
bindPose[i] = (skeleton[i].absoluteXForm);
invBindPose[i] = glm::inverse(bindPose[i]);

}

When we apply any new transformation (an animation sequence for example) to the skeleton, we have to first undo the bind pose transformation so that the bones can be first moved  to their origin (so that the bones transform is identity and it transform is its parent's transform). This is similar to how we apply composite transformation in OpenGL and DirectX i.e. we first move the object to world origin by untranslating it and then do the transformation at origin. At the time of calculation of the bindpose matrices, we store the inverse of the bindpose matrix. Any new animated transformation on the bone has to be multiplied by the inverse of the bindpose matrix.The final matrix that we get from this process is called the skinning matrix (also called the final bone matrix). Continuing to the example given in the previous paragraph, lets say we have modified the bone's relative transforms using the animation sequence. We can then generate the skinning matrix as follows,

for(size_t i=0;i < skeleton.size(); i++) {
Bone& b = skeleton[i];
if(b.parent==-1)
b.absoluteXForm = b.relativeXForm;
else
b.absoluteXForm = skeleton[b.parent].absoluteXForm * b.relativeXForm;

skinnedXForm[i] = b.absoluteXForm*invBindPose[i];
}

So as you can see this is simple to do.

Skinning on the GPU
I think there are plenty of places where GPU skinning has been explained elsewhere so I wont touch up on it. I will detail my skinning vertex shader which is as follows,

attribute vec4 blendIndices;
attribute vec4 blendWeights;
varying vec3 out_normal;
varying vec3 es_vpos;

uniform mat4 Bones[62];

void main() {

vec4 newVertex;
vec3 newNormal;

int index =  int(blendIndices.x);

newVertex = (Bones[index] * gl_Vertex) *  blendWeights.x;
newNormal = (Bones[index] * vec4(gl_Normal, 0.0)) *  blendWeights.x;

index= int(blendIndices.y);
newVertex = (Bones[index] * gl_Vertex) * blendWeights.y + newVertex;
newNormal = (Bones[index] * vec4(gl_Normal, 0.0)) * blendWeights.y  + newNormal;

index= int(blendIndices.z);
newVertex = (Bones[index] * gl_Vertex) *  blendWeights.z  + newVertex;
newNormal = (Bones[index] * vec4(gl_Normal, 0.0)) *  blendWeights.z  + newNormal;

index= int(blendIndices.w);
newVertex = (Bones[index] * gl_Vertex) *  blendWeights.w   + newVertex;
newNormal = (Bones[index] * vec4(gl_Normal, 0.0)) *  blendWeights.w  + newNormal;

out_normal  = normalize(gl_NormalMatrix * newNormal);
es_vpos = (gl_ModelViewMatrix * vec4(newVertex.xyz,1)).xyz;
gl_Position = gl_ProjectionMatrix * vec4(es_vpos.xyz,1);

gl_TexCoord[0] = gl_MultiTexCoord0;
}

As you can see, I use a previous gen shader version since my focus is on detailing the crux. Basically, the skeletal animated mesh will store the skinning information (blend weights and blend indices). You need to pass these to the relevant attributes and finally, you need to pass the skinning matrices. I pass my skinning matrices in one shot as follows,

There are two things that you have to be careful about.
1) Make sure that the size of the bones array is correct. Often times, you will find that part of the mesh is skinned fine while the other parts are not skinned correctly. If so make sure the size of the bones array is correct.
2) Make sure that the proper vertex attrib array is enabled and the correct parameter types are passed. For my skinning vertex shader, I use the following code to pass this data to the shader.

glVertexAttribPointer(shader["blendIndices"], 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), &vertices[0].blendIndices);

If you have done all of these steps fine, you should have a nice skinned skeletal mesh on screen. I hope that I have clarified the skeletal animation steps so that it is easier to grasp and understand. I don't know why this information is not detailed in nice simple manner anywhere.

Based on a couple of requests, here is the source code for an EZMesh animation viewer.
https://www.dropbox.com/s/u11f3ldf2nxeayi/EzMeshAnimationViewerSkinned.zip?dl=0

This is what the output looks like

You can get details about EZMesh format from here:
http://codesuppository.blogspot.sg/2009/11/test-application-for-meshimport-library.html

and the EZMesh project is hosted here:  http://code.google.com/p/meshimport/

Controls for the program:
Press ',' to go to previous animation frame
Press '.' to go to next animation frame
Press 'l' to enable looping otherwise the animation stops at the last/first frame.

Enjoy !!!

Thanks,
Mobeen

Siphonife said...

You are a god. I hope this will help me understand skinning. It really does seem like a little amount of code compared to all other sources out there. The harder part is supporting mesh and animation formats... Right now I'm stuck lerping between sampled animations.

Hi Siphonife,
I beg to disagree (I am a human being like all others). I had a hard time finding the crux of skinning when I came to it, I thought of posting it online for others.

Chris F said...

Thank you so much ! The tip about multiplying by the inverse of the bindpose matrix is crucially important, but none of the other tutorials that I've looked at bother to mention this. You saved me hours of frustration and dead ends - thanks again !!

You are always welcome Chris I am glad it has helped you. When I was working on this, I could not find any reference on what are the bare minimum steps to implement character skinning. After looking out to all available resources, several demos and tutorials, I realized that it was simpler than what others were saying that's when I decided to write this blog post.

Firangi4u said...

I would like some clarification to match these matrices to Assimp's matrices.

Assimp provides the following data:
-a hierarchy of aiNodes which contain a transform
-an array of bones, that have to be matched with aiNodes and contain the inverse bind pose but they call it offsetmatrix (also the weights & indices to be attached to vertices)
-an array of animations, each with an array of channels (joints really) with SRTs.

From my understanding, the array of channels's SRTs (usually interpolated but that's for later) would be used as "localtransforms" in your above code.

The offsetmatrices would be stored during init and used as inv bind poses. I don't need to calculate it as shown above since I already obtain it for free.

"globaltransforms" is calculated myself by traversing the hierarchy with above code.

Finaltransforms is globaltransforms multiplied with inv bind pose and voilĂ , I can send this to my shader.

Is my understanding correct or did I miss something?
It seems I don't need the transforms that were in ainodes.

If it works the animation result should be same as in the modelling package.

Fredrik Haikarainen said...

This is GOLD! Thank you! Finally some clean details on the subject. I will probably write an article on the subject later, can I reference this article if I do?

Thanks Fredrik. Glad you liked it. Sure share the link so that I may forward others to your article.

Stewart Connor said...

Thank you so much, this helped a lot. If I understand it correctly, for every frame of animation you go through the bones starting at the root, compute their absolute matrix for that frame using their parents absolute matrix for that frame and whatever rotation you want, then get the skin matrix by using that absolute matrix in the inverse bind matrix for the bone? That's what I'm doing now and it seems to work, I just want to know if I'm not doing more than I need to (eg computing too much per frame). Thank you again for the article.