Three.js / jsCAD: how to smooth a mesh normals for an outline effect?

Question

I'm trying to create an outline effect on some 3D meshes using the following method :

• Draw the original initial shaded mesh
• smooth the mesh normals
• push the vertives outward following their normals
• draw the back faces only with a solid outline color

I got it all working except the smoothing part. Smoothing the mesh is required to create a nice continuous outside outline (else there would be irregularities at the angles). I've allready asked how to smooth a mesh normals. The smoothing process goes like so :

  let geometry = new THREE.BoxGeometry();
  geometry.deleteAttribute('normal');
  geometry.deleteAttribute('uv');
  geometry = THREE.BufferGeometryUtils.mergeVertices(geometry);
  geometry.computeVertexNormals();

Although this works well for simple well defined geometries, it creates artifact on more complexe models. I believe this is due to discarding the original normals. computeVertexNormals() then interprets some faces inside-out.

Any idea how to fix that ?

before smoothing normals

after smoothing normals

---

sidenotes: 1/ I can't modify the original meshes topology because they are generated at runtime using jscad.

2/ I tried to write an algo to fix that but without great success. My reasoning went as follow:

Let's assume computeVertexNormals() interprets a triangle being face-up or face-down based on the order of the vertices composing it. I'm going to find the triangles with vertices which normals get inverted and exchange two of the triangle vertices to flip it. A vertex normal is inverted if the dot product of the normal after computeVertexNormals() and before computeVertexNormals() is negative.

Here is the code for that:

const fixNormals: (bufferGeometry: THREE.BufferGeometry) => THREE.BufferGeometry
  = (bufferGeometry) => {
    // this function is only made for non indexed buffer geometry
    if (bufferGeometry.index)
      return bufferGeometry;
    const positionAttribute = bufferGeometry.getAttribute('position');
    if (positionAttribute == undefined)
      return bufferGeometry;
    let oldNormalAttribute = bufferGeometry.getAttribute('normal').clone();
    if (oldNormalAttribute === undefined)
      return bufferGeometry;

    bufferGeometry.deleteAttribute('normal');
    bufferGeometry.deleteAttribute('uv');
    bufferGeometry.computeVertexNormals();

    let normalAttribute = bufferGeometry.getAttribute('normal');
    if (normalAttribute === undefined) {
      console.error("bufferGeometry.computeVertexNormals() resulted in empty normals")
      return bufferGeometry;
    }

    const pA = new THREE.Vector3(),
      pB = new THREE.Vector3(),
      pC = new THREE.Vector3();
    const onA = new THREE.Vector3(),
      onB = new THREE.Vector3(),
      onC = new THREE.Vector3();
    const nA = new THREE.Vector3(),
      nB = new THREE.Vector3(),
      nC = new THREE.Vector3();

    for (let i = 0, il = positionAttribute.count; i < il; i += 3) {
      pA.fromBufferAttribute(positionAttribute, i + 0);
      pB.fromBufferAttribute(positionAttribute, i + 1);
      pC.fromBufferAttribute(positionAttribute, i + 2);

      onA.fromBufferAttribute(oldNormalAttribute, i + 0);
      onB.fromBufferAttribute(oldNormalAttribute, i + 1);
      onC.fromBufferAttribute(oldNormalAttribute, i + 2);

      nA.fromBufferAttribute(normalAttribute, i + 0);
      nB.fromBufferAttribute(normalAttribute, i + 1);
      nC.fromBufferAttribute(normalAttribute, i + 2);

      // new normals for this triangle are  inverted, 
      // need to switch 2 vertices order to keep right face up
      if (onA.dot(nA) < 0 && onB.dot(nB) < 0 && onC.dot(nC) < 0) {
        positionAttribute.setXYZ(i + 0, pB.x, pB.y, pB.z)
        positionAttribute.setXYZ(i + 1, pA.x, pA.y, pA.z)
      }
    }

    bufferGeometry.deleteAttribute('normal');
    bufferGeometry.deleteAttribute('uv');
    bufferGeometry.computeVertexNormals();

    return bufferGeometry;
  }

Answer 1

THREE performs computeVertexNormals by using the winding order of the face to determine the normal. If a face is defined using left-hand winding, then you'll get normals pointing in wrong direction. This should also affect lighting unless you aren't using a shaded material.

That said, you could also see scenarios where the computed normals are different between faces. To avoid this, you'll need to track all of the normals for a given vertex position (not just a single vertex, since the position can be duplicated elsewhere in the buffer).

Step 1: Create a vertex map

The idea here is that you want to create a map of actual vertex positions to some attributes you define that will be used later. Having indices and groups can affect this processing, but in the end you want to created a Map and assign its key as a serialized version of the vertex values. For example, vertex values x = 1.23, y = 1.45, z = 1.67 could become the key 1.23|1.45|1.67. This is important because you need to be able to reference the vertex position, and keying off two new Vector3s would actually map to two different keys. Anyway...

The data you need to save is the normal for each vertex, and the final smoothed normal. So the value you'll push to each key is an object:

let vertMap = new Map()

// while looping through your vertices...
if( vertMap.has( serializedVertex ) ){
  vertMap.get( serializedVert ).array.push( vector3Normal );
}
else{
  vertMap.set( serializedVertex, {
    normals: [ vector3Normal ],
    final: new Vector3()
  } );
}

Step 2: Smooth the normals

Now that you have all of the normals for each vertex position, iterate through vertMap, averaging the normals array and assigning the final value to the final property for each entry.

vertMap.forEach( ( entry, key, map ) => {

  let unifiedNormal = entry.normals.reduce( ( acc, normal ) => {
    acc.add( normal );
    return acc;
  }, new Vector3() );

  unifiedNormal.x /= entry.normals.length;
  unifiedNormal.y /= entry.normals.length;
  unifiedNormal.z /= entry.normals.length;
  unifiedNormal.normalize();

  entry.final.copy( unifiedNormal );

} );

Step 3: Reassign the normals

Now loop back through your vertices (the position attribute). Because vertices and normals are paired, you can use the index of the current vertex to access the index of its associated normal. Reference the current vertex by again serializing it and using that value to reference vertMap. Assign the values of the final property to the associated normal.

let pos = geometry.attributes.position.array;
let nor = geometry.attributes.normal.array;

for( let i = 0, l = pos.length; i < l; i += 3 ){
  //serializedVertex = however you chose to do it during vertex mapping
  let newNor = vertMap.get( serializedVertex ).final;
  nor[ i ] = newNor.x;
  nor[ i + 1 ] = newNor.y;
  nor[ i + 2 ] = newNor.z;
}
geometry.attributes.normal.needsUpdate = true;

Caveat

There are probably more elegant or API-driven ways to perform the actions I described above. The code I provided was not meant to be efficient nor complete, but simply give you an idea of how you can approach the problem.