How to allow user's custom shader for an openGL software

Question

In softwares like Unity or Unreal, for example, how do they allow users to add their own custom shaders to an object? Is this custom shader just a normal fragment shader or is it another kind of shader? And if it is just a fragment shader, how do they deal with the lights?

I'm not gonna post the code here because it's big and would pollute the page, but I'm starting to learn through here: https://github.com/opentk/LearnOpenTK/blob/master/Chapter2/6-MultipleLights/Shaders/lighting.frag (it's a series of tutorials, this is the shader from the last one), and they say we should put the light types in functions, inside the fragment shader, to calculate the colors of each fragment.

For example, this function to calculate a directional light, extracted from the code I sent above:

vec3 CalcDirLight(DirLight light, vec3 normal, vec3 viewDir)
{
    vec3 lightDir = normalize(-light.direction);
    //diffuse shading
    float diff = max(dot(normal, lightDir), 0.0);
    //specular shading
    vec3 reflectDir = reflect(-lightDir, normal);
    float spec = pow(max(dot(viewDir, reflectDir), 0.0), material.shininess);
    //combine results
    vec3 ambient  = light.ambient  * vec3(texture(material.diffuse, TexCoords));
    vec3 diffuse  = light.diffuse  * diff * vec3(texture(material.diffuse, TexCoords));
    vec3 specular = light.specular * spec * vec3(texture(material.specular, TexCoords));
    return (ambient + diffuse + specular);
}

But I've never seen people adding lights in their shaders in Unity for example, people just add textures and mess with colors, unless they really want to specifically mess with the lights.

Is there a way of making just one fragment shader that will compute all the types of light, and the user could then apply another shader, just for the object material, on top of that?

If you don't know how to answer but have some good reading material, or place where I could learn more about openGL and GLSL it would be of great value as well.

Answer 1

There are a couple of different ways to structure shader files, each with different pros and cons.

• As individual programs. You make each file it's own shader program. Simple to add new programs, and would allow your users to just write a program in GLSL, HLSL, or an engines custom shader language. You will have to provide some way for the user to express what kind of data the program expects, unless you query it from the engine, but it might get complicated to make something that's generic enough.

• Über Shader! Put all desired functionality in one shader and let the behavior be controlled by control flow or preprocessor macros, such as #ifdef. So the user would just have to write the main function (which the application adds to the rest of the shader). This allows you to let the user use all the predefined variables and functions. The obvious downside is that it could be big and hard to handle and small changes might break many shaders.

• Micro Shaders. Each program contains a small, common functionality, and the application concatenate them all to a functioning shader. The user just write the main function and tells the program which functionality to add. The problem is that it's easy to get conflicts in names unless you're careful and is harder to implement than the über shader.

• Effect files. Provided by Microsoft’s effect framework or NVIDIA’s CgFX framework (deprecated).

• Abstract Shade Trees. Don't actually know what this is, but it's suppose to be a solution.

You can also combine some of these techniques or try to invent your own solution based on your needs. Here's the solutions discussed (in sector 2.3.3 Existing Solutions).