The problem you are trying to solve is called "Principal Component Analysis (PCA)". The process involves quite a lot of math, computing Eigenvalues, etc. I'm not entirely familiar with how it works either but just throwing this term at you, so you can use the name for further research
A memory leak won't cause access violations! A memory leak is when you allocate space on the heap, by using malloc or new and then simply abandon it without ever releasing it with free or delete. Smart pointers should do that sort of management for you and even if you were to do that wrong, it would simply cause the memory used by your program to rise steadily.
An access violation occurs, when you are trying to access memory in a way you're not supposed to. This means, it is either not in an allocated state (never allocated or allocated and then freed) or it has protections active. In most cases, this happens, when you freed some object but did not set all pointers to null. This sort of problem can cause very weird types of errors. In the worst case, the memory you freed is reallocated for a different purpose and the next time you write to it, you modify some completely different structure
The error message tells you that you're missing a fourth argument (#5, because "self.entity" is implicitly #1 and r,g,b are #2, #3, #4), which should be a number! Looking at the documentation (https://www.leadwerks.com/learn?page=API-Reference_Object_Entity_Camera_SetFogColor), you can see that there is also an alpha-value expected.
