1
2
3
4
2
3
4
> x = 10^-10 > y = x > =(cos(x*y)-1)/(x*y*x*y) 0
Looks like it's a pretty simple answer.
Now let's be a bit more pedantic and check with wolframalpha
http://www.wolframalpha.com/input/?i=%5Clim_%7Bx+%5Crightarrow+0%7D%5Cfrac%7Bcos%28x%5E2%29-1%7D%7Bx%5E4%7D
-1/2? What the hell, Lua just told me that the answer is 0. What went wrong?
Here's your problem: Try to explain why
1
(-sin(x*y)^2)/(x*y*x*y*(cos(x*y)+1))
gives the correct answer of -0.5
but
1
(cos(x*y)-1)/(x*y*x*y)
does not.
Hint: every input and computation is associated with a relative error delta that is less than some constant epsilon that we can call the machine epsilon = O(2^-52), so try to figure out why subtraction amplifies this relative error per computation
These types of truncation errors occur often in numerical computation problems (especially within large linear systems) and are often referred to as cancellation error.