It's not just Processing that doesn't allow square roots of negative numbers; it's the entire family of real numbers that don't allow it :-)
You are not deriving the c2 variable correctly It should be normalized to the 0..1 range, because it's divided by double the product of L1 and L2.
Have you properly implemented this line?
(x^2 + y^2 - L1^2 - L2^2) / (2 * L1 * L2)
Also, note that the "^" operator in C means "bit or" whereas in this code, it's intended to indicate squaring (so the actual C code would say x*x, not x^2).
If you look at what the values mean, "x*x + y*y" is just "square of distance to x,y coordinate" which must be less than the length of the two links, or the arm won't reach. Thus, it is possible that the numbers you're trying to put in are simply impossible to satisfy.
Another option for what could be going wrong is the frame of reference -- is the origin (x=0, y=0) at the center of the J1 joint, or somewhere else? (The tutorial doesn't mention this, but it looks like it assumes the x,y coordinates of the J1 joint are 0,0)