The problem with this, and with casting annualInterest as an int ((int) annualInterest) is precision. interest is an int variable, annualInterest is a double variable. Here is an example of what could happen as a result of this. Suppose we have an int variable x, and a double variable y. If y = 8.8, and you try to evaluate the statement x = y/2, java assigns 4 to x. This is because 8.8/2 = 4.4, however, when java assigns 4.4 to an int value, it truncates(chops off) the decimal portion. When looking at interest rates, this could result in a problem.
interest = annualInterest / 12;
casting the variable as an int, just performs the truncation before the calculation, yielding the same result.
as for this part:
The problem lies with this: ((1/interest)-(1/(interest*Math.pow(1+interest,months)))) , this part of the equation is evaluating to 0 when your program comes across it. I would recommend running through your program with pencil and paper and evaluating the expression yourself to see where the problem may be.(it may have to do with the possible truncation from the previous part)
Anyway, hope this helps a little bit, good luck.