You're over-thinking what you need to do with this.
Since you already have a method to intersect, and you can cast to a list, its simple to remove what is in a collection from another.
PHP Code:
public static Integer[] difference(Integer[] lhs, Integer[] rhs)
{
ArrayList<Integer> alResult = new ArrayList<Integer>(Arrays.asList(lhs));
alResult.removeAll(Arrays.asList(intersect(lhs, rhs)));
return alResult.toArray(new Integer[alResult.size()]);
}
That assumes that the intersect's returned IntSet is castable to a list as you have here in your example.
Unless I misunderstand what you are referring to as the difference, that should do what you are needing.