You're over-thinking what you need to do with this.
Since you already have a method to intersect, and you can cast to a list, its simple to remove what is in a collection from another.
public static Integer difference(Integer lhs, Integer rhs)
ArrayList<Integer> alResult = new ArrayList<Integer>(Arrays.asList(lhs));
return alResult.toArray(new Integer[alResult.size()]);
That assumes that the intersect's returned IntSet is castable to a list as you have here in your example.
Unless I misunderstand what you are referring to as the difference, that should do what you are needing.