I've never done it and never seen it done, but it should be possible. I believe the javac application itself is written in java so it would just be a matter of finding the api and bundling those classes with your application. I found this one link to a similar question.
Here is another link to a likely even more useful thread I found on javaranch where they essentially discuss your question. They even have some sample code (not for the faint of heart).