Ok I have decided after a lot of reading to forget the last modified header since I have come across this 'unset Last-Modified' several times.
And since the htaccess I have above doesn't seem to cause any issuse I'll just leave it as is even tho I have no idea if those headers are actually set.
Ask not what can I do for myself, but what can I do for others
"The greatest revenge is to accomplish what others say you cannot do."
I used to be indecisive, but now I'm not so sure.