This has been a frustrating issue that has been occurring (seemingly) at random throughout the life of our web application. I'll do my best to explain it and I hope somebody can at least point me in the right direction.
It seems to be primarily (possibly even exclusively) affecting the content when accessed securely (https prefix) and only on certain browsers (although which browsers are affected is also random.) The contents of certain js or css files become permanently corrupt unless they are modified. If I had to pick a pattern...I would say we more commonly have issues with css files becoming corrupt in Chrome...and js files in IE/Firefox. Keep in mind the files themselves don't actually change - the modified date never changes and they always look fine once downloaded via ftp. Also...months can pass without issue and some files never seem to have been affected even though they're in the same folders and accessed at the same time. When it happens, the affected browsers simply cannot load the affected file(s).
For example, we had an issue today where nobody using IE or Firefox could view certain areas of our website. However, there were no issues with Chrome. Upon inspection, it was once again an issue with a js file appearing corrupt in IE and Firefox (when accessed directly via https) - the content of the file looked fine via non-https access. On Chrome, the file looked fine regardless...and after downloading it via the FTP, deleting the existing file from the server, and uploading it back (without making any changes to the file itself) everything works fine (instantly) across all browsers.
I can find very little information on this issue anywhere on the Internet and nothing seems to resolve this thing! Unfortunately, it's also difficult to reproduce and when it happens it's so important I can't risk downtime by showing people the issue. It has to be something with the way browsers are caching the files or something...but I need to get this resolved! We had another issue that went on for days (because it was never reported) where our main css file was corrupt when viewed via Chrome. Because we had only been accessing the site ourselves in Firefox we never noticed until it was reported. However, upon downloading the "corrupt" file it looked completely fine (as is always the case.)
Our website is hosted on an Ubuntu server through Rackspace with an Apache/Varnish combo. We have mod-pagespeed running as well as memcache running. If I were to guess where to point my finger I would say Varnish...but that's nothing more than a gut feeling. This issue has become a complete nightmare because there is no warning and seemingly no way to prevent it! Also, it never seems to work itself out. When it happens, the file(s) appear corrupt/scrambled (odd symbols) to EVERYONE that atttempts to view them in the respective browser(s) having the issues until manual intervention. It also never seems to resolve itself until the file is downloaded, deleted from the server, and reuploaded. It typically affects a single file at a time but we have seen this same issue on a number of different files randomly. There seems to be no rhyme or reason to the issue!
SOMEBODY has to have experienced this before. Rebooting the server does nothing...clearing local cache does nothing. We are to the point now where we have hundreds of people relying on our website daily...so when this issue occurs we sometimes wake up to 35+ support tickets. We have lost several customers as a result - I estimate at least 10 lost customers from today's incident alone.
I am desperate and would greatly appreciate any help I could get. Thanks!