View Full Version : Randomly corrupt static files (css & js) - cache issue?

01-19-2012, 11:13 AM
This has been a frustrating issue that has been occurring (seemingly) at random throughout the life of our web application. I'll do my best to explain it and I hope somebody can at least point me in the right direction.

It seems to be primarily (possibly even exclusively) affecting the content when accessed securely (https prefix) and only on certain browsers (although which browsers are affected is also random.) The contents of certain js or css files become permanently corrupt unless they are modified. If I had to pick a pattern...I would say we more commonly have issues with css files becoming corrupt in Chrome...and js files in IE/Firefox. Keep in mind the files themselves don't actually change - the modified date never changes and they always look fine once downloaded via ftp. Also...months can pass without issue and some files never seem to have been affected even though they're in the same folders and accessed at the same time. When it happens, the affected browsers simply cannot load the affected file(s).

For example, we had an issue today where nobody using IE or Firefox could view certain areas of our website. However, there were no issues with Chrome. Upon inspection, it was once again an issue with a js file appearing corrupt in IE and Firefox (when accessed directly via https) - the content of the file looked fine via non-https access. On Chrome, the file looked fine regardless...and after downloading it via the FTP, deleting the existing file from the server, and uploading it back (without making any changes to the file itself) everything works fine (instantly) across all browsers.

I can find very little information on this issue anywhere on the Internet and nothing seems to resolve this thing! Unfortunately, it's also difficult to reproduce and when it happens it's so important I can't risk downtime by showing people the issue. It has to be something with the way browsers are caching the files or something...but I need to get this resolved! We had another issue that went on for days (because it was never reported) where our main css file was corrupt when viewed via Chrome. Because we had only been accessing the site ourselves in Firefox we never noticed until it was reported. However, upon downloading the "corrupt" file it looked completely fine (as is always the case.)

Our website is hosted on an Ubuntu server through Rackspace with an Apache/Varnish combo. We have mod-pagespeed running as well as memcache running. If I were to guess where to point my finger I would say Varnish...but that's nothing more than a gut feeling. This issue has become a complete nightmare because there is no warning and seemingly no way to prevent it! Also, it never seems to work itself out. When it happens, the file(s) appear corrupt/scrambled (odd symbols) to EVERYONE that atttempts to view them in the respective browser(s) having the issues until manual intervention. It also never seems to resolve itself until the file is downloaded, deleted from the server, and reuploaded. It typically affects a single file at a time but we have seen this same issue on a number of different files randomly. There seems to be no rhyme or reason to the issue!

SOMEBODY has to have experienced this before. Rebooting the server does nothing...clearing local cache does nothing. We are to the point now where we have hundreds of people relying on our website daily...so when this issue occurs we sometimes wake up to 35+ support tickets. We have lost several customers as a result - I estimate at least 10 lost customers from today's incident alone.

I am desperate and would greatly appreciate any help I could get. Thanks!

01-19-2012, 11:56 AM
Just wanted to add that I updated mod-pagespeed with strange results...so I disabled it for now. The entire website seems to be running better than ever.

This is likely completely unrelated to my issue...and I can't find any information about pagespeed causing anything like it...but just thought I would add that it is disabled for now.

01-20-2012, 07:25 PM
We also had an issue on our website today that is similar. Our site CSS and JS files in FireFox (and only FF) were corrupt. Upon closer inspection with firebug we could see the CSS/JSS files were corrupted and about 10% of each file in the users browsers cache was binary data, and not really the content on the server.

At first it looked like a server side issue, perhaps memory corruption etc.. but after doing a ctrl+F5 in firefox to refresh the page and download new content (regardless of the timestamp stored on client/server to check to see if the content has changed) the issue went away.

What is really odd is this happened to several users, not just one.. so you can't blame it on a users machine. It is undoubtedly a FF issue at this point. We do use a lot of javascript on the particular page, jQuery is one of them.

01-24-2012, 10:15 AM
Definitely a similar issue - Firebug would show similar errors in our tests also. However, the issue also occurs in IE...and clearing the cache of the local browser doesn't seem to correct the issue. In fact, when it occurs, you can type the full path to the js (or css) file in the browser and view the full document in its unreadable garbled/symbol form. The same path in Chrome, for example, returns the fully-readable file and looks completely fine (as such, it obviously loads everything fine also.)

Rebooting the server doesn't solve the issue...and this issue seems to occur on everyone's machine in the affected browsers.

Since uninstalling mod-pagespeed, we have not seen the issue. However, that in no way proves it was the culprit - we sometimes went weeks without any issues. We're just trying to eliminate variables at this point. I'm wondering if perhaps it's an issue resulting from so many different cache/compression methods running at once - an encoded file getting cached live by Varnish before being properly decoded (or something along those lines.)

Very frustrating - this issue is literally damaging to our business and nobody seems to have an answer. I'm tempted to fire up a new server and set everything up all over again from scratch.

Appreciate the response - at least I know I'm not alone!