...

View Full Version : Mysql crashed site and keeps giving error 500 :(



prezaeis
12-28-2011, 09:41 PM
hi guys
i run a drupal site my sites been down for a couple of days now after crashing
i installed a new theme and was working on themeing it and moving blocks around and then suddenly the site crashed and wont load again, i keep getting server 500 errors.

im on a dedicated server and iv tried restarting the server, nothing works

iv pasted below a copy of mysql log file, does anyone understand it? i tried repairing the crashed tables it mention in it but still no joy

i should note i have a large site with a lot of music on it and quite high traffic.. i also checked my hard disk space using root access and i have enough space available, i have copied the results also below:



[ ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 3.7G 346M 3.4G 10% /
/dev/mapper/vg00-usr 4.0G 1.2G 2.9G 30% /usr
/dev/mapper/vg00-var 60G 16G 45G 27% /var
/dev/mapper/vg00-home
4.0G 4.2M 4.0G 1% /home
none 990M 4.0K 990M 1% /tmp
tmpfs 990M 0 990M 0% /usr/local/psa/handlers/before-local
tmpfs 990M 0 990M 0% /usr/local/psa/handlers/before-queue
tmpfs 990M 0 990M 0% /usr/local/psa/handlers/before-remote
tmpfs 990M 0 990M 0% /usr/local/psa/handlers/info
tmpfs 990M 0 990M 0% /usr/local/psa/handlers/spool




For some reason i couldnt paste the whole mysql log file here, it wont show up, so i removed some of the code that was being repeated over and over again in the log file:

MYSQL LOG FILE



101222 16:34:42 mysqld started
101222 16:34:42 InnoDB: Started; log sequence number 0 949146
101222 16:34:42 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
101230 4:22:33 [Note] /usr/libexec/mysqld: Normal shutdown

101230 4:22:33 InnoDB: Starting shutdown...
101230 4:22:35 InnoDB: Shutdown completed; log sequence number 0 1703924
101230 4:22:35 [Note] /usr/libexec/mysqld: Shutdown complete

101230 04:22:35 mysqld ended

101230 04:24:17 mysqld started
101230 4:24:18 InnoDB: Started; log sequence number 0 1703924
101230 4:24:18 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110321 0:36:32 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 0:46:38 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 0:56:49 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 1:06:54 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 1:16:59 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 1:27:04 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 1:36:34 [Note] /usr/libexec/mysqld: Normal shutdown

110321 1:36:36 [Warning] /usr/libexec/mysqld: Forcing close of thread 235813 user: 'phh2011'

110321 1:37:10 InnoDB: Starting shutdown...
110321 1:37:12 InnoDB: Shutdown completed; log sequence number 0 46716386
110321 1:37:12 [Note] /usr/libexec/mysqld: Shutdown complete

110321 01:37:12 mysqld ended

110321 01:38:57 mysqld started
110321 1:38:58 InnoDB: Started; log sequence number 0 46716386
110321 1:38:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110321 1:39:55 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 2:20:15 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 2:29:49 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_filter.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110321 2:30:20 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD'110321 23:40:11 [Note] /usr/libexec/mysqld: Normal shutdown

110321 23:40:13 [Warning] /usr/libexec/mysqld: Forcing close of thread 919 user: 'phh2011'

110321 23:40:13 [Warning] /usr/libexec/mysqld: Forcing close of thread 881 user: 'phh2011'

110321 23:40:21 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './phh2011/cache_menu.MYI'; try to repair it
110321 23:40:27 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './phh2011/cache_content.MYI'; try to repair it
110321 23:40:27 InnoDB: Starting shutdown...
110321 23:40:28 InnoDB: Shutdown completed; log sequence number 0 46853392
110321 23:40:28 [Note] /usr/libexec/mysqld: Shutdown complete

110321 23:40:28 mysqld ended

110321 23:42:40 mysqld started
110321 23:42:40 [Warning] Can't create test file /var/lib/mysql/s15428472.lower-test
110321 23:42:40 [Warning] Can't create test file /var/lib/mysql/s15428472.lower-test
110321 23:42:41 InnoDB: Started; log sequence number 0 46853392
110321 23:42:41 [ERROR] Can't start server : Bind on unix socket: No space left on device
110321 23:42:41 [ERROR] Do you already have another mysqld server running on socket: /var/lib/mysql/mysql.sock ?
110321 23:42:41 [ERROR] Aborting

110321 23:42:41 InnoDB: Starting shutdown...
110321 23:42:42 InnoDB: Shutdown completed; log sequence number 0 46853392
110321 23:42:42 [Note] /usr/libexec/mysqld: Shutdown complete

110321 23:42:42 mysqld ended

110322 00:01:16 mysqld started
110322 0:01:16 InnoDB: Started; log sequence number 0 46853392
110322 0:01:16 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:16 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:16 mysqld restarted
110322 0:01:16 InnoDB: Started; log sequence number 0 46853392
110322 0:01:17 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:17 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:17 mysqld restarted
110322 0:01:17 InnoDB: Started; log sequence number 0 46853392
110322 0:01:18 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:18 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:18 mysqld restarted
110322 0:01:18 InnoDB: Started; log sequence number 0 46853392
110322 0:01:18 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:18 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:18 mysqld restarted
110322 0:01:18 InnoDB: Started; log sequence number 0 46853392
110322 0:01:19 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:19 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:19 mysqld restarted
110322 0:01:19 InnoDB: Started; log sequence number 0 46853392
110322 0:01:19 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:19 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:20 mysqld restarted
110322 0:01:20 InnoDB: Started; log sequence number 0 46853392
110322 0:01:20 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:20 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:01:20 mysqld restarted
110322 0:01:20 InnoDB: Started; log sequence number 0 46853392
110322 0:01:21 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 0:01:21 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 00:110322 1:11:26 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
110322 1:11:27 [ERROR] Can't start server: can't create PID file: No space left on device

Number of processes running now: 0
110322 01:11:27 mysqld restarted
110322 1:11:27 InnoDB: Started; log sequence number 0 46853392
110322 1:11:27 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110322 1:12:20 [Note] /usr/libexec/mysqld: Normal shutdown

110322 1:12:20 InnoDB: Starting shutdown...
110322 1:12:22 InnoDB: Shutdown completed; log sequence number 0 46860641
110322 1:12:22 [Note] /usr/libexec/mysqld: Shutdown complete

110322 01:12:22 mysqld ended

110322 01:13:59 mysqld started
110322 1:14:00 InnoDB: Started; log sequence number 0 46860641
110322 1:14:00 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110322 1:14:46 [ERROR] /usr/libexec/mysqld: Table './phh2011/cache_menu' is marked as crashed and should be repaired
110322 1:14:46 [ERROR] /usr/libexec/mysqld: Table './phh2011/cache_menu' is marked as crashed and should be repaired
110322 1:14:46 [ERROR] /usr/libexec/mysqld: Table './phh2011/cache_menu' is marked as crashed and should be repaired
110322 1:14:46 [ERROR] /usr/libexec/mysqld: Table './phh2011/cache_menu' is marked as crashed and should be repaired
110322 1:14:46 [ERROR] /usr/libexec/mysqld: Table './phh2011/cache_menu' is marked as crashed and should be repaired

110328 21:09:14 [Note] /usr/libexec/mysqld: Normal shutdown

110328 21:09:14 InnoDB: Starting shutdown...
110328 21:09:17 InnoDB: Shutdown completed; log sequence number 0 47206606
110328 21:09:17 [Note] /usr/libexec/mysqld: Shutdown complete

110328 21:09:17 mysqld ended

110328 21:09:19 mysqld started
110328 21:09:19 InnoDB: Started; log sequence number 0 47206606
110328 21:09:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

Number of processes running now: 0
110411 16:17:06 mysqld restarted
110411 16:53:43 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110411 16:53:44 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110411 16:53:45 InnoDB: Started; log sequence number 0 47561352
110411 16:53:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110524 15:22:44 [ERROR] /usr/libexec/mysqld: Disk is full writing '/var/lib/mysql/phh2011/files.MYI' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110525 3:26:14 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110525 3:30:32 [ERROR] /usr/libexec/mysqld: Disk is full writing '/var/lib/mysql/phh2011/url_alias.MYI' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110525 3:36:19 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
110525 3:40:37 [ERROR] /usr/libexec/mysqld: Disk is110525 03:45:42 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110525 3:45:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110525 3:45:44 InnoDB: Started; log sequence number 0 181902913
110525 3:45:44 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110525 4:01:35 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './phh2011/url_alias.MYI'; try to repair it
110525 4:01:35 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './phh2011/url_alias.MYI'; try to repair it
110525 4:02:23 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './phh2011/url_alias.MYI'; try to repair it
110525 4:02:23 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './phh2011/url_alias.MYI'; try to repair it

110525 4:45:01 [Note] Found 2512 of 2511 rows when repairing './phh2011/url_alias'
110601 15:44:06 [Note] /usr/libexec/mysqld: Normal shutdown

110601 15:44:06 InnoDB: Starting shutdown...
110601 15:44:08 InnoDB: Shutdown completed; log sequence number 0 228158435
110601 15:44:08 [Note] /usr/libexec/mysqld: Shutdown complete

110601 15:44:08 mysqld ended

110601 15:45:57 mysqld started
110601 15:45:58 InnoDB: Started; log sequence number 0 228158435
110601 15:45:58 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110613 05:30:50 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110613 5:30:50 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110613 5:30:51 InnoDB: Started; log sequence number 0 245155726
110613 5:30:51 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
110615 19:04:11 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110615 19:04:11 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110615 19:04:13 InnoDB: Started; log sequence number 0 245271838
110615 19:04:13 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

Number of processes running now: 0
110902 02:29:26 mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
110902 6:21:57 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
110902 6:30:19 InnoDB: Started; log sequence number 0 246683547
110902 6:30:39 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
111010 15:46:40 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
111010 15:46:41 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
111010 15:46:44 InnoDB: Started; log sequence number 0 247215587
111010 15:46:44 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

Number of processes running now: 0
111023 06:30:00 mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
111023 7:00:52 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
111023 7:21:10 InnoDB: Started; log sequence number 0 247372715
111023 7:21:11 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
111029 14:29:49 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
111029 14:29:50 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
111029 14:29:51 InnoDB: Started; log sequence number 0 247412191
111029 14:29:51 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

Number of processes running now: 0
111104 06:55:14 mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
111104 7:00:02 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
111104 7:00:05 InnoDB: Started; log sequence number 0 247573533
111104 7:00:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution

prezaeis
12-28-2011, 09:42 PM
any help would be muchh appreciated :(

Old Pedant
12-28-2011, 10:29 PM
I think it is self-explanatory.

Look at all the messages of the form

110525 3:26:14 [ERROR] /usr/libexec/mysqld: Disk is full writing './phh2011/cache_menu.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
MySQL simply had no space left on that disk at that time.

After that point, your database was apparently badly corrupted, as witness the many "Starting crash recovery" and similar messages.

In the process of all the recovery and repair attempts, you apparently freed up space on the drive, but by now the damage is long done.

The best I can suggest is to see if you can get MySQL to come up *AT ALL* and, if so, dump out all the tables you can. I suspect that you will get errors from some of them during the dump process and they simply may be lost.

Part of your ongoing problem will be: Where do you dump the tables to if the disk is near full? Will there be enough space on the disk for the ".sql" dump files?

I hate to say it, but you are probably going to lose a lot of data if you didn't have it backed up somewhere.

Anyway, if you *can* dump out the tables, then I'd wipe the database(s) clean and start over, loading the dumped tables from scratch.

prezaeis
12-28-2011, 11:45 PM
thank u so much for the detaild info

before doing all that can i allocate space from one partiton to where mysql is? so that i dont run into this problem again?

Old Pedant
12-29-2011, 12:18 AM
Depends on the operating system you are using and/or what utilities you have available to you.

You say this is a dedicated server, yes? So why do you need/want to use partitions at all? If you only have the one disk drive, you might be better off making it all one partition.

Anyway, it's too generic a question to give an absolute answer to.

prezaeis
12-29-2011, 03:50 AM
im sorry to keep bugging you

i made a dump of my whole database and downloaded it into a folder on my ftp (database.sql). it is 143mb in size

i have created a new drupal install with a fresh database on a seperate domain name and i have now moved the database.sql file from the previous server to the new one. how can i now import the data from within that sql dump into the new database?

could u help me with the command? im quite new to mysql and ssh and worry im going to mess it up

much appreciated

Old Pedant
12-29-2011, 08:42 PM
Ther easiest way is from the command line:



mysql -uUSERNAME -pPASSWORD < dumpfile.sql

That is, you just redirect STDIN to the mysql command processors and you are done.

143 MB is big but not that big. How long it takes depends on how many indexes you have/had on the table(s).

If your DUMP didn't specify the database name, you might need to put that in the command line:


mysql -uUSERNAME -pPASSWORD -DDATABASENAME < dumpfile.sql



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum