|
Looping through and comparing is a tremendous overhead for this. Depending on the size of the file as well. To me, it sounds like you may have an improper comparison type likely during the selection process, so you'd need to check that the datatype comparison matches the field datatype.
For a batch insert like this, I'd suggest abandoning the check completely. Use an INSERT IGNORE or an ON DUPLICATE KEY syntax if the new record is potentially different requiring an update instead. No need to see if it already exists, if it does it will fail to insert and keep going. If you are using a language like PHP, there will be an affected rows method or value that can be used to see how many records were actually inserted (if this is of any value to you). The client will tell you the same, but will tell you the number of records total and how many duplicates it encountered.
|