Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 5 of 5
  1. #1
    New Coder
    Join Date
    Oct 2005
    Posts
    46
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Searching millions of files.

    Hi,

    I'm currently writing an email application, it reads a pop box and stores the mail body on the file system.

    Theres currently nearly 1 million files on the file system called EMAIL_ID.dat

    EMAIL_ID being the same as the other details for that mail in the db.

    The reason i chose the file system to store the mail body is because the mysql table was too big the other way. over 2gig and ORDER BY was very slow.

    Anyone got any ideas on how i can search the file system quickly or if indeed it can be done better ?

    Cheers
    Mart

  • #2
    Regular Coder
    Join Date
    Jun 2007
    Location
    Los Angeles
    Posts
    545
    Thanks
    81
    Thanked 5 Times in 5 Posts
    Can you please clarify:

    1. You are storing your email body text in a file system vs a database correct?

    2. When you had the body in a database what field data type were you using?

    3. Did you index that email body database field or no?

    I have an idea but would like to know the above answers please.
    RalphF
    Business Text Messaging Services
    https://www.MobileTextingService.com

  • #3
    New Coder
    Join Date
    May 2006
    Posts
    62
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by mattis2k View Post
    Hi,

    I'm currently writing an email application, it reads a pop box and stores the mail body on the file system.

    Theres currently nearly 1 million files on the file system called EMAIL_ID.dat

    EMAIL_ID being the same as the other details for that mail in the db.

    The reason i chose the file system to store the mail body is because the mysql table was too big the other way. over 2gig and ORDER BY was very slow.

    Anyone got any ideas on how i can search the file system quickly or if indeed it can be done better ?

    Cheers
    Mart
    You will likely have to create a relevant index of keywords from the files in mysql that would allow quick lookups. Try looking at how one of the open source search engines would index those files to figure out how to create the index.
    Fiji Web Design - where i do Joomla Web Design
    Bucabay.com - My blog

  • #4
    New Coder
    Join Date
    Oct 2005
    Posts
    46
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by rfresh View Post
    Can you please clarify:

    1. You are storing your email body text in a file system vs a database correct?

    2. When you had the body in a database what field data type were you using?

    3. Did you index that email body database field or no?

    I have an idea but would like to know the above answers please.
    Hi,

    Thanks for the reply.. I inherited this system, the email body was stored as a TEXT field, there is currently no index on it in the previous system.

    I'm open to using the database to store them, but not sure if having a db with large TEXT fields is efficient.

  • #5
    New Coder
    Join Date
    Oct 2005
    Posts
    46
    Thanks
    0
    Thanked 0 Times in 0 Posts
    anyone ?


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •