1. Hello Guest! It's with a heavy heart that we must announce the removal of the Legacy GMC Archive. If you wish to save anything from it, now's the time! Please see this topic for more information.
    Dismiss Notice

Large file question

Discussion in 'Programming' started by William Roberts, Apr 1, 2019.

  1. William Roberts

    William Roberts Member

    Joined:
    May 11, 2018
    Posts:
    12
    I have a file that has over 150,000 lines in it with 3 fields (comma delimited). Right now, I am reading the entire file into an array... But I feel like this isn't the smartest approach. Is there a better way to read a large file? Thanks for the support.

    Will
     
  2. FrostyCat

    FrostyCat Member

    Joined:
    Jun 26, 2016
    Posts:
    4,684
    All those 150k lines will be read anyways, so you probably won't find any room for optimization in the reading loop. If you do want to optimize this, look into how the data is structured after you read it. For example, if you will be indexing them like a dictionary, then reading into a map is much smarter than reading into an array.
     
  3. dannyjenn

    dannyjenn Member

    Joined:
    Jul 29, 2017
    Posts:
    569
    If the text file doesn't need to be human-readable, then maybe you could format the data as a buffer and then use buffer_read() instead of a text file? Just a thought.
     
  4. Pfap

    Pfap Member

    Joined:
    Apr 30, 2017
    Posts:
    551
    To expand on what @dannyjenn said, you could also load it asynchronously if you were to use buffers and the function buffer_load_async(). Then it would not block while it is loading, although I haven't really gotten it to work. The manual basically says it's the only way to load for html5. Otherwise, efficiency when reading a file doesn't matter too much as it will just block until the full file is read; which is what @FrostyCat was pointing out. Once, all that data is in memory how you use it is what will matter.
     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice