Easier method for deleting oldest lines from a data file as it grows?

[Kent Searight 0]

I'm using Write To Data File script step to keep track of the outcomes of batch transactions in a text file. It's basically a log that users can view to see line-by-line what happened when they ran a batch process (which records succeeded, failed, queued for retry, etc.).

Entries into this log have a lot of value to users in the short term, but have little to no value to them as they age.

I'd like to purge the oldest entries (at the beginning/top of the file) when the file reaches a certain size. For example, if I want the max file size to be 128K and the new entries make it 130K I'd like to remove the oldest 2K of text in the file at a ¶ or line feed (so I'm only deleting whole lines, not partials).

The only way I've figured out how to do this is to read the entire file into a variable, use the Right() function to grab the rightmost 128K, apply some logic to locate the return, then write over the file with the new data in the variable. Is there an easier way of doing this? I feel like I'm taking a Rube Goldberg approach.

Thanks in advance!

-Kent

[comment 1,753]

Putting all other kinds of potential issues aside for a moment, my main worry with the method you describe would be leaving a tail of old data when the new data is shorter than the current contents of the data file.

What would be the advantage of managing data stored in an external text file, when you have a database application at your disposal? If, for some reason, users prefer to view the log data in a text file, the simplest method would be to export it - overwriting the previous file.

 

[Kent Searight 0]

Exporting is a good option. Thank you.