I have written a perl script that will load the entire data file into an array and then I would check the value of the specific column and then if interested I will write to a good file else I will write it to a bad file.
But here, the problem is that if the data file is a huge file then storing in an array would cause a memory utilization issue. So i thought I have to read the data file line by line and then check for the column values.
Instead of storing the entire data file into an array (memory) , could someone please advice how can I read the data file line by line so that it doesn't uses much memory.
Really appreciate your thoughts and time. Thanks a lot for looking into this.
I think Corona's suggestion is ok, but inside the loop you are saving each input line to arrays, witch are filling the memory. This piece of code:
Regards,
Birei
I have a data file which contains ~3.5Million records and would have a header and a footer.It has many number of columns with a pipe ("|") delimited.
Example:
I need to check the values for the columns 21,22,23 and then write to a file(Badfile) if they are matched else write to a different file.(good file)
Once the files are written , I need to include the header and footer to the Good file.
Since the data file is little huge , Memory is filling up.
Hi All,
i want to write a shell script read below file line by line and want to exclude the lines which contains empty value for MOUNTPOINT field.
i am using centos 7 Operating system.
want to read below file.
# cat /tmp/d5
NAME="/dev/sda" TYPE="disk" SIZE="60G" OWNER="root"... (4 Replies)
Hi,
I want to read a live log file line by line and considering those line which start from time stamp;
Below code I am using, which read line but throws an exception when comparing line that does not contain error code
tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Hi,
i have a scenario were i should compare a few fields from each line then increment a variable based on that.
Example file
989878|8999|Y|0|Y|N|V
989878|8999|Y|0|N|N|V
989878|8999|Y|2344|Y|N|V
i have 3 conditions to check and increment a variable on every line
condition 1 if ( $3... (4 Replies)
I need some help. I would like to read in a text file.
Take a variable such as ROW-D-01, compare it to what's in one line in the text file such as PROD/VM/ROW-D-01 and only input PROD/VM into a variable without the /ROW-D-01.
Is this possible? any help is appreciated. (2 Replies)
So, the beginning of my script will cat & grep a file with the output directed to a new file. The data I have in this file needs to be parsed, read and evaluated.
Basically, I need to identify the latest date/time stamp and then calculate whether or not it is within 15 minutes of the current... (1 Reply)
So, I want to read line-by-line a text file with unknown number of files....
So:
a=1
b=1
while ; do
b=`sed -n '$ap' test`
a=`expr $a + 1`
$here do something with b etc
done
the problem is that sed does not seem to recognise the $a, even when trying
sed -n ' $a p'
So, I cannot read... (3 Replies)
Hi,
I am a beginner in shell scripting. I have written the following script, which is supposed to process the while loop for each line in the sid_home.txt file. But I'm getting the 'end of file' unexpected for the last line. The file sid_home.txt gets generated as expected, but the script... (6 Replies)
HI
I need to read each line (test.txt) and store it in a array (@test)
How to do it in perl.
Suppose i have a file test.txt. I have to read each line of the test.txt file and store it in a array @test. How to do it in perl.
Regards
Harikrishna (3 Replies)