Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Reading and writing in same file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reading and writing in same file
# 1  
Old 09-05-2012
Reading and writing in same file
Hi All,

Here is my requirement. I am grepping through the log files and cutting some fields from the file to generate a csv file. Now I have to check if 2nd field is having some fixed value then with the help of 4th field I have to look in same log and run another grep command to retrieve the corresponding 2nd field to replace the fixed value.

Example:-
Code:
12345,none1111,55,link1
56789,dsadsad,66,ewqrrwe
23456,none1111,77,link2
65655,yuytuytds,88,ertywd

If second column equals 'none1111' then I have to run a grep command on same log file by searching for link1
Suppose the grep command returns me value 'abcde' for first row then I have to replace the 'none1111' with 'abcde'.
Same applies for link2.

Final output:-
Code:
12345,abcde,55,link1
56789,dsadsad,66,ewqrrwe
23456,efghi,77,link2
65655,yuytuytds,88,ertywd

Thanks in Advance.
Last edited by Franklin52; 09-06-2012 at 04:17 AM.. Reason: Please use code tags for data and code samples
# 2  
Old 09-06-2012
Quote:
Originally Posted by kmajumder
...
Example:-

12345,none1111,55,link1
56789,dsadsad,66,ewqrrwe
23456,none1111,77,link2
65655,yuytuytds,88,ertywd

If second column equals 'none1111' then I have to run a grep command on same log file by searching for link1
Suppose the grep command returns me value 'abcde' for first row then I have to replace the 'none1111' with 'abcde'.

Final output:-

12345,abcde,55,link1
56789,dsadsad,66,ewqrrwe
23456,efghi,77,link2
65655,yuytuytds,88,ertywd

...
Where is 'abcde' and 'efghi' in your sample (original) data?

tyler_durden
# 3  
Old 09-06-2012
Hi tyler,

1. 'abcde' and 'efghi' wont be there in original data. That is what I tried to explain.
The original data is not having the expected value. 'none1111' is the default value if no proper data is found after the 1st grep command.
2. So we have to run a 2nd grep command to replace those default value with expected value with the help of 4th column.
3. 4th column is the only connector b/w the two different log to get the actual 2nd column. So from the first set of data we need to pick up the 4th column (whose 2nd column is 'none1111') and we need to look for other log where we have the same 4th column to pick up the right 2nd column from next log.

Example:- First set of data
1. Consider some grep command gave below result.

Code:
12345,none1111,55,link1
56789,dsadsad,66,ewqrrwe
23456,none1111,77,link2
65655,yuytuytds,88,ertywd

2. In second column we get 'none1111' that means we have to run another grep command to pick up right 2nd column value.
Consider we ran another different grep command which takes 'link1' as input and return expected 2nd column i.e 'abcde'.
3. Then we have to replace all the 'none1111' value by iterating the first set of data and following the same procedure.

Hope it makes a bit clear to you.
Last edited by Scrutinizer; 09-07-2012 at 04:26 AM.. Reason: Code tags
# 4  
Old 09-06-2012
kinda long.
Code:
#!/bin/sh

while read line
do
        flag=`echo $line | grep "none1111"`
        if [ ! -z $flag ] ; then
                lastcol=`echo $line | awk -F, '{print $4}'`
                if [ $lastcol = "link1" ] ; then
                        echo $line | sed 's/none1111/abcde/' >> out
                elif [ $lastcol = "link2" ] ; then
                         echo $line | sed 's/none1111/efghi/' >> out
                fi
        else
                echo $line  >> out
        fi
done < inputfile

Last edited by ryandegreat25; 09-06-2012 at 04:12 AM..
# 5  
Old 09-06-2012
Quote:
Originally Posted by kmajumder
Hi tyler,

1. 'abcde' and 'efghi' wont be there in original data. That is what I tried to explain.
The original data is not having the expected value. 'none1111' is the default value if no proper data is found after the 1st grep command.
2. So we have to run a 2nd grep command to replace those default value with expected value with the help of 4th column.
3. 4th column is the only connector b/w the two different log to get the actual 2nd column. So from the first set of data we need to pick up the 4th column (whose 2nd column is 'none1111') and we need to look for other log where we have the same 4th column to pick up the right 2nd column from next log.

Example:- First set of data
1. Consider some grep command gave below result.

12345,none1111,55,link1
56789,dsadsad,66,ewqrrwe
23456,none1111,77,link2
65655,yuytuytds,88,ertywd

2. In second column we get 'none1111' that means we have to run another grep command to pick up right 2nd column value.
Consider we ran another different grep command which takes 'link1' as input and return expected 2nd column i.e 'abcde'.
3. Then we have to replace all the 'none1111' value by iterating the first set of data and following the same procedure.

Hope it makes a bit clear to you.
What is clear to me is:
  1. You have a log file. We don't know what this log file looks like.
  2. You have run grep on the this log file and manipulated the lines that came back to produce a CSV file containing four columns.
  3. You won't tell us how you manipulated the data you grepped to produce the CSV file.
  4. You have another log file. We don't know what this log file looks like either.
  5. You want to match the 4th field in the CSV file you created against something in this second log file and replacenone1111with something else in the second log file.
  6. You aren't giving us enough data to help you.
Show us the names and contents of both log files. Explain the procedures you use to create your CSV file from the first log file. Show us the data in the 2nd log file that we are to match and show us how we determine the replacement fornone1111when we find the correct line in the 2nd log file. Then we may be able to help you.

Note that ryandegreat25 gave you a script that does what you have requested, for the values you've shown us, but almost certainly is not a general solution to the problem you're trying to solve.

Please give us enough information to be able to help you.Smilie
# 6  
Old 09-06-2012
While each of Don Cragun's arguments holds true, just for the exercise I'm trying to infer the logics from your first post: field 4 is the link that glues together two records, of which field 2 is filled with "none1111" if the system did not have the correct value yet by the time of file creation. Both records need to be included at their original location in the file, and "none1111" needs to be replaced by the correct value found later in the same file for field 4. I'm sure there will be much more elegant solutions, but, as you are grepping several times, so do I. We need to run through the input file at least three times - a) to find the field 4 values, b) to find the field 2 values, and c) to replace. Step b) will be repeated for every pattern occurrence in the ifile. Here we go:
Code:
grep none1111 infile |
      { IFS=","; while read a b c d; do grep $d infile |
            { while read e f g h; do [ $f != 'none1111' ] &&
                  echo s/none1111\\\(.*$h\\\)/$f\\1/ >>sedfile; done;
            }; done
      }; sed -f sedfile infile; rm sedfile

Well, this one would remove the problem of repeated grep runs using xargs (whose known problem hopefully won't hit us in this case):
Code:
grep none1111 infile |
      { IFS=","; while read a b c d; do echo -n $d"|"; done; echo "#"; } |
      xargs -I xy grep -E "xy" infile |
      { IFS=","; while read e f g h; do [ $f != 'none1111' ] &&  echo s/none1111\\\(.*$h\\\)/$f\\1/ >>sedfile; done; }
      sed -f sedfile infile; rm sedfile

And, finally, a "oneliner" in which all parameters for sed are being created in a command substitution:
Code:
sed $(
      grep none1111 infile | cut -d, -f4 | xargs -Ixy grep xy infile | cut -d, -f2,4 | grep -v none1111 |
      sed 's/\(.*\),\(.*\)/-e s#none1111\\\(.*\\\)\2#\1\\1\2#/'
     ) infile

Like awk better? Try this:
Code:
awk     'BEGIN {FS=OFS=","}
         {n = split($0, g); j++; for (i=1; i<=n; i++) h[j,i]=g[i]}
         !  /none1111/ {f[$4]=$2}
         END {for (i=1; i<=j; i++) print h[i,1], h[i,2]=="none1111" ? f[h[i,4]] : h[i,2], h[i,3], h[i,4]}
        '  infile

Last edited by RudiC; 09-06-2012 at 03:09 PM..
# 7  
Old 09-06-2012
Sorry for the confusion
Let me give you a complete example what I am trying to achieve.

1. Below is the log file structure where I need 2,5 and 14th column of the logs after grepping through the linkId=1ddoic.

Log file structure:-

abc.com 20120829001415 127.0.0.1 app none11111 sas 0 0 N clk Mozilla/5.0 id=82c6a15ca06b2372c3b3ec2133fc8b14 referrer=google.com linkId=1ddoic

abc.com 20120829001416 127.0.0.1 dyn UD3BSAp8appncXlZ UD3BSAp8app4xHbz 0 0 N page Mozilla/5.0 id=82c6a15ca06b2372c3b3ec2133fc8b14 segments=

2. Now for the 1st log you can see I have invalid(none11111) 5th column. So I have to look for the actual 5th column value. 'id' column will help you to find that. So you have to run another grep based on the 'id' value so that you can find the actual 5th column in the same log file.
3. If you see the second log it has the exact matching 'id' value. So what I have to do I have to take the 5th column(UD3BSAp8appncXlZ) from the second log instead of the invalid one(none11111).

Output:-

20120829001415, UD3BSAp8appncXlZ, linkId=1ddoic

Note:- I have bunch of log files where I have to perform the above procedure. But I have to come up with a single file as output after grepping through all the log files.
It has a format like abc-2012-10-01_00000,abc-2012-10-01_00001.... etc.


Hope this time it makes clear.Smilie
Thanks for looking into it.
Last edited by kmajumder; 09-06-2012 at 05:48 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading and Writing a conf file - Suggestions and improvements?

Hello all As part of my TUI - (line based) Text User Interface, i do have 2 commands to assist working with conf files. Now, for me they work, but since i wrote them, i automaticly use them they way they should be used... you know what i mean. ;) Anyway, they are designed to read 'simple'... (3 Replies)
Discussion started by: sea
3 Replies

2. Shell Programming and Scripting

reading a file extracting information writing to a file

Hi I am trying to extract information out of a file but keep getting grep cant open errors the code is below: #bash #extract orders with blank address details # # obtain the current date # set today to the current date ccyymmdd format today=`date +%c%m%d | cut -c24-31` echo... (8 Replies)
Discussion started by: Bruble
8 Replies

3. Programming

unexpected values received when writing and reading from file

In the code below i try to write and read from a file, but i get unexpected results, like after writing i cannot open the file, and when reading the file the value entered earlier is not shown bool creat_fragments(int nFragment) { int fd, rand_value; char frag_path, buf; for(int... (8 Replies)
Discussion started by: saman_glorious
8 Replies

4. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

5. Shell Programming and Scripting

Reading data from DataBase and Writing to a file

Hi All, Please help me in writing data to a file in one row. In database there is a column which contains large data which does not fit in the file in one row. The column contains list of paths. I want to write these paths to a file in one row. Please find the code below writes : ... (2 Replies)
Discussion started by: rajeshorpu
2 Replies

6. Programming

I need help with file reading/writing in C

Hello everybody, I'm trying to code a program which makes the following: It sends an ARP request frame and when it gets the reply, extracts the IP address of source and writes it to a .txt file. This is gonna be done with many hosts (with a for() loop), so, the text file would look like... (2 Replies)
Discussion started by: Zykl0n-B
2 Replies

7. UNIX for Dummies Questions & Answers

Log File Writing and Reading

Hi all, I have the following shell script code which tries to sftp and writes the log into the log file. TestConnection () { echo 'Connection to ' $DESTUSERNAME@$DESTHOSTNAME $SETDEBUG if ]; then rm $SCRIPT ; fi touch $SCRIPT echo "cd" $REMOTEDIR >> $SCRIPT echo "quit" >>... (10 Replies)
Discussion started by: valluvan
10 Replies

8. UNIX for Dummies Questions & Answers

reading ,writing,appending ,manipulating a file.

Hi my prob statement is to create a new file or to append to the 1tst file the followign chages. File 1: txt file. portfolio No a b c d abc 1 Any Any Any charString cds 2 values values values charString efd 3 can can can charString fdg 4 come come come charString... (4 Replies)
Discussion started by: szchmaltz
4 Replies

9. UNIX for Advanced & Expert Users

Reading a file and writing the file name to a param file.

Hi All, Not sure if this would be in a dummies sectiin or advanced. I'm looking for a script if someone has doen something like this. I have list of files - adc_earnedpoints.20070630.txt adc_earnedpoints.20070707.txt adc_earnedpoints.20070714.txt adc_earnedpoints.20070721.txt... (1 Reply)
Discussion started by: thebeginer
1 Replies

10. Programming

Reading and Writing file on LAN

Hi gurus I am not a C programmer but I need to read and write files on a computer on LAN using IP address. Suppose on a computer that has an IP 192.168.0.2 Any help or code example. I did in JAVA using URL, but do not know how to do in ANSI-C. In java: ------- URL url = new... (3 Replies)
Discussion started by: lucky001
3 Replies
Login or Register to Ask a Question