Sponsored Content
Top Forums Shell Programming and Scripting Reading and writing in same file Post 302698385 by kmajumder on Monday 10th of September 2012 03:21:52 AM
Old 09-10-2012
Hi Don,

Thank you for your cooperation. Here I am trying to list the answer of your questions.
Quote:
What are the actual commands you execute to convert your log files into the CSV file that you want processed?
Code:
grep -e linkId=1ddoic abc-2012-10-01_000* | cut -f 2,5,14 | awk '{$1=$1;print}' OFS=, > /tmp.output.xls

Quote:
Does abc-2012-10-01_000* match the names of all of the log files (and only those log files) that you want to process?
Yes.
Quote:
When you find none11111 in your CSV file, will the id=xxx field ever match more than one line (not containing none11111 ) in your log files that aren't exact duplicates of other lines?
Yes. it matches more than one line.
Quote:
Am I correct in assuming that the line matching theid=xxx field with the value needed to replacenone11111 in your CSV file, will not be on a line that was selected by a grep on the linkId field you're processing?
Yes. of course. It will never be in same line.
Quote:
Is the field separator you want in your output file"," or", " ?
Only comma. No space.
Quote:
Does the order of lines in your output file matter?
Yes. It matters. It has to be in sorted order of timestamp.
Quote:
What is the purpose of having an additional single-line output file containing a comma separated list of all of your log files? If you need a file containing a list of the log files processed, wouldn't it be better to have the filenames on separate lines instead of separated by commas on a single line?
This is not the additional file. This is the output file that I use as input to generate the final output. After I create the output file after replacing the invalid 'none1111' field I read that file and on top of those values I do some database call and then creates a report.
Quote:
Will thelinkId=zzz field ever appear in any log file that isn't exactly of the same form as the following example line from one of your log files?
Yes. It will appear. To resolve that problem we have to run the second grep like below.
Code:
grep id= 82c6a15ca06b2372c3b3ec2133fc8b14 abc-2012-10-01_000* | grep 'page|clk'

The purpose of running above grep is that. The id
Code:
82c6a15ca06b2372c3b3ec2133fc8b14

can appear in two different event, either 'page' or 'clk'. We can take any one 5th column from that log. And also this log will be found in the same file where 'none11111' was found.
Suppose for linkId=1ddoic we found one invalid 'none11111' value in 5th column of the log file abc-2012-10-01_00002 then in abc-2012-10-01_00002 file only the corresponding id should be found with proper 5th column.

Thank you a lot Don for looking into it.
 

10 More Discussions You Might Find Interesting

1. Programming

Reading and Writing file on LAN

Hi gurus I am not a C programmer but I need to read and write files on a computer on LAN using IP address. Suppose on a computer that has an IP 192.168.0.2 Any help or code example. I did in JAVA using URL, but do not know how to do in ANSI-C. In java: ------- URL url = new... (3 Replies)
Discussion started by: lucky001
3 Replies

2. UNIX for Advanced & Expert Users

Reading a file and writing the file name to a param file.

Hi All, Not sure if this would be in a dummies sectiin or advanced. I'm looking for a script if someone has doen something like this. I have list of files - adc_earnedpoints.20070630.txt adc_earnedpoints.20070707.txt adc_earnedpoints.20070714.txt adc_earnedpoints.20070721.txt... (1 Reply)
Discussion started by: thebeginer
1 Replies

3. UNIX for Dummies Questions & Answers

reading ,writing,appending ,manipulating a file.

Hi my prob statement is to create a new file or to append to the 1tst file the followign chages. File 1: txt file. portfolio No a b c d abc 1 Any Any Any charString cds 2 values values values charString efd 3 can can can charString fdg 4 come come come charString... (4 Replies)
Discussion started by: szchmaltz
4 Replies

4. UNIX for Dummies Questions & Answers

Log File Writing and Reading

Hi all, I have the following shell script code which tries to sftp and writes the log into the log file. TestConnection () { echo 'Connection to ' $DESTUSERNAME@$DESTHOSTNAME $SETDEBUG if ]; then rm $SCRIPT ; fi touch $SCRIPT echo "cd" $REMOTEDIR >> $SCRIPT echo "quit" >>... (10 Replies)
Discussion started by: valluvan
10 Replies

5. Programming

I need help with file reading/writing in C

Hello everybody, I'm trying to code a program which makes the following: It sends an ARP request frame and when it gets the reply, extracts the IP address of source and writes it to a .txt file. This is gonna be done with many hosts (with a for() loop), so, the text file would look like... (2 Replies)
Discussion started by: Zykl0n-B
2 Replies

6. Shell Programming and Scripting

Reading data from DataBase and Writing to a file

Hi All, Please help me in writing data to a file in one row. In database there is a column which contains large data which does not fit in the file in one row. The column contains list of paths. I want to write these paths to a file in one row. Please find the code below writes : ... (2 Replies)
Discussion started by: rajeshorpu
2 Replies

7. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

8. Programming

unexpected values received when writing and reading from file

In the code below i try to write and read from a file, but i get unexpected results, like after writing i cannot open the file, and when reading the file the value entered earlier is not shown bool creat_fragments(int nFragment) { int fd, rand_value; char frag_path, buf; for(int... (8 Replies)
Discussion started by: saman_glorious
8 Replies

9. Shell Programming and Scripting

reading a file extracting information writing to a file

Hi I am trying to extract information out of a file but keep getting grep cant open errors the code is below: #bash #extract orders with blank address details # # obtain the current date # set today to the current date ccyymmdd format today=`date +%c%m%d | cut -c24-31` echo... (8 Replies)
Discussion started by: Bruble
8 Replies

10. Shell Programming and Scripting

Reading and Writing a conf file - Suggestions and improvements?

Hello all As part of my TUI - (line based) Text User Interface, i do have 2 commands to assist working with conf files. Now, for me they work, but since i wrote them, i automaticly use them they way they should be used... you know what i mean. ;) Anyway, they are designed to read 'simple'... (3 Replies)
Discussion started by: sea
3 Replies
All times are GMT -4. The time now is 05:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy