Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Adding filename and line number from multiple files to final file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Adding filename and line number from multiple files to final file
# 1  
Old 04-17-2013
Linux Adding filename and line number from multiple files to final file
Hi all,
I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1.
I need following kind of output in a single file:
Filename Line number 2ndcolumn 4thcolumn
Quote:
1 1 23.01 119.00
1 2 56.00
.. ....
1 1000 -09.00 89.00
..
2 1
2 2
.. ....
2 1000
...
20 1
20 2
..
20 1000
I was able to remove first 2 lines from each file (starting with #) and getting only 2nd and 4th column. Here is the code:
Code:
cat *.txt | awk '{print $2, $4} | sed "/#\/d" > out.txt

Please guide me.
Thanks
# 2  
Old 04-17-2013
Quote:
Originally Posted by bioinfo
Hi all,
I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1.
I need following kind of output in a single file:
Filename Line number 2ndcolumn 4thcolumn

I was able to remove first 2 lines from each file (starting with #) and getting only 2nd and 4th column. Here is the code:
Code:
cat *.txt | awk '{print $2, $4} | sed "/#\/d" > out.txt

Please guide me.
Thanks
If you don't show us sample output that matches what you say you want to have done, we can't figure out what you really want. You say you only want the 2nd and 4th columns, but your sample output file starts with:
Code:
1 1 23.01 119.00
1 2 56.00
.. ....
1 1000 -09.00 89.00

each of which has three or four columns.

Then there is the script you said you use to get what you want, but the sed command in this pipeline (sed "/#\/d" is not a valid command and will generate a diagnostic something like:
Code:
sed: 1: "/#\/d": unterminated regular expression

Please:
  1. Use CODE tags (not QUOTE tags) when showing us input and output files.
  2. Show us input files. And,
  3. Show us output that matches your description of your desired output.
This User Gave Thanks to Don Cragun For This Post:
bioinfo
# 3  
Old 04-17-2013
I used following, I think previous was my typo error. Sorry for that.
Code:
cat *.txt | awk '{print $2, $4} | sed "/#ainst\|#Time/d" > out.txt

I am getting only two columns from my code but I want to have four (1st column as digit from file name and 2nd column as line number), which I showed previously.
Format of my input files:
Code:
#ainst
#Time                   tem                 pre                    apot                inst                kin
      10.000         1.95221         0.0000079230          919.62689       149.40629      3858.88908
      20.000         1.22713         0.0000000379           27.40189      -110.08021      2303.82262
....
    10000.000         0.63837        -0.0000007208         -256.43974      -242.08325      1590.95448

I wish to have following output with 1st column as digit from file name and 2nd column as line number:
Code:
001   1      1.95221          919.62689       
001   2      1.22713           27.40189      
....
001   1000   0.63837         -256.43974      
002   1      4.98221           19.62689       
002   2      10.52713         127.40189      
....
002    1000   0.43837         -956.43974   
.....
020   1      8.98981           56.62689       
020   2      10.52713          29.40189      
.... 
020    1000   9.43837         -56.43974

Thanks
# 4  
Old 04-17-2013
Quote:
Originally Posted by bioinfo
I used following, I think previous was my typo error. Sorry for that.
Code:
cat *.txt | awk '{print $2, $4} | sed "/#ainst\|#Time/d" > out.txt

I am getting only two columns from my code but I want to have four (1st column as digit from file name and 2nd column as line number), which I showed previously.
Format of my input files:
Code:
#ainst
#Time                   tem                 pre                    apot                inst                kin
      10.000         1.95221         0.0000079230          919.62689       149.40629      3858.88908
      20.000         1.22713         0.0000000379           27.40189      -110.08021      2303.82262
....
    10000.000         0.63837        -0.0000007208         -256.43974      -242.08325      1590.95448

I wish to have following output with 1st column as digit from file name and 2nd column as line number:
Code:
001   1      1.95221          919.62689       
001   2      1.22713           27.40189      
....
001   1000   0.63837         -256.43974      
002   1      4.98221           19.62689       
002   2      10.52713         127.40189      
....
002    1000   0.43837         -956.43974   
.....
020   1      8.98981           56.62689       
020   2      10.52713          29.40189      
.... 
020    1000   9.43837         -56.43974

Thanks
You also have mismatched single quotes in the awk command in your pipeline. Please be more careful in the future when you post samples of code.

It is hard to tell with the minimal sample input provided, but I think the following awk script does what you want:
Code:
awk '
FNR==1{ fn = substr(FILENAME, 5, 3)
        n = 0
}
/^#/{   n++
        next
}
{       printf("%s\t%d\t%s\t%s\n", fn, FNR - n, $2, $4)
}' *.txt

As always, if you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of awk.
This User Gave Thanks to Don Cragun For This Post:
bioinfo
# 5  
Old 04-17-2013
Thanks a lot. Smilie
I would highly appreciate if you explain it.
Thanks again.
# 6  
Old 04-17-2013
Does this help?
Code:
awk '
FNR==1{ # This is the first line in a new file...
        fn = substr(FILENAME, 5, 3) # Save 3 characters from this filename
        n = 0   # Clear number of comments found in this file
}
/^#/{   n++     # Increment number of comments found in this file
        next    # Do not do any other processing on comment lines
}
{       # Print saved characters from filename, number of non-comment lines,
        # and fields 2 and 4 from the current line
        printf("%s\t%d\t%s\t%s\n", fn, FNR - n, $2, $4)
}' *.txt # Process all files ending with ".txt" in the current directory

This User Gave Thanks to Don Cragun For This Post:
bioinfo
# 7  
Old 04-17-2013
Thanks Smilie
This script is awesome but somewhat tough for me as I am a beginner. Is there any possibililty to add something easy to my previous code (below) to do the same thing.
Code:
cat *.txt | awk '{print $2, $4}' | sed "/#ainst\|#Time/d" > out.txt
or
cat *.txt | awk '{NR >=3 && NR <= 1002 {print $2, $4}' > out.txt

Thanks
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Insert the line number from text file to filename output

Hi everyone :) I have a file "words.txt" containing hundreds of lines of text. Each line contains a slogan. Using the code below i am able to generate an image with the slogan text from each line. The image filename is saved matching the last word on each line. Example: Line 1: We do... (2 Replies)
Discussion started by: martinsmith
2 Replies

2. Shell Programming and Scripting

Adding user name to file, and then displaying new line number

Hi all - I'm completely stumped by a script I'm working on... The short version is I have a file called 'lookup' and in it are hundreds of names (first and last). I have a script that basically allows the user to enter a name, and what I need to have happen is something like this: Record... (8 Replies)
Discussion started by: sabster
8 Replies

3. Shell Programming and Scripting

adding line number to *end* of records in file

Given a file like this: abc def ghi I need to get to somestandardtext abc1 morestandardtext somestandardtext def2 morestandardtext somestandardtext ghi3 morestandardtext Notice that in addition to the standard text there is the line number added in as well. What I conceived is... (4 Replies)
Discussion started by: edstevens
4 Replies

4. Shell Programming and Scripting

editing line in text file adding number to value in file

I have a text file that has data like: Data "12345#22" Fred ID 12345 Age 45 Wilma Dino Data "123#22" Tarzan ID 123 Age 33 Jane I need to figure out a way of adding 1,000,000 to the specific lines (always same format) in the file, so it becomes: Data "1012345#22" Fred ID... (16 Replies)
Discussion started by: say170
16 Replies

5. Shell Programming and Scripting

insert filename into each line of multiple files

I need to insert <filename + comma> into each line of multiple files. Any idea how to script that? Regards, Manu (5 Replies)
Discussion started by: linux.yahoo
5 Replies

6. Shell Programming and Scripting

Adding filename to each line of the file

Hi, I am a relative new bee in scripting. I need to develop a script such that the code would iterate through each file in a source directory and append every line of the file with '|' and the corresponding file filename. eg INPUT file IF927_1.dat - H|abc... (4 Replies)
Discussion started by: scripting_newbe
4 Replies

7. Shell Programming and Scripting

Adding text in final line

Dear Friends, I have a flat file where last line of it has word D$mhtt I want to add a space and back slash after it. Also wanna add -S "J" in the last line. Following example will make it clear. I have this in the last line of file D$mhtt I want D$mhtt \ -S "J" Please... (5 Replies)
Discussion started by: anushree.a
5 Replies

8. Shell Programming and Scripting

Adding a columnfrom a specifit line number to a specific line number

Hi, I have a huge file & I want to add a specific text in column. But I want to add this text from a specific line number to a specific line number & another text in to another range of line numbers. To be more specific: lets say my file has 1000 lines & 4 Columns. I want to add text "Hello"... (2 Replies)
Discussion started by: Ezy
2 Replies

9. Shell Programming and Scripting

Adding multiple line at the end of the file

I have 2 files which contains the following lines file1.txt line4 line5 line6 file2.txt line1 line2 line3 When i execute a script , I want my file2.txt will looks like this: line1 line2 line3 line4 line5 (2 Replies)
Discussion started by: kaibiganmi
2 Replies

10. Shell Programming and Scripting

Grabing Date from filename and adding to the end of each line in the file.

Hi, I have 24 .dat files something like below. The file name starts with “abc” followed by two digit month and two digit year. Is there a way to grab the month and year from each filename and append it to the end of each line. Once this is done I want to combine all the files into file... (1 Reply)
Discussion started by: rkumar28
1 Replies
Login or Register to Ask a Question