Visit The New, Modern Unix Linux Community


Concatenate select lines from multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Concatenate select lines from multiple files
# 1  
Concatenate select lines from multiple files

I have about 6000 files of the following format (three simplified examples shown; actual files have variable numbers of columns, but the same number of lines). I would like to concatenate the ID (*Loc*) and data lines, but not the others, as shown below. The result would be one large file (or several slightly less large files).

fileA
Code:
header line A
A_Loc1, A_Loc2, A_Loc3, A_Loc4
Pop
gt , 02 04 01 01 
gt , 02 04 01 01 
Pop
gt , 03 04 01 01 
gt , 03 04 01 01

fileB
Code:
header line B
B_Loc1, B_Loc2
Pop
gt , 03 03
gt , 02 03
Pop
gt , 02 03 
gt , 02 03

fileC
Code:
header line C
C_Loc1, C_Loc2, C_Loc3
Pop
gt , 03 03 02
gt , 03 03 02
Pop
gt , 03 03 02
gt , 03 03 02

Output:
Code:
header line A
A_Loc1, A_Loc2, A_Loc3, A_Loc4, B_Loc1, B_Loc2, C_Loc1, C_Loc2, C_Loc3
Pop
gt , 02 04 01 01 03 03 03 03 02
gt , 02 04 01 01 02 03 03 03 02
Pop
gt , 03 04 01 01 02 03 03 03 02
gt , 03 04 01 01 02 03 03 03 02

I know paste can concatenate lines, but I'm not sure how to only paste selective lines, not how to add a comma in the ID lines. Would be grateful if someone could show me a way.
# 2  
If "ls file*" lists only those files in the directory, then
Code:
awk 'NR == FNR {a[FNR] = $0; next} NF > 1 {if($0 !~ /^gt/) {$2 = $0}; a[FNR] = (a[FNR] $2)} END {for(i = 1; i <= FNR; i++) {print a[i]}}' FS=',' file*

# 3  
This is a general join on field#1:
Code:
awk '($1==k[FNR]) {$1=""} {a[FNR]=a[FNR] FS $0} (NR==FNR) {k[FNR]=$1} END {for (i=1;i<=FNR; i++) print a[i]}' file*


Last edited by MadeInGermany; 04-13-2014 at 12:15 PM..
# 4  
If there are too many files and you get : "Too many arguments", you could try:

Code:
for f in file*
do
   [ -f "$f" ] || continue
   cat "$f"
done | 

awk -F, '
  /header/ {
    n=c
    c=0
  } 
  !(++c in A) {
    A[c]=$0
    next
  }
  /Loc/ {
    A[c]=A[c]", "$0
  }
  $1~/^gt/ {
    A[c]=A[c] $2
  }
  END {
    for(i=1; i<=n; i++)print A[i]
  }
'


Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #490
Difficulty: Medium
Java source code is compiled into Java bytecode which is actual machine code.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell Scripting - Select multiple files from numbered list

I am trying to have the user select two files from a numbered list which will eventually be turned into a variable then combined. This is probably something simple and stupid that I am doing. clear echo "Please Select the Show interface status file" select FILE1 in *; echo "Please Select the... (3 Replies)
Discussion started by: dis0wned
3 Replies

2. UNIX for Beginners Questions & Answers

Concatenate column values when header is Matching from multiple files

there can be n number of columns but the number of columns and header name will remain same in all 3 files. Files are tab Delimited. a.txt Name 9/1 9/2 X 1 7 y 2 8 z 3 9 a 4 10 b 5 11 c 6 12 b.xt Name 9/1 9/2 X 13 19 y 14 20 z 15 21 a 16 22 b 17 23 c 18 24 c.txt Name 9/1 9/2... (14 Replies)
Discussion started by: Nina2910
14 Replies

3. Shell Programming and Scripting

Removing carriage returns from multiple lines in multiple files of different number of columns

Hello Gurus, I have a multiple pipe separated files which have records going over multiple Lines. End of line separator is \n and records going over multiple lines have <CR> as separator. below is example from one file. 1|ABC DEF|100|10 2|PQ RS T|200|20 3| UVWXYZ|300|30 4| GHIJKL|400|40... (7 Replies)
Discussion started by: dJHa
7 Replies

4. Shell Programming and Scripting

Select multiple column from multiple files

Hi Friends, $ cat test1.txt emeka:1438 shelley:1439 dmeyer:1440 kurtarn:1441 abdul:1442 $ cat test2.txt 1:a 2:b 3:c 4:d $ cat test3.txt cat:dog:bat man:hot:cold (5 Replies)
Discussion started by: Jewel
5 Replies

5. Shell Programming and Scripting

[Solved] Script to concatenate 2 files with the same number of lines

Hi everyone, I have two files, namely: file1: file1Col1Row1;file1Col2Row1;file1Col3Row1 file1Col1Row2;file1Col2Row2;file1Col3Row2 file1Col1Row3;file1Col2Row3;file1Col3Row3file2: file2Col1Row1;file2Col2Row1;file2Col3Row1 file2Col1Row2;file2Col2Row2;file2Col3Row2... (0 Replies)
Discussion started by: gacanepa
0 Replies

6. Shell Programming and Scripting

Concatenate columns from multiple files

Hi all, I want the 2nd column of every file in the directory in a single file with the file name as column header. $cat file1.txt a b c d e f $cat file2.txt f g h g h j $cat file3.txt a b d f g h (2 Replies)
Discussion started by: newbie83
2 Replies

7. Shell Programming and Scripting

want to concatenate multiple files based on the rest of ls -lrt

uadm@4132> ls -lrt -rw------- 1 uadm uadm 3811819 Jun 6 04:08 data_log-2010.05.30-10:04:08.txt -rw------- 1 uadm uadm 716246 Jun 13 01:38 data_log-2010.06.06-10:04:08.txt -rw------- 1 uadm uadm 996 Jun 13 04:00 data_log-2010.06.06-10:04:22.txt -rw------- 1 uadm uadm 7471 Jun 20 02:03... (5 Replies)
Discussion started by: mail2sant
5 Replies

8. Shell Programming and Scripting

Concatenate multiple lines based.

Hello, I have been searching the forum for concatenation based on condition. I have been close enough but not got th exact one. infile: -----DB_Name ABC (X, Y,Z). DB_Name DEF (T). DB_Name GHI (U ,V,W). Desired Output file should be: ---------------------------DB_Name ABC... (8 Replies)
Discussion started by: indrajit_u
8 Replies

9. Shell Programming and Scripting

Need script to select multiple files from archive directory based on the date range

hi all, here is the description to my problem. input parameters: $date1 & $date2 based on the range i need to select the archived files from the archived directory and moved them in to working directory. can u please help me in writing the code to select the multiple files based on the... (3 Replies)
Discussion started by: bbc17484
3 Replies

10. Shell Programming and Scripting

Find multiple patterns on multiple lines and concatenate output

I'm trying to parse COBOL code to combine variables into one string. I have two variable names that get literals moved into them and I'd like to use sed, awk, or similar to find these lines and combine the variables into the final component. These variable names are always VAR1 and VAR2. For... (8 Replies)
Discussion started by: wilg0005
8 Replies

Featured Tech Videos