Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Need to merge multiple text files vertically and place comma between fields
Top Forums Shell Programming and Scripting Need to merge multiple text files vertically and place comma between fields Post 302905784 by Don Cragun on Friday 13th of June 2014 08:36:19 PM
Old 06-13-2014
If you don't understand the scripts you're given here; ask questions. I haven't seen any of the volunteers here who were unwilling to explain how their code worked if anyone bothered to ask.

Your description of your input files is a little bit sparse. Making the assumptions:
  1. Some timestamps might not appear in all input files.
  2. The first file processed containing a given timestamp might not contain four fields.
  3. The output should have a minimum of 6 fields.
  4. If the output for a given timestamp doesn't include 5 fields (after adding the date to the start of the line), an empty field should be added and the final field should be 100.
With these assumptions, you could try the following code:
Code:
awk -v dt=$(date +%m/%d/%Y) '
{	# For each line in each input file...
	# For each field after the 1st field on each line...
	for(i = 2; i <= NF; i++) {
		# Add a comma and the contents of that field to the data
		# accumulated for the timestamp in the 1st field of this line.
		d[$1] = d[$1]","$i
		# Increment the number of fields we have found for this
		# timestamp.  If it is the 3rd field for this timestamp, also
		# save it in f3[].
		if(++c[$1] == 3)
			f3[$1] = $i
	}
}
END {	# We have hit EOF on the last input file...
	# For each different timestamp that was in the 1st field of any of the
	# input files...
	for(i in d) {
		# If the number of fields associated with this timestamp was
		# less than three, add empty fields...
		while(c[i]++ < 3)
			d[i] = d[i]","
		# Print the date, the timestamp, the input data fields
		# associated with this timestamp, and 100 - the 3rd data field
		# found for this timestamp with commas as field separators.
		print dt","i d[i]","(100 - f3[i])
	}
}' File?? | sort

If we add an additional input file (named File04) to those you had before:
Code:
17:40:00	1
17:45:00	2	3
17:50:00	4	5	6
17:50:01	300
17:55:00	7	8	9.1112	10

then running the above script today produces the output:
Code:
06/13/2014,17:40:00,1,,,100
06/13/2014,17:40:01,569056,2538672,81.69,436568,1771600,6213560,64,0.00,0,0,130,0.00,0.00,0.00,180,18.31
06/13/2014,17:45:00,2,3,,100
06/13/2014,17:45:01,570420,2537308,81.65,436568,1771600,6213560,64,0.00,0,0,126,0.00,0.00,0.00,190,18.35
06/13/2014,17:50:00,4,5,6,94
06/13/2014,17:50:01,569676,2538052,81.67,436568,1771624,6213560,64,0.00,0,0,129,0.06,0.03,0.01,200,300,18.33
06/13/2014,17:55:00,7,8,9.1112,10,90.8888

Does this do what you want?

Is there anything in this awk script you don't understand?
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Update text files in place (string substitution) ??

The auditors have nailed us for world writeable files.... Apparently in years gone by, quite a number of our kornshell scripts have had: umask 000 put in the script. We have been able to turn off world writeable for existing dirs & files, but as these scripts run, new files keep getting... (1 Reply)
Discussion started by: kornshellmaven
1 Replies

2. Shell Programming and Scripting

How do you delete multiple text from a comma delimited file

I would like to know code that will delete multiple text from a comma delimited file. For example, how would the comma delimited file below delete the word 'PEST' in Perl language (previously an excel file that was converted to a csv and the last column was PEST): 1, 2,43,34, bosx,PEST 1,... (1 Reply)
Discussion started by: dolo21taf
1 Replies

3. Shell Programming and Scripting

Merge text files while combining the multiple header/trailer records into one each.

Situation: Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to... (4 Replies)
Discussion started by: oordonez
4 Replies

4. Shell Programming and Scripting

Place a comma on lines

Is it possible to place a comma in the desired places, like 10spaces after or 15 spaces after, irrespective of the contents??? Ex:File: TEST TEST: vimalthomaswants to place a comma can he do it in the desired places? as per the above file, i need to place a comma after 10th space... (4 Replies)
Discussion started by: vj8436
4 Replies

5. Shell Programming and Scripting

AWK: merge two files and replace some fields

Need some code tweak: awk 'END { for (i=1; i<=n; i++) if (f2]) print f2] } NR == FNR { f2 = $1] = $0 next } $1 in f2 { delete f2 }1' FS=, OFS=, 2.csv 1.csv > 3.csvfile 1.csv have: $1,$2,$3,$4,$5,$6,$7,$8,$9...... file 2.csv have: $1,$2,$3,$4,$5,$6 (2 Replies)
Discussion started by: u10
2 Replies

6. Shell Programming and Scripting

Insert comma in place of column

Hi all, I have a file in which I have to insert commna between entries of 2 column and createa new file separated by commas not a columns if input is FHIT Adenosine Monotungstate Not Available CS Trifluoroacetonyl Coenzyme A Not Available Theo expected output is ... (5 Replies)
Discussion started by: manigrover
5 Replies

7. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

8. Shell Programming and Scripting

How to merge two or more fields from two different files where there is non matching column?

Hi, Please excuse for often requesting queries and making R&D, I am trying to work out a possibility where i have two files field separated by pipe and another file containing only one field where there is no matching columns, Could you please advise how to merge two files. $more... (3 Replies)
Discussion started by: karthikram
3 Replies

9. UNIX for Dummies Questions & Answers

Merge two text files by two fields and mixed output

Hello, I'm back again looking for your precious help- This time I need to merge two text files with matching two fields, output only common records with mixed output. Let's look at the example: FILE1 56153;AAA0708;3;TEST1TEST1; 89014;BBB0708;3;TEST2TEST2; 89014;BBB0708;4;TEST3TEST3; ... (7 Replies)
Discussion started by: emare
7 Replies

10. UNIX for Beginners Questions & Answers

How to extract fields from a CSV i.e comma separated where some of the fields having comma as value?

can anyone help me!!!! How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement like i have to extract different field and assign them into different... (4 Replies)
Discussion started by: J.Jena
4 Replies

LEARN ABOUT CENTOS

git-merge-index

GIT-MERGE-INDEX(1)						    Git Manual							GIT-MERGE-INDEX(1)

NAME

       git-merge-index - Run a merge for files needing merging

SYNOPSIS

       git merge-index [-o] [-q] <merge-program> (-a | [--] <file>*)

DESCRIPTION

       This looks up the <file>(s) in the index and, if there are any merge entries, passes the SHA-1 hash for those files as arguments 1, 2, 3
       (empty argument if no file), and <file> as argument 4. File modes for the three files are passed as arguments 5, 6 and 7.

OPTIONS

       --
	   Do not interpret any more arguments as options.

       -a
	   Run merge against all files in the index that need merging.

       -o
	   Instead of stopping at the first failed merge, do all of them in one shot - continue with merging even when previous merges returned
	   errors, and only return the error code after all the merges.

       -q
	   Do not complain about a failed merge program (a merge program failure usually indicates conflicts during the merge). This is for
	   porcelains which might want to emit custom messages.

       If git merge-index is called with multiple <file>s (or -a) then it processes them in turn only stopping if merge returns a non-zero exit
       code.

       Typically this is run with a script calling Git's imitation of the merge command from the RCS package.

       A sample script called git merge-one-file is included in the distribution.

       ALERT ALERT ALERT! The Git "merge object order" is different from the RCS merge program merge object order. In the above ordering, the
       original is first. But the argument order to the 3-way merge program merge is to have the original in the middle. Don't ask me why.

       Examples:

	   torvalds@ppc970:~/merge-test> git merge-index cat MM
	   This is MM from the original tree.			 # original
	   This is modified MM in the branch A. 		 # merge1
	   This is modified MM in the branch B. 		 # merge2
	   This is modified MM in the branch B. 		 # current contents

       or

	   torvalds@ppc970:~/merge-test> git merge-index cat AA MM
	   cat: : No such file or directory
	   This is added AA in the branch A.
	   This is added AA in the branch B.
	   This is added AA in the branch B.
	   fatal: merge program failed

       where the latter example shows how git merge-index will stop trying to merge once anything has returned an error (i.e., cat returned an
       error for the AA file, because it didn't exist in the original, and thus git merge-index didn't even try to merge the MM thing).

GIT

       Part of the git(1) suite

Git 1.8.3.1							    06/10/2014							GIT-MERGE-INDEX(1)
All times are GMT -4. The time now is 09:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy