Sponsored Content
Top Forums Shell Programming and Scripting Need to merge multiple text files vertically and place comma between fields Post 302905767 by prvnrk on Friday 13th of June 2014 04:41:28 PM
Old 06-13-2014
Need to merge multiple text files vertically and place comma between fields

Hello expert friends,

I'm writing a script to capture stats using sar and stuck up at report generation.

I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after every field. For ex:

Code:
# cat File01

17:40:01       569056   2538672     81.69    436568   1771600   6213560        64      0.00         0
17:45:01       570420   2537308     81.65    436568   1771600   6213560        64      0.00         0
17:50:01       569676   2538052     81.67    436568   1771624   6213560        64      0.00         0

Code:
#cat File02

17:40:01            0       130      0.00      0.00      0.00
17:45:01            0       126      0.00      0.00      0.00
17:50:01            0       129      0.06      0.03      0.01

Code:
#cat File03
17:40:01            180
17:45:01            190
17:50:01            200

Then Output file should be:
Code:
17:40:01,       569056,   2538672,     81.69,    436568,   1771600,   6213560,        64,      0.00,         0,   0,       130,      0.00,      0.00,      0.00,   180
17:45:01,       570420,   2537308,     81.65,    436568,   1771600,   6213560,        64,      0.00,         0,   0,       126,      0.00,      0.00,      0.00,   190
17:50:01,       569676,   2538052,     81.67,    436568,   1771624,   6213560,        64,      0.00,         0,   0,       129,      0.06,      0.03,      0.01,   200

Many thanks,

Last edited by prvnrk; 06-13-2014 at 05:51 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Update text files in place (string substitution) ??

The auditors have nailed us for world writeable files.... Apparently in years gone by, quite a number of our kornshell scripts have had: umask 000 put in the script. We have been able to turn off world writeable for existing dirs & files, but as these scripts run, new files keep getting... (1 Reply)
Discussion started by: kornshellmaven
1 Replies

2. Shell Programming and Scripting

How do you delete multiple text from a comma delimited file

I would like to know code that will delete multiple text from a comma delimited file. For example, how would the comma delimited file below delete the word 'PEST' in Perl language (previously an excel file that was converted to a csv and the last column was PEST): 1, 2,43,34, bosx,PEST 1,... (1 Reply)
Discussion started by: dolo21taf
1 Replies

3. Shell Programming and Scripting

Merge text files while combining the multiple header/trailer records into one each.

Situation: Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to... (4 Replies)
Discussion started by: oordonez
4 Replies

4. Shell Programming and Scripting

Place a comma on lines

Is it possible to place a comma in the desired places, like 10spaces after or 15 spaces after, irrespective of the contents??? Ex:File: TEST TEST: vimalthomaswants to place a comma can he do it in the desired places? as per the above file, i need to place a comma after 10th space... (4 Replies)
Discussion started by: vj8436
4 Replies

5. Shell Programming and Scripting

AWK: merge two files and replace some fields

Need some code tweak: awk 'END { for (i=1; i<=n; i++) if (f2]) print f2] } NR == FNR { f2 = $1] = $0 next } $1 in f2 { delete f2 }1' FS=, OFS=, 2.csv 1.csv > 3.csvfile 1.csv have: $1,$2,$3,$4,$5,$6,$7,$8,$9...... file 2.csv have: $1,$2,$3,$4,$5,$6 (2 Replies)
Discussion started by: u10
2 Replies

6. Shell Programming and Scripting

Insert comma in place of column

Hi all, I have a file in which I have to insert commna between entries of 2 column and createa new file separated by commas not a columns if input is FHIT Adenosine Monotungstate Not Available CS Trifluoroacetonyl Coenzyme A Not Available Theo expected output is ... (5 Replies)
Discussion started by: manigrover
5 Replies

7. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

8. Shell Programming and Scripting

How to merge two or more fields from two different files where there is non matching column?

Hi, Please excuse for often requesting queries and making R&D, I am trying to work out a possibility where i have two files field separated by pipe and another file containing only one field where there is no matching columns, Could you please advise how to merge two files. $more... (3 Replies)
Discussion started by: karthikram
3 Replies

9. UNIX for Dummies Questions & Answers

Merge two text files by two fields and mixed output

Hello, I'm back again looking for your precious help- This time I need to merge two text files with matching two fields, output only common records with mixed output. Let's look at the example: FILE1 56153;AAA0708;3;TEST1TEST1; 89014;BBB0708;3;TEST2TEST2; 89014;BBB0708;4;TEST3TEST3; ... (7 Replies)
Discussion started by: emare
7 Replies

10. UNIX for Beginners Questions & Answers

How to extract fields from a CSV i.e comma separated where some of the fields having comma as value?

can anyone help me!!!! How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement like i have to extract different field and assign them into different... (4 Replies)
Discussion started by: J.Jena
4 Replies
TABMERGE(1p)						User Contributed Perl Documentation					      TABMERGE(1p)

NAME
tabmerge - unify delimited files on common fields SYNOPSIS
tabmerge [action] [options] file1 file2 [...] Actions: --min Take only fields present in all files [DEFAULT] --max Take all fields present -f|--fields=f1[,f2] Take only the fields mentioned in the comma-separated list Options: -l|--list List available fields --fs=x Use "x" as the field separator (default is tab " ") --rs=x Use "x" as the record separator (default is newline " ") -s|--sort=f1[,f2] Sort data ASCII-betically on field(s) --stdout Print data in original delimited format (i.e., not in a table format) --help Show brief help and quit --man Show full documentation DESCRIPTION
This program merges the fields -- not the rows -- of delimited text files. That is, if several files are almost but not quite entirely unlike each other in their structure (in their field names, numbers or orders), this script allows you to easily unify the files into one file with all the same fields. The output can be based on fields as determined by the three "action" flags. For the following examples, consider three files that contain the following fields: +------------+---------------------------------+ | File | Fields | +------------+---------------------------------+ | merge1.tab | name, type, position | | merge2.tab | name, type, position, lod_score | | merge3.tab | name, position | +------------+---------------------------------+ To list all available fields in the files and the number of times they are present: $ tabmerge --list merge* +-----------+-------------------+ | Field | No. Times Present | +-----------+-------------------+ | lod_score | 1 | | name | 3 | | position | 3 | | type | 2 | +-----------+-------------------+ To merge the files on the minimum overlapping fields: $ tabmerge merge* +----------+----------+ | name | position | +----------+----------+ | RM104 | 2.30 | | RM105 | 4.5 | | TX5509 | 10.4 | | UU189 | 19.0 | | Xpsm122 | 3.3 | | Xpsr9556 | 4.5 | | DRTL | 2.30 | | ALTX | 4.5 | | DWRF | 10.4 | +----------+----------+ To merge the files and include all the fields: $ tabmerge --max merge* +-----------+----------+----------+--------+ | lod_score | name | position | type | +-----------+----------+----------+--------+ | | RM104 | 2.30 | RFLP | | | RM105 | 4.5 | RFLP | | | TX5509 | 10.4 | AFLP | | 2.4 | UU189 | 19.0 | SSR | | 1.2 | Xpsm122 | 3.3 | Marker | | 1.2 | Xpsr9556 | 4.5 | Marker | | | DRTL | 2.30 | | | | ALTX | 4.5 | | | | DWRF | 10.4 | | +-----------+----------+----------+--------+ To merge and extract just the "name" and "type" fields: $ tabmerge -f name,type merge* +----------+--------+ | name | type | +----------+--------+ | RM104 | RFLP | | RM105 | RFLP | | TX5509 | AFLP | | UU189 | SSR | | Xpsm122 | Marker | | Xpsr9556 | Marker | | DRTL | | | ALTX | | | DWRF | | +----------+--------+ To merge the files on just the "name" and "lod_score" fields and sort on the name: $ tabmerge -f name,lod_score -s name merge* +----------+-----------+ | name | lod_score | +----------+-----------+ | ALTX | | | DRTL | | | DWRF | | | RM104 | | | RM105 | | | TX5509 | | | UU189 | 2.4 | | Xpsm122 | 1.2 | | Xpsr9556 | 1.2 | +----------+-----------+ To do the same but mimic the original tab-delimited input: $ tabmerge -f name,lod_score -s name --stdout merge* name lod_score ALTX DRTL DWRF RM104 RM105 TX5509 UU189 2.4 Xpsm122 1.2 Xpsr9556 1.2 Why would you want to do this? Suppose you have several delimited text files with nearly the same structure and want to create just one file from them, but the fields may be in a different order in each file and/or some files may contain more or fewer fields than others. (As far-fetched as it may seem, it happens to the author more than he'd like.) SEE ALSO
o Text::RecordParser o Text::TabularDisplay AUTHOR
Ken Youens-Clark <kclark@cpan.org>. LICENSE AND COPYRIGHT
Copyright (C) 2006-10 Ken Youens-Clark. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. perl v5.10.1 2010-07-26 TABMERGE(1p)
All times are GMT -4. The time now is 10:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy