Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Merging all (48) CSV files from a directory
Top Forums Shell Programming and Scripting Merging all (48) CSV files from a directory Post 302532326 by alister on Monday 20th of June 2011 03:45:03 PM
Old 06-20-2011
Quote:
Originally Posted by Corona688
Code:
                # Print out every file, ignoring first two lines
                tail "$FILE" -n +3

I believe the OP is interested in merging like lines in files in a manner congruent to a paste(1) or join(1), and not cat(1). My understanding is that none of the lines in any file need to be skipped and that the first comma-delimited field is a key (although since all files use the same order, a solution may not need to inspect it) which needs to be removed from all but the first file's line to avoid redundant fields in each line in the final output.

Regards,
Alister

---------- Post updated at 03:45 PM ---------- Previous update was at 03:38 PM ----------

Perhaps something along these lines:
Code:
paste *.csv | sed 's/<tab>[^,]*//g'

Where <tab> represents a literal tab character and I have assumed that there are no tabs in your source files. If there are, you can choose a suitable delimiter by informing paste with the -d option and using it instead of a tab in the sed command.

Regards,
Alister
This User Gave Thanks to alister For This Post:
RissaR
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging files belonging to same date in a directory

Hi, I want to merge files which belong to the same date in a directory. For ex: The below two files are modified on 24 Apr 2007. SystemOut.log SystemOut_07.04.24_19.46.40.log The merged file should be of the format SystemOut.log.2007.04.24 (1 Reply)
Discussion started by: rkaarthikeyan4u
1 Replies

2. UNIX for Dummies Questions & Answers

Merging 2 .CSV files in Unix

I need a little help as I am a complete novice at scripting in unix. However, i am posed with an issue...:eek: i have two csv files in the following format@ FILE1.CSV: HEADER HEADER Header , , HEADER 001X ,,200 002X ,,300 003X ... (6 Replies)
Discussion started by: chachabronson
6 Replies

3. Shell Programming and Scripting

Merging files to create CSV file

Hi, I have different files of the same type, as: Time: 100 snr: 88 perf: 10 other: 222 Each of these files are created periodically. What I need to do is to merge all of them into one but having the following form: (2 Replies)
Discussion started by: Ravendark
2 Replies

4. UNIX for Dummies Questions & Answers

Merging two CSV files by 3 primary keys (columns)

Hi there! I have the following problem: I have a set of files called rates_op_yyyyddmm with the format below (which corresponds to the file rates_op_20090130) 30-JAN-2009,ED,FEB09,C,96.375,,,0,,,,,,2.375,,,,,, 30-JAN-2009,ED,FEB09,C,96.5,,,0,,,,,,2.25,,,,,,... (2 Replies)
Discussion started by: Pep Puigvert
2 Replies

5. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

6. Shell Programming and Scripting

Merging Very large CSV files in Unix

Hi, I have two very large CSV files, which I want to merge (equi-join) based on a key (column). One of the file (say F1) would have ~30 MM records and 700 columns. The other file (~f2) would have same # of records and lesser columns (say 50). I want to create an output file joining on a... (3 Replies)
Discussion started by: student_007
3 Replies

7. Shell Programming and Scripting

Help with merging two CSV files

Hi, I have following 2 CSV files file1.txt A1,B1,C1,D1,E1 A2,B2,C2,D2,E2 A3,B3,C3,D3,E3 .... file2.txt A1,B1,P1,Q1,R1,S1,T1,U1 A1,B1,P2,Q2,R2,S2,T2,U2 A1,B1,P3,Q3,R3,S3,T3,U3 A2,B2,X1,Y1,Z1,I1,J1,K1 A2,B2,X2,Y2,Z2,I2,J2,K2 A2,B2,X3,Y3,Z3,I3,J3,K3 A2,B2,X4,Y4,Z4,I4,J4,K4... (2 Replies)
Discussion started by: learnoutmore99
2 Replies

8. Shell Programming and Scripting

Merging fields in CSV

Hi experts, I have a csv file which has one field (ID) repeated multiple times with corresponding other field values. I need to convert this file in a format where for a ID all other values has to be present in single field. For Eg : Here in below file ID 1 is repeated 3 times with different... (7 Replies)
Discussion started by: bharathbangalor
7 Replies

9. UNIX for Beginners Questions & Answers

How to zip csv files having specific pattern in a directory using UNIX shell script?

I have files in a Linux directory . Some of the file is listed below -rw-rw-r--. 1 roots roots 0 Dec 23 02:17 zzz_123_00000_A_1.csv -rw-rw-r--. 1 roots roots 0 Dec 23 02:18 zzz_121_00000_A_2.csv -rw-rw-r--. 1 roots roots 0 Dec 23 02:18 zzz_124_00000_A_3.csv drwxrwxr-x. 2 roots roots 6 Dec 23... (4 Replies)
Discussion started by: Balraj
4 Replies

10. UNIX for Beginners Questions & Answers

Check for null values in a columns. I have dozen of CSV files in a directory.

Hi Folks, I'm trying to write a simple file sanity check script. I have a directory with dozen CSV files containing id,edname,firstname,lastname,suffix,email. I like to write a awk script to check if first field contain a number and is not empty. and fields number 3,4 & 6 are not empty and... (3 Replies)
Discussion started by: dc34684
3 Replies

LEARN ABOUT CENTOS

processcsv.py

PROCESSCSV.PY(1)					      Virtualization Support						  PROCESSCSV.PY(1)

NAME

       processcsv.py - process virt-top CSV files

SUMMARY

	virt-top --csv data.csv
	processcsv.py < data.csv

DESCRIPTION

       virt-top is a top(1)-like utility for showing stats of virtualized domains.

       processcsv.py is a simple Python script that post-processes the output of "virt-top --csv".

       It is used like this:

	virt-top --csv data.csv
	processcsv.py < data.csv

       The second command will overwrite the following files in the current directory:

       "global.csv"
	   This contains the global (host) statistics columns from the CSV file.

       "domainNN.csv" (multiple files)
	   For each libvirt domain ID NN, a file is created containing the per-domain statistics from the CSV file.

SEE ALSO

       virt-top(1)

AUTHORS

       Richard W.M. Jones <rjones @ redhat . com>

COPYRIGHT

       (C) Copyright 2007-2012 Red Hat Inc., Richard W.M. Jones http://libvirt.org/

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
       Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

virt-top-1.0.8							    2013-12-29							  PROCESSCSV.PY(1)
All times are GMT -4. The time now is 11:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy