Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Duplicate specifc lines in csv Post 302990659 by iz_82 on Monday 30th of January 2017 10:50:56 AM
Old 01-30-2017
Duplicate specifc lines in csv

Hi,

I want to duplicate lines of a csv file with header if a column has multiple values. The csv uses semicolon as separator while multiple values are separated with comma. Only the type3 column can have multiple values.

input:
Code:
type1;type2;type3
a1;b1;x,y
a2;b2;z

output:
Code:
a1;b1;x
a1;b1;y
a2;b2;z

Thanks,
Z

Last edited by vbe; 01-30-2017 at 12:00 PM.. Reason: please use code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

obtain duplicate keys in csv file

Hi, having two csv files, both sorted, by key (column1), f1 containing duplicate keys and f2 containing no duplicate keys, how can I obtain all rows from f1 with the keys listed in file2? Example: f1 is: k1,gsj01fd k2,vi982cj k2,1fjk01e k3,81kjfds k4,sd9dasi f2 is: k2 k3 and I... (3 Replies)
Discussion started by: oscarmon
3 Replies

2. Shell Programming and Scripting

Duplicate rows in CSV files based on values

I want to duplicate a row if found two or more values in a particular column for corresponding row which is delimitted by comma. Input abc,line one,value1 abc,line two, value1, value2 abc,line three,value1 needs to converted to abc,line one,value1 abc,line two, value1 abc,line... (8 Replies)
Discussion started by: Incrediblian
8 Replies

3. Shell Programming and Scripting

Duplicate rows in CSV files based on values

I am new to this forum and this is my first post. I am looking at an old post with exactly the same name. Can not paste URL because I do not have 5 posts My requirement is exactly opposite. I want to get rid of duplicate rows and try to append the values of columns in those rows ... (10 Replies)
Discussion started by: vbhonde11
10 Replies

4. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

5. Shell Programming and Scripting

CSV to SQL insert: Awk for strings with multiple lines in csv

Hi Fellows, I have been struggling to fix an issue in csv records to compose sql statements and have been really losing sleep over it. Here is the problem: I have csv files in the following pipe-delimited format: Column1|Column2|Column3|Column4|NEWLINE Address Type|some descriptive... (4 Replies)
Discussion started by: khayal
4 Replies

6. Shell Programming and Scripting

Using AWK to match CSV files with duplicate patterns

Dear awk users, I am trying to use awk to match records across two moderately large CSV files. File1 is a pattern file with 173,200 lines, many of which are repeated. The order in which these lines are displayed is important, and I would like to preserve it. File2 is a data file with 456,000... (3 Replies)
Discussion started by: isuewing
3 Replies

7. Shell Programming and Scripting

removing duplicate records comparing 2 csv files

Hi All, I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need. File1.csv: RAJAK,ACTIVE,1 VIJAY,ACTIVE,2 TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies

8. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

9. UNIX for Beginners Questions & Answers

Merging specifc lines of three files into one?

Recall that nc-county-pop.dat has 100 lines, and each line corresponds to a county. The files girls.dat and boys.dat each has 50 lines. Assume that the girls come from the first 50 counties and each county has only one girl. That is the first girl comes from the first county, the second girl comes... (1 Reply)
Discussion started by: novicep11
1 Replies

10. UNIX for Beginners Questions & Answers

Get duplicate rows from a csv file

How can i get the duplicates rows from a file using unix, for example i have data like a,1 b,2 c,3 d,4 a,1 c,3 e,5 i want output to be like a,1 c,3 (4 Replies)
Discussion started by: ggupta
4 Replies
COLUMN(1)							   User Commands							 COLUMN(1)

NAME
column - columnate lists SYNOPSIS
column [options] file... DESCRIPTION
The column utility formats its input into multiple columns. Rows are filled before columns. Input is taken from file or, by default, from standard input. Empty lines are ignored. OPTIONS
-c, --columns width Output is formatted to a width specified as number of characters. -t, --table Determine the number of columns the input contains and create a table. Columns are delimited with whitespace, by default, or with the characters supplied using the separator. Table output is useful for pretty-printing. -s, --separator separators Specify possible table delimiters (default is whitespace). -o, --output-separator separators Specify table output delimiter (default is two whitespaces). -x, --fillrows Fill columns before filling rows. -h, --help Print help and exit. ENVIRONMENT
The environment variable COLUMNS is used to determine the size of the screen if no other information is available. EXAMPLES
sed 's/#.*//' /etc/fstab | column -t BUGS
The util-linux version 2.23 changed -s option to be non-greedy, for example: $ printf "a:b:c 1::3 " | column -t -s ':' old output: a b c 1 3 new output (since util-linux 2.23) a b c 1 3 SEE ALSO
colrm(1), ls(1), paste(1), sort(1) HISTORY
The column command appeared in 4.3BSD-Reno. AVAILABILITY
The column command is part of the util-linux package and is available from ftp://ftp.kernel.org/pub/linux/utils/util-linux/. util-linux October 2010 COLUMN(1)
All times are GMT -4. The time now is 05:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy