Sponsored Content
Top Forums Shell Programming and Scripting Egrep or awk for removing values within CSV file? Post 302936478 by macurdy on Wednesday 25th of February 2015 09:56:23 PM
Old 02-25-2015
Egrep or awk for removing values within CSV file?

Hello,

I have a large CSV file that contains values all on the same column, and in one very long row (e.g. no line breaks till end, with all data values separated by a comma).

The file has two types of data for the values. One begins with the letters rs and some numbers. The other begins with the letter i and some numbers. An example is below (id's are genome identifiers).

Code:
rs28931576,rs11542040,rs28931577,rs429358,i6007484,i6007510,rs28931578,i6007500,i6007489,i5000217,i6007504,i6007493,rs769455,i6007507,i6007497,i6007512,i6007495,i6007485,i6007492,i5000216,i5000205,rs7412

My Unix command line knowledge was enough to use the cat and cut commands to get the above data to this point.

I can't seem to figure out how to remove all of the values that begin with the letter i. I've tried some awk and egrep commands, but don't have the mastery yet to get this figured out.

I also need a way to get rid of duplicate commas after the i values are removed.

Right now, I'm using Find-Replace with TextEdit on mac to do these steps, however I'd love to be able to script this.

Any help is much appreciated!

Last edited by Don Cragun; 02-26-2015 at 01:29 AM.. Reason: Add CODE tags.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to compare two csv files values and write into another csv file

Hi all, Am new to scripting. So i just need your ideas to help me out. Here goes my requirement. I have two csv files 1.csv 2.csv abc,1.24 abc,1 def,2.13 def,1 I need to compare the first column of 1.csv with 2.csv and if matches then need to compare... (2 Replies)
Discussion started by: chinnahyd
2 Replies

2. UNIX for Dummies Questions & Answers

Help removing strings from one file that match any of the values in a second file.

Hello, I have a file that lists a few hundred values. Example: abca abcb abcc abcd I have a 2nd file with a few thousand lines. I need to remove every line from the 2nd file that contains any of the values listed in first file. Example of strings to delete: line1 *abca* end of... (1 Reply)
Discussion started by: upstate_boy
1 Replies

3. Shell Programming and Scripting

How to grep/awk/egrep two values for given output?

Dear Friends, I have a command which can result following output. Packet is: /var/adm/yyyy/pkt6043 Intended for network : /vob/repo I would like to retrive pkt6043 and /vob/repo using single command. Blue color test will be always contstant and red color text will be dynamic ... (2 Replies)
Discussion started by: baluchen
2 Replies

4. Linux

Removing non printing characters from a csv file

Hi, I have an csv file and there are some non printable characters(extended ascii) so I am trying to create a clean copy of the csv file . I am using this command: tr -cd "" < /opt/informatica/PowerCenter8.6.0/server/infa_shared/SrcFiles/ThirdParty/locations.csv > ... (4 Replies)
Discussion started by: gerkus
4 Replies

5. Shell Programming and Scripting

Removing lines of a .csv file

Hello, Does anyone have a one-liner to remove lines of a csv file if the value in a specific column is zero? For example, I have this file, 12345,COM,5,0,N,29.95,Y 12345,MOM,1,0,N,29.95,Y 12345,COM,4,0,N,9.99,Y 12345,MOM,0,2,N,9.99,Y 12345,REN,0,1,N,9.99,Y and I want to remove lines... (4 Replies)
Discussion started by: palex
4 Replies

6. Shell Programming and Scripting

Removing zero values from text file

Hi all, I wrote the following code to remove the value which are 0 in the input file (a columns if numbers). awk 'BEGIN { for (i=1; i<=NF; i++) if ($i) printf("%13.6e\n",$i) }' $1 >> $2 The script works if the zeros are written as 0.0000 but not as 0.000000e+00 In... (10 Replies)
Discussion started by: f_o_555
10 Replies

7. Shell Programming and Scripting

removing lines with similar values from file

Hello, got a file with this structure: 33274 171030 02/29/2012 37897 P_GEH 2012-02-29 10:31:26 33275 171049 02/29/2012 38132 P_GEH 2012-02-29 10:35:27 33276 171058 02/29/2012 38515 P_GEH 2012-02-29 10:43:26 33277 170748 02/29/2012 40685 P_KOM ... (3 Replies)
Discussion started by: krecik28
3 Replies

8. Shell Programming and Scripting

Removing \n from .csv file

Hi, I have a requirement like my .csv file is generating from a db2 table using export command like below: file format: ----------- 2011 4 0 0 N S C C "BHPC BHPC" 0 0 0 2011 5 0 0 N S C C "BHPC BHPC" 0 0 0 here BHPC is having new line character and because this when i am trying... (4 Replies)
Discussion started by: RRVARMA
4 Replies

9. Shell Programming and Scripting

Bash script help - removing certain rows from .csv file

Hello Everyone, I am trying to find a way to take a .csv file with 7 columns and a ton of rows (over 600,000) and remove the entire row if the cell in forth column is blank. Just to give you a little background on why I am doing this (just in case there is an easier way), I am pulling... (3 Replies)
Discussion started by: MrTuxor
3 Replies

10. UNIX for Beginners Questions & Answers

Removing commas from CSV file

Hi I'm creating a sh script to generate a csv file. The CSV contains the values from a sql table. The content looks this: a,b,c,c2,c3,,,,,,,,,,,d,e I have some code that can separate the fields using the comma as delimiter, but some values actually contain commas, such as... (2 Replies)
Discussion started by: preema
2 Replies
TABFUNC(1)						      General Commands Manual							TABFUNC(1)

NAME
tabfunc - convert table to functions for rcalc, etc. SYNOPSIS
tabfunc [ -i ] func1 [func2 ..] DESCRIPTION
Tabfunc reads a table of numbers from the standard input and converts it to an expression suitable for icalc(1), rcalc(1) and their cousins. The input must consist of a M x N matrix of real numbers, with exactly one row per line. The number of columns must always be the same in each line, separated by whitespace and/or commas, with no missing values. The first column is always the independent variable, whose value indexes all of the other elements. This value does not need to be evenly spaced, but it must be either monotonically increas- ing or monotonically decreasing. (I.e. it cannot go up and then down, or down and then up.) Maximum input line width is 4096 characters and the maximum number of data rows is 1024. Input lines not beginning with a numerical value will be silently ignored. The command-line arguments given to tabfunc are the names to be assigned to each column. Tabfunc then produces a single function for each column given. If there are some columns which should be skipped, the dummy name "0" may be given instead of a valid identifier. (It is not necessary to specify a dummy name for extra columns at the end of the matrix.) The -i option causes tabfunc to produce a description that will interpolate values in between those given for the independent variable on the input. EXAMPLE
To convert a small data table and feed it to rcalc for some calculation: rcalc -e `tabfunc f1 f2 < table.dat` -f com.cal AUTHOR
Greg Ward SEE ALSO
cnt(1), icalc(1), neaten(1), rcalc(1), rlam(1), total(1) RADIANCE
10/8/97 TABFUNC(1)
All times are GMT -4. The time now is 12:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy