sed csv remove conditionally


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed csv remove conditionally
# 1  
Old 08-03-2007
sed csv remove conditionally

Hello,
I have many csv file, but I would like to delete lines with some values in a column conditionally.

My example look like this,

ex1e, ex2g, ex39, pasg, ssg, mrlc, pc, kb, coop
-112, -53, -177, 64, 62, 71, 1, 487, 20
-101, -61, -53, 0, 32767, 51, 0, 239, 30
-57, -95, -215, 144, 174, 42, 0, 316, 12
23, 47, 32, 42, -300, 71, 0, 112, -23
11, 45, 44, -20, 300, 51, 1, 279, 32
31, 52, 63, 74, 301, 40, 1, 28, -2
82, 35, 23, 77, -299, 51, 1, 0, -1

What I want to do is to look at the field, ssg (column5), then find values only greater than or equal to -300 and less than or equal to 300. So, the result should be look like this.
ex1e, ex2g, ex39, pasg, ssg, mrlc, pc, kb, coop
-112, -53, -177, 64, 62, 71, 1, 487, 20
-57, -95, -215, 144, 174, 42, 0, 316, 12
23, 47, 32, 42, -300, 71, 0, 112, -23
11, 45, 44, -20, 300, 51, 1, 279, 32
82, 35, 23, 77, -299, 51, 1, 0, -1

I tried to use sed below, but I got stuck after that.
sed '/^\([^,]*,\)\{6\}300,......

Please help me.

Jae
# 2  
Old 08-03-2007
Code:
awk -F", " '{ if ( $5 >= -300 && $5 <= 300 ) { print } }' filename

# 3  
Old 08-03-2007
But, header row was removed

Thank you for code, matrixmadhan.
But, header row was removed.
I want to keep header row, so the result should be.

ex1e, ex2g, ex39, pasg, ssg, mrlc, pc, kb, coop
-112, -53, -177, 64, 62, 71, 1, 487, 20
-57, -95, -215, 144, 174, 42, 0, 316, 12
23, 47, 32, 42, -300, 71, 0, 112, -23
11, 45, 44, -20, 300, 51, 1, 279, 32
82, 35, 23, 77, -299, 51, 1, 0, -1

Thanks,

Jae
# 4  
Old 08-03-2007
Here is one way keeping the header:
Code:
#!/bin/ksh
typeset -i f5
sed '1q' input_file
IFS=','
sed '1d' input_file | \
while read f1 f2 f3 f4 f5 f6 f7 f8 f9
do
  if [ ${f5} -ge -300 -a ${f5} -le 300 ]; then
    echo $f1','$f2','$f3','$f4','$f5','$f6','$f7','$f8','$f9
  fi
done

# 5  
Old 08-03-2007
Code:
awk -F", " '{ if ( NR == 1 ) { print }; if ( $5 >= -300 && $5 <= 300 ) { print } }' filename

I know this is not a cleaner way ! Smilie
# 6  
Old 08-03-2007
Resolved, but looks dirty

Thank you for code, matrixmadhan and Shell_life.
But, I simply do that, looks dirty though.
head -1 filename > filename1.csv
awk -F", " '{ if ( $5 >= -300 && $5 <= 300 ) { print } }' filename >> filename1.csv

Cheers,

Jae
# 7  
Old 08-03-2007
Quote:
Originally Posted by matrixmadhan
Code:
awk -F", " '{ if ( NR == 1 ) { print }; if ( $5 >= -300 && $5 <= 300 ) { print } }' filename

I know this is not a cleaner way ! Smilie
Code:
nawk -F',' 'FNR>1 && $5 >= -300 && $5 <= 300' filename

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to conditionally display and remove first line only?

I have a maildir hierarchy of 90k eml files and; 1) I would like to walk the tree and display the first line from any file, whose first line begins with; From - That's "From space dash space" and only if it's the first seven characters, of the first line in the file. 2) I would also... (12 Replies)
Discussion started by: jasn
12 Replies

2. Shell Programming and Scripting

Remove ^L from csv not working...

hello... i have a requirement to convert a xls file to csv in RHEL 6.5 and ftp it to a windows location. i am using xls2csv utility to convert the file in linux. Input xls file attached... I have used below commands to convert the file to csv: xls2csv -x test.xls -s cp1252 -d 8859-1... (3 Replies)
Discussion started by: rhel65
3 Replies

3. Shell Programming and Scripting

Using sed to remove a column from a CSV

I found that the following works to remove the first column in my file when my CSV is delimited with a simple comma: sed -i 's/*,//' file.csv However, I have a new file where the fields are encapsulated with double quotes - general example of file: "Internal ID", "External ID", "Name"... (8 Replies)
Discussion started by: lojkyelo
8 Replies

4. Shell Programming and Scripting

sed to conditionally delete multiple lines

Hello I'd like to remove any line in an output file that is preceded by one or more warning messages (each warning is on a separate line). Example : WARNING: Estimation of lower confidence limit of \rho failed; setting it to 0. 822 28447 ... (4 Replies)
Discussion started by: jossojjos
4 Replies

5. Shell Programming and Scripting

Need to remove some value in CSV file

Hi All A .csv file containing multiple value on it. i need to remove some value from it. Please advise how can i find multiple value from that and remove itself. Please provide solution for this. Thanks in advance. (4 Replies)
Discussion started by: yash1978
4 Replies

6. Shell Programming and Scripting

Remove file conditionally between two server using sftp

Hi, I am having 2 servers, Need to delete files from server1 if those files exist in server2 other wise no action using sftp .And the process is non-interactive way. I have got confused how to check the condition in sftp because there is non of the shell condition or loop command is executing.... (2 Replies)
Discussion started by: posix
2 Replies

7. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in... (6 Replies)
Discussion started by: metronomadic
6 Replies

8. Shell Programming and Scripting

Remove text from a csv file using sed

I am trying to remove the ita from this file: "1234ita","john","smith" "56789ita","jim","thomas" the command i am using is: sed '/^ita/d' infile.csv > outfile.csv it is running but doing nothing, very rarely use sed so trying to learn it any help would be appreciated (2 Replies)
Discussion started by: Pablo_beezo
2 Replies

9. Shell Programming and Scripting

Remove newline character conditionally

Hi All, I have 5000 records like this Request_id|Type|Status|Priority|Ticket Submitted Date and Time|Actual Resolved Date and Time|Current Ticket Owner Group|Case final Ticket Owner Group|Customer Severity|Reported Symptom/Request|Component|Hot Topic|Reason for Missed SLA|Current Ticket... (2 Replies)
Discussion started by: j_53933
2 Replies

10. Shell Programming and Scripting

using sed to conditionally extract stanzas of a file based on a search string

Dear All, I have a file with the syntax below (composed of several <log ..... </log> stanzas) I need to search this file for a number e.g. 2348022225919, and if it is found in a stanza, copy the whole stanza/section (<log .... </log>) to another output file. The numbers to search for are... (0 Replies)
Discussion started by: aitayemi
0 Replies
Login or Register to Ask a Question