Deleting consecutive equal values in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting consecutive equal values in a file
# 15  
Old 11-20-2014
If the data list will always have a comma or comma-space separator, then this might work...
This modifies original data.
Code:
# File data1 = BS0000, BS0000 solution, CS0000, CS0000, CS0000 InterCompany
               a,b,c,d,e,e,f,g
#!/bin/bash
cp ./data1 /tmp/data
sed -i 's/,\s*/\n/g' /tmp/data
while read x
do
    if [ $(egrep -c "^$x$" /tmp/data) -gt 1 ]; then
        sed -i "s/\b$x,\s*//g" ./data1
   fi
done < /tmp/data
rm /tmp/data
cat ./data1
### eof #

output
--------
B50000, BS0000 solution, CS0000 InterCompany
a,b,c,d,f,g


Last edited by ongoto; 11-21-2014 at 10:50 PM.. Reason: add \b flag to sed regex "s/\b$x,\s*//g"
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete row if both percentage values are equal to zero

Hello, I have compiled a script but I have stucked at one point. Each line contains two pcs of % value and what I want to do is to delete any line if both % values are zero. data: expected output: ow3 should be deleted as both percentage value in related line are equal to zero. ... (2 Replies)
Discussion started by: baris35
2 Replies

2. Shell Programming and Scripting

awk print values between consecutive lines

I have a file in below format: file01.txt TERM TERM TERM ABC 12315 68.53 12042013 165144 ABC 12315 62.12 12042013 165145 ABC 12315 122.36 12052013 165146 ABC 12315 582.18 12052013 165147 ABC 12316 2.36 12052013 165141 ABC 12316 ... (8 Replies)
Discussion started by: alex2005
8 Replies

3. Shell Programming and Scripting

Adding the corresponding values for every 5th consecutive numbers

Dear All, I have a file which is as follows: Input File: 231 100.1 233 99 235 200.9 238 80.1 239 90.2 240 77.0 243 99.5 245 16.20 246 13.55 247 11.8 249 13.7 250 99.6 (1 Reply)
Discussion started by: NamS
1 Replies

4. UNIX for Dummies Questions & Answers

Replacing values in a column if they equal a certain value

Hi, I have a tab delimited text file where some lines have the string "NA" in the second column. For these lines, I want to replace NA with the value in the first column, the symbol underscore followed by the value in the fourth column. How do I go about doing that? Thanks! Input: 1 ... (3 Replies)
Discussion started by: evelibertine
3 Replies

5. Shell Programming and Scripting

How to compare the values of a column in awk in a same file and consecutive lines..

I would like to compare the values of 2nd column of consecutive lines of same file in such a way so that if the difference between first value and second value is more than 100 it should print complete line else ignore line. Input File ========== PDB 2500 RTDB 123 RTDB-EAGLE 122 VSCCP 2565... (4 Replies)
Discussion started by: manuswami
4 Replies

6. Shell Programming and Scripting

AWK: combining consecutive values in a field

Hi, Here is my sample input X 2 AAA Y 3 BBB Y 2 CCC Z 4 DDD In field 1, if the value of one line is same as that of next line, I want to concatenate the corresponding value of the second line in the third field with the value of the third field of first line. And I dont need the third... (2 Replies)
Discussion started by: polsum
2 Replies

7. Shell Programming and Scripting

compare columns for equal values and output a summary

Hi all I am trying to scan a file that has 3 columns: red blue 123351 red blue 848655 red blue 126354 red blue 023158 black white 654896 red blue 650884 I want an output that sums the rows that have matching columns 1 and 2 :wall: red blue has 5 entries black white has 1 entry ... (4 Replies)
Discussion started by: reno
4 Replies

8. Programming

FORTRAN: Getting Consecutive DIST Values

I have this code and I want to get the first two consecutive distances (obtained by calling GET_TH(IT, 'DIST', DIST) at index IT, the result stored in DIST) for which (ITFLG(IT) .NE. 1). L_FIRST get the first DIST for which ITFLG(IT) .NE. 1. Getting a bit confused on how to achieve this. ... (0 Replies)
Discussion started by: kristinu
0 Replies

9. Shell Programming and Scripting

concatenate consecutive field values

Hi, I have a file like this A Bob A Sam A John B David C Paul C Sandra If the consecutive field values in column one is same, then concatenate the corresponding strings. So, I need an output like this, A Bob_Sam_John B David C Paul_Sandra I usually work with excel but... (3 Replies)
Discussion started by: polsum
3 Replies

10. Shell Programming and Scripting

My Values are Equal but They are Not

Does anybody understand why this is not being interpreted as true. Script: #!/bin/bash errored=`grep "errored" new_update_scripts.txt` echo $errored = "errored" if ; then echo true else echo false fi Output: $ UpdateScripts errored = errored false (7 Replies)
Discussion started by: scottwmackey
7 Replies
Login or Register to Ask a Question
ATF-SH(1)						    BSD General Commands Manual 						 ATF-SH(1)

NAME
atf-sh [-s shell] -- interpreter for shell-based test programs SYNOPSIS
atf-sh script DESCRIPTION
atf-sh is an interpreter that runs the test program given in script after loading the atf-sh(3) library. atf-sh is not a real interpreter though: it is just a wrapper around the system-wide shell defined by ATF_SHELL. atf-sh executes the inter- preter, loads the atf-sh(3) library and then runs the script. You must consider atf-sh to be a POSIX shell by default and thus should not use any non-standard extensions. The following options are available: -s shell Specifies the shell to use instead of the value provided by ATF_SHELL. ENVIRONMENT
ATF_LIBEXECDIR Overrides the builtin directory where atf-sh is located. Should not be overridden other than for testing purposes. ATF_PKGDATADIR Overrides the builtin directory where libatf-sh.subr is located. Should not be overridden other than for testing purposes. ATF_SHELL Path to the system shell to be used in the generated scripts. Scripts must not rely on this variable being set to select a specific interpreter. EXAMPLES
Scripts using atf-sh(3) should start with: #! /usr/bin/env atf-sh Alternatively, if you want to explicitly choose a shell interpreter, you cannot rely on env(1) to find atf-sh. Instead, you have to hardcode the path to atf-sh in the script and then use the -s option afterwards as a single parameter: #! /path/to/bin/atf-sh -s/bin/bash ENVIRONMENT
ATF_SHELL Path to the system shell to be used in the generated scripts. SEE ALSO
atf-sh(3) BSD
September 27, 2014 BSD