Sponsored Content
Top Forums Shell Programming and Scripting sed to delete items in an array from a file Post 302976030 by bash_in_my_head on Wednesday 22nd of June 2016 05:44:25 PM
Old 06-22-2016
The items to delete are in the input file. This script is just to prove the loop works. input is the file I want to delete from. I could use sed edit in place or not. Some internet posts said the sed -i option was not working in a loop. I did not verify.

I have an 18MB data file with a lot of redundant repeated lines that need to be deleted. Having an outside file containing a list of those lines to be deleted is best for this script. I want to read into the loop the source file and output to a second file. The source file has about 15 MB of repeated useless data to remove so looping over the file seems the best choice. I dont really care what construct I use but I am not having luck with those I have tested.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete line in file with sed

sed "/$titlesearch/d" movielist will delete any line in the file movielist that matches $titlesearch, but this is only a screen print correct ? how can I have sed to actually delete the line from the file so that it doesnt appear next time I open the file ? thanks Steffen (8 Replies)
Discussion started by: forever_49ers
8 Replies

2. Shell Programming and Scripting

Delete particular value from file using 'sed'

Hi, I have two files test1,test2. If the number in test1 file exist in test2 then i want to remove that from test2 file. Ex: File- test1 12 13 14 15 ============== File- test2 1A~12 2B~13 3C~33 4D~22 I want to remove row which contains 12,13 from test2. I am using this sed... (2 Replies)
Discussion started by: sai_nj
2 Replies

3. Shell Programming and Scripting

awk between items including items

OS=HP-UX ksh The following works, except I want to include the <start> and <end> in the output. awk -F '<start>' 'BEGIN{RS="<end>"; OFS="\n"; ORS=""} {print $2} somefile.log' The following work in bash but not in ksh sed -n '/^<start>/,/^<end>/{/LABEL$/!p}' somefile.log (4 Replies)
Discussion started by: Ikon
4 Replies

4. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "§" : §field1§$field2§$field3§$field4§$field5§$field6§$field§ in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies

5. UNIX for Dummies Questions & Answers

sed how to delete between two words within a file

I'm hoping someone could help me out please :) I have several .txt files with several hundred lines in each that look like this: 10241;</td><td>10241</td><td class="b">x2801;</td><td>2801</td><td>TEXT-1</td></tr> 10242;</td><td>10242</td><td... (4 Replies)
Discussion started by: martinsmith
4 Replies

6. Shell Programming and Scripting

sed - delete everything from the first IP in the file

Hi, I have a text file with content as follows: servers list ips 10.10.10.1 test 2 3 4 desktop station 10.10.10.3 4 test 4 laptops quesy 2013 2012 10.100.18.0 test 4 6 8 all I need is: servers list ips desktop station laptops quesy 2013 2012All I need is the correct command or script that... (4 Replies)
Discussion started by: e_mikey_2000
4 Replies

7. Shell Programming and Scripting

Add items to ksh array

Hi I need a code help put below items in to array and display in order like this Thanks, Ashan 0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616 ------ 0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616 (1 Reply)
Discussion started by: ashanabey
1 Replies

8. UNIX for Advanced & Expert Users

Sed: couldn't write 1378 items to stdout: No space left on device

Hi , In file first line start with "",when trying to remove using sed i am getting the below error .Please advise sed -e 's///g' -e 's/$]//g' segment1.txt >>segment_m1 sed: couldn't write 1378 items to stdout: No space left on device Thanks, Mohan (1 Reply)
Discussion started by: mohan705
1 Replies

9. Shell Programming and Scripting

Removing duplicate items from an array

Hello Experts, I am trying to write a shell script to find duplicate items in an array, this is what i have tried : #!/bin/bash echo "This is another sample Progg to remove duplicate items from an array" echo "How many number do you want to insert" read n for (( i=0; i<$n; i++ )) do ... (5 Replies)
Discussion started by: mukulverma2408
5 Replies

10. Shell Programming and Scripting

Pass an array to awk to sequentially look for a list of items in a file

Hello, I need to collect some statistical results from a series of files that are being generated by other software. The files are tab delimited. There are 4 different sets of statistics in each file where there is a line indicating what the statistic set is, followed by 5 lines of values. It... (8 Replies)
Discussion started by: LMHmedchem
8 Replies
ARRAY_UNIQUE(3) 							 1							   ARRAY_UNIQUE(3)

array_unique - Removes duplicate values from an array

SYNOPSIS
array array_unique (array $array, [int $sort_flags = SORT_STRING]) DESCRIPTION
Takes an input $array and returns a new array without duplicate values. Note that keys are preserved. array_unique(3) sorts the values treated as string at first, then will keep the first key encountered for every value, and ignore all following keys. It does not mean that the key of the first related value from the unsorted $array will be kept. Note Two elements are considered equal if and only if (string) $elem1 === (string) $elem2 i.e. when the string representation is the same, the first element will be used. PARAMETERS
o $array - The input array. o $sort_flags - The optional second parameter $sort_flags may be used to modify the sorting behavior using these values: Sorting type flags: o SORT_REGULAR - compare items normally (don't change types) o SORT_NUMERIC - compare items numerically o SORT_STRING - compare items as strings o SORT_LOCALE_STRING - compare items as strings, based on the current locale. RETURN VALUES
Returns the filtered array. CHANGELOG
+--------+---------------------------------------------------+ |Version | | | | | | | Description | | | | +--------+---------------------------------------------------+ |5.2.10 | | | | | | | Changed the default value of $sort_flags back to | | | SORT_STRING. | | | | | 5.2.9 | | | | | | | Added the optional $sort_flags defaulting to | | | SORT_REGULAR. Prior to 5.2.9, this function used | | | to sort the array with SORT_STRING internally. | | | | +--------+---------------------------------------------------+ EXAMPLES
Example #1 array_unique(3) example <?php $input = array("a" => "green", "red", "b" => "green", "blue", "red"); $result = array_unique($input); print_r($result); ?> The above example will output: Array ( [a] => green [0] => red [1] => blue ) Example #2 array_unique(3) and types <?php $input = array(4, "4", "3", 4, 3, "3"); $result = array_unique($input); var_dump($result); ?> The above example will output: array(2) { [0] => int(4) [2] => string(1) "3" } SEE ALSO
array_count_values(3). NOTES
Note Note that array_unique(3) is not intended to work on multi dimensional arrays. PHP Documentation Group ARRAY_UNIQUE(3)
All times are GMT -4. The time now is 02:52 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy