06-22-2016
The items to delete are in the input file. This script is just to prove the loop works. input is the file I want to delete from. I could use sed edit in place or not. Some internet posts said the sed -i option was not working in a loop. I did not verify.
I have an 18MB data file with a lot of redundant repeated lines that need to be deleted. Having an outside file containing a list of those lines to be deleted is best for this script. I want to read into the loop the source file and output to a second file. The source file has about 15 MB of repeated useless data to remove so looping over the file seems the best choice. I dont really care what construct I use but I am not having luck with those I have tested.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
sed "/$titlesearch/d" movielist
will delete any line in the file movielist that matches $titlesearch, but this is only a screen print correct ?
how can I have sed to actually delete the line from the file so that it doesnt appear next time I open the file ?
thanks
Steffen (8 Replies)
Discussion started by: forever_49ers
8 Replies
2. Shell Programming and Scripting
Hi,
I have two files test1,test2. If the number in test1 file exist in test2 then i want to remove that from test2 file.
Ex:
File- test1
12
13
14
15
==============
File- test2
1A~12
2B~13
3C~33
4D~22
I want to remove row which contains 12,13 from test2. I am using this sed... (2 Replies)
Discussion started by: sai_nj
2 Replies
3. Shell Programming and Scripting
OS=HP-UX ksh
The following works, except I want to include the <start> and <end> in the output.
awk -F '<start>' 'BEGIN{RS="<end>"; OFS="\n"; ORS=""} {print $2} somefile.log'
The following work in bash but not in ksh
sed -n '/^<start>/,/^<end>/{/LABEL$/!p}' somefile.log (4 Replies)
Discussion started by: Ikon
4 Replies
4. Shell Programming and Scripting
Hello
i have a big file with a specific format and delimiter is "§" :
§field1§$field2§$field3§$field4§$field5§$field6§$field§
in this file we have a field which are very long (more than 20000 chars !!!!)
so through vi i cant manipulate them.
despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies
5. UNIX for Dummies Questions & Answers
I'm hoping someone could help me out please :)
I have several .txt files with several hundred lines in each that look like this:
10241;</td><td>10241</td><td class="b">x2801;</td><td>2801</td><td>TEXT-1</td></tr>
10242;</td><td>10242</td><td... (4 Replies)
Discussion started by: martinsmith
4 Replies
6. Shell Programming and Scripting
Hi,
I have a text file with content as follows:
servers list ips 10.10.10.1 test 2 3 4
desktop station 10.10.10.3 4 test 4
laptops quesy 2013 2012 10.100.18.0 test 4 6 8 all I need is:
servers list ips
desktop station
laptops quesy 2013 2012All I need is the correct command or script that... (4 Replies)
Discussion started by: e_mikey_2000
4 Replies
7. Shell Programming and Scripting
Hi
I need a code help put below items in to array and display in order like this
Thanks,
Ashan
0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616
------
0AD4
R2
03A1
000198701333
0AD4
R1
270B
000198701616 (1 Reply)
Discussion started by: ashanabey
1 Replies
8. UNIX for Advanced & Expert Users
Hi ,
In file first line start with "",when trying to remove using sed i am getting the below error .Please advise
sed -e 's///g' -e 's/$]//g' segment1.txt >>segment_m1
sed: couldn't write 1378 items to stdout: No space left on device
Thanks,
Mohan (1 Reply)
Discussion started by: mohan705
1 Replies
9. Shell Programming and Scripting
Hello Experts,
I am trying to write a shell script to find duplicate items in an array, this is what i have tried :
#!/bin/bash
echo "This is another sample Progg to remove duplicate items from an array"
echo "How many number do you want to insert"
read n
for (( i=0; i<$n; i++ ))
do
... (5 Replies)
Discussion started by: mukulverma2408
5 Replies
10. Shell Programming and Scripting
Hello,
I need to collect some statistical results from a series of files that are being generated by other software. The files are tab delimited. There are 4 different sets of statistics in each file where there is a line indicating what the statistic set is, followed by 5 lines of values. It... (8 Replies)
Discussion started by: LMHmedchem
8 Replies
LEARN ABOUT PHP
array_unique
ARRAY_UNIQUE(3) 1 ARRAY_UNIQUE(3)
array_unique - Removes duplicate values from an array
SYNOPSIS
array array_unique (array $array, [int $sort_flags = SORT_STRING])
DESCRIPTION
Takes an input $array and returns a new array without duplicate values.
Note that keys are preserved. array_unique(3) sorts the values treated as string at first, then will keep the first key encountered for
every value, and ignore all following keys. It does not mean that the key of the first related value from the unsorted $array will be kept.
Note
Two elements are considered equal if and only if (string) $elem1 === (string) $elem2 i.e. when the string representation is the
same, the first element will be used.
PARAMETERS
o $array
- The input array.
o $sort_flags
- The optional second parameter $sort_flags may be used to modify the sorting behavior using these values: Sorting type flags:
o SORT_REGULAR - compare items normally (don't change types)
o SORT_NUMERIC - compare items numerically
o SORT_STRING - compare items as strings
o SORT_LOCALE_STRING - compare items as strings, based on the current locale.
RETURN VALUES
Returns the filtered array.
CHANGELOG
+--------+---------------------------------------------------+
|Version | |
| | |
| | Description |
| | |
+--------+---------------------------------------------------+
|5.2.10 | |
| | |
| | Changed the default value of $sort_flags back to |
| | SORT_STRING. |
| | |
| 5.2.9 | |
| | |
| | Added the optional $sort_flags defaulting to |
| | SORT_REGULAR. Prior to 5.2.9, this function used |
| | to sort the array with SORT_STRING internally. |
| | |
+--------+---------------------------------------------------+
EXAMPLES
Example #1
array_unique(3) example
<?php
$input = array("a" => "green", "red", "b" => "green", "blue", "red");
$result = array_unique($input);
print_r($result);
?>
The above example will output:
Array
(
[a] => green
[0] => red
[1] => blue
)
Example #2
array_unique(3) and types
<?php
$input = array(4, "4", "3", 4, 3, "3");
$result = array_unique($input);
var_dump($result);
?>
The above example will output:
array(2) {
[0] => int(4)
[2] => string(1) "3"
}
SEE ALSO
array_count_values(3).
NOTES
Note
Note that array_unique(3) is not intended to work on multi dimensional arrays.
PHP Documentation Group ARRAY_UNIQUE(3)