Remove lines that are subsets of other lines in File
Hello everyone,
Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done.
My problem is the following:
I have a file where each line is a sequence of IP addresses, example :
Code:
10.0.0.1 10.0.0.2
10.0.0.5 10.0.0.1 10.0.0.2
...
What I'd like to do, is to remove lines that are completely matched in other lines. In the previous example, "Line 1" would be deleted as it is contained in "Line 2".
So far, I've worked with python and set() objects to get the job done but I've got more than 100K lines and sets lookups are becoming time consuming as the program goes :/
Thanks for you help
Moderator's Comments:
Use code tags, thanks.
Last edited by zaxxon; 04-22-2015 at 06:31 AM..
Reason: code tags and missing a dot
All,
I have a text file with several entries like below:
personname
personname.domain.com
I know there is a way to use vi to remove only the personname.domain.com line. Can someone help? I believe that it involves /s/g/ something...I just can't remember the exact syntax.
Thanks (2 Replies)
Hi gurus,
i'm trying to remove a number of lines from a large file using the following command:
sed '1,5000d' oldfile > newfile
Somehow the lines in the old file are not deleted...
Am I doing this wrongly? Any suggestions? :confused:
Thanks! :)
wee (10 Replies)
A small question
I have a test.txt file
I have contents as:
a:google
b:yahoo
:
c:facebook
:
d:hotmail
How do I remove the line with :
my output should be
a:google
b:yahoo
c:facebook
d:hotmail (5 Replies)
Hi,
I'm not a expert in shell programming, so i've come here to take help from u gurus.
I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command.
I've a datatable csv of the below format -
--in file format
xx,xx,xx ,xx , , , , ,,xx,
xxxx,, ,, xxx,... (11 Replies)
Hey Gang-
I have a list of servers. I want to exclude servers that begin with and end with certain characters. Is there an easy command to do this?
Example
wvm1234dev
wvm1234pro
uvm1122dev
uvm1122bku
uvm1344dev
I want to exclude any lines that start with "wvm" OR "uvm" AND end... (7 Replies)
Hi,
I have a huge file which has Lacs of lines. File system got full.
I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
I have a file that contains the following:
Party_Id1;Party_id2;Party_id3;
1;2;3;
0
0
4;5;6;
0
7;8;9;
How can I adjust the file so it looks like this:
Party_Id1;Party_id2;Party_id3;
1;2;3;
4;5;6;
7;8;9;
I Think the '0' is something like a carriage return, I don't know. But how... (2 Replies)
I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted.
keepout:
user1
buser3
anuser19
notheruser27
database:
user1,2343,"information about",field,blah,34
user2,4231,"mo info",etc,stuff,43
notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
I have been searching and trying to come up with an awk that will perform the following on a
converted text file (original is a pdf).
1. Since the first two lines are (begin with) text they are removed
2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies
LEARN ABOUT ULTRIX
dhb
dhb(4) Kernel Interfaces Manual dhb(4)Name
dhb - DHB32 communications multiplexer
Syntax
device dmb0 at vaxbi? node? flags 0x????
vector dmbsint dmbaint dmblint
Description
A DHB32 device provides 16 asynchronous communication lines with full modem control. The DHB32 and the DMB32 share a common software
device driver. For this reason, the configuration line is the same for both the DHB32 and DMB32.
Each line attached to a DHB32 serial line port behaves as described in Input and output for each line can independently be set to run at
any of 16 speeds. See for the encoding.
Bit i of flags may be specified for a to say that a line should be treated as a hardwired connection with carrier always present. If bit i
of flags is not set, the line will operate under full modem control. Modem lines will operate in accordance to the CD (carrier detect),
DSR (data set ready) and CTS (clear to send) leads. Thus, specifying ``flags 0x0004'' in the specification of would cause line 2 on the
DHB32 to be treated as hardwired with carrier always present. In this example, the remainder of the lines will be modem control lines.
Diagnostics
dmbinit: async lines unavailable
This message is produced at system boot time if the DHB32 fails its internal self test indicating that the asynchronous lines have failed
to configure.
dmb%d: fifo overflow
The character input fifo overflowed before it could be serviced. This can happen if the CPU is running with elevated priority for too long
a period of time. Overflow errors may indicate that configuration constraints have been reached.
dmb%d: DMA Error. tbuf = 0x%x
A DMA output transfer failed. This problem can result from a memory error or an invalid pte (page table entry). For a description of the
error code in the ``tbuf'' register. See the DHB32 documentation.
dmb%d: DMB Hardware Error. TX.DMA.START failed
The failed to clear the start bit. Normally, this is cleared to signal that a DMA transfer has completed.
dmb%d: Modem Error. tbuf = 0x%x
Indicates a problem with a modem or its cable. For a description of the error code in the ``tbuf'' register. See the DHB32 documentation.
dmb%d: Internal Error. tbuf = 0x%x
Indicates that the DHB32 detected an internal error. For a description of the error code in the ``tbuf'' register. See the DHB32 User
Guide.
Files
(modem lines only)
See Alsotty(4), MAKEDEV(8), dmbsp(4)dhb(4)
04-22-2015
