Sponsored Content
Top Forums Shell Programming and Scripting omitting lines from file A that are in file B Post 302168501 by gneen on Monday 18th of February 2008 03:22:26 PM
Old 02-18-2008
omitting lines from file A that are in file B

I've got file A with (say) 1M lines in it ... ascii text, space delimited ...

I've got file B with (say) 10M lines in it ... same structure.

I want to remove any lines from A that appear (identically) in B and print the remaining (say) 900K lines. (And I want to do it in zero time of course!)

Best I've come up with so far is somehow marking the lines in A, then doing a sort and applying an awk script to the result so that the marked lines are only printed if the following (or previous) line isn't "identical" except for the mark.

But after 1000 years of shell programming I've GOT to believe I'm missing an easier/faster solution ... I'm using bash and cygwin tools - and compiling is not an option.

ADVthanksANCE for your help!
=Gneen
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies

2. Shell Programming and Scripting

Strings from one file which exactly match to the 1st column of other file and then print lines.

Hi, I have two files. 1st file has 1 column (huge file containing ~19200000 lines) and 2nd file has 2 columns (small file containing ~6000 lines). ################################# huge_file.txt a a ab b ################################## small_file.txt a 1.5 b 2.5 ab ... (4 Replies)
Discussion started by: AshwaniSharma09
4 Replies

3. Shell Programming and Scripting

Extract some lines from one file and add those lines to current file

hi, i have two files. file1.sh echo "unix" echo "linux" file2.sh echo "unix linux forums" now the output i need is $./file2.sh unix linux forums (3 Replies)
Discussion started by: snreddy_gopu
3 Replies

4. Shell Programming and Scripting

Omitting sections of file that contain word

I have a configuration file that contains hundreds of these chunks. Each "chunk" is the section that begins with "define service {" and ends with "}". define service { check_command check_proc!java hostgroup_name service_description ... (5 Replies)
Discussion started by: SkySmart
5 Replies

5. Shell Programming and Scripting

Bash script to send lines of file to new file based on Regex

I have a file that looks like this: cat includes CORP-CRASHTEST-BU e:\crashplan\ CORP-TEST /usr/openv/java /usr/openv/logs /usr/openv/man CORP-LABS_TEST /usr/openv/java /usr/openv/logs /usr/openv/man What I want to do is make three new files with just those selections. So the three... (4 Replies)
Discussion started by: newbie2010
4 Replies

6. Shell Programming and Scripting

Put the lines from file A to end of lines in file B

I really can't figure this one out. I have 2 files, one file is a list of hostnames and the other is a list of their corresponding IPs: fileA: example.com another.org thirdie.net fileB: 1.1.1.1 2.2.2.2 3.3.3.3 I want to create a fileC that looks like: example.com 1.1.1.1... (2 Replies)
Discussion started by: zstar
2 Replies

7. UNIX for Dummies Questions & Answers

Add strings from one file at the end of specific lines in text file

Hello All, this is my first post so I don't know if I am doing this right. I would like to append entries from a series of strings (contained in a text file) consecutively at the end of specifically labeled lines in another file. As an example: - the file that contains the values to be... (3 Replies)
Discussion started by: gus74
3 Replies

8. Shell Programming and Scripting

Trying to take file numbers from a file, pass them to sed to change strings in corresponding lines

I have a bunch of file numbers in the file 'test': I'm trying the above command to change all the instances of "H" to "Na+" in the file testsds.pdb at the line numbers indicated in the file 'test'. I've tried the following and various similar alternatives but nothing is working: cat test |... (3 Replies)
Discussion started by: crunchgargoyle
3 Replies

9. Shell Programming and Scripting

How to compare 2 files and create a result file with unmatched lines from first file.?

HI, I have 2 text files. file1 and file2. file1.txt (There are no duplicates in this file) 1234 3232 4343 3435 6564 6767 1213 file2.txt 1234,wq,wewe,qwqw 1234,as,dfdf,dfdf 4343,asas,sdds,dsds 6767,asas,fdfd,fdffd I need to search each number in file1.txt in file2.txt's 1st... (6 Replies)
Discussion started by: Little
6 Replies

10. Shell Programming and Scripting

Find all lines in file such that each word on that line appears in at least n lines of the file

I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies
COLCRT(1)							   User Commands							 COLCRT(1)

NAME
colcrt - filter nroff output for CRT previewing SYNOPSIS
colcrt [options] [file ...] DESCRIPTION
colcrt provides virtual half-line and reverse line feed sequences for terminals without such capability, and on which overstriking is destructive. Half-line characters and underlining (changed to dashing `-') are placed on new lines in between the normal output lines. OPTIONS
-, --no-underlining Suppress all underlining. This option is especially useful for previewing allboxed tables from tbl(1). -2, --half-lines Causes all half-lines to be printed, effectively double spacing the output. Normally, a minimal space output format is used which will suppress empty lines. The program never suppresses two consecutive empty lines, however. The -2 option is useful for sending output to the line printer when the output contains superscripts and subscripts which would otherwise be invisible. -V, --version Output version information and exit. -h, --help Output help and exit. EXAMPLES
A typical use of colcrt would be tbl exum2.n | nroff -ms | colcrt - | more SEE ALSO
nroff(1), troff(1), col(1), more(1), ul(1) BUGS
Should fold underlines onto blanks even with the '-' option so that a true underline character would show. Can't back up more than 102 lines. General overstriking is lost; as a special case '|' overstruck with '-' or underline becomes '+'. Lines are trimmed to 132 characters. Some provision should be made for processing superscripts and subscripts in documents which are already double-spaced. HISTORY
The colcrt command appeared in 3.0BSD. AVAILABILITY
The colcrt command is part of the util-linux package and is available from Linux Kernel Archive <ftp://ftp.kernel.org/pub/linux/utils/util- linux/>. util-linux September 2011 COLCRT(1)
All times are GMT -4. The time now is 06:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy