Sponsored Content
Top Forums Shell Programming and Scripting Remove lines that are subsets of other lines in File Post 302941866 by MisterJellyBean on Wednesday 22nd of April 2015 07:17:47 AM
Old 04-22-2015
Hello RudiC,


Well, I managed to trim down the dataset with "sort -u input > output", but this will only remove pure duplicates. But still, running my script on this filtered dataset will take ages :/

I presume 'sed' could help me but I can't figure out what regex I should feed him..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove lines from file

file: 1 xxxxxxx 2 xxx xxx 5 xxx xxx ... 180 xxxxxx 200 xxx how to remove any lines with the first number range 1-180 (9 Replies)
Discussion started by: bluemoon1
9 Replies

2. UNIX for Dummies Questions & Answers

vi to remove lines in file

All, I have a text file with several entries like below: personname personname.domain.com I know there is a way to use vi to remove only the personname.domain.com line. Can someone help? I believe that it involves /s/g/ something...I just can't remember the exact syntax. Thanks (2 Replies)
Discussion started by: kjbaumann
2 Replies

3. Shell Programming and Scripting

remove lines from file

Hi gurus, i'm trying to remove a number of lines from a large file using the following command: sed '1,5000d' oldfile > newfile Somehow the lines in the old file are not deleted... Am I doing this wrongly? Any suggestions? :confused: Thanks! :) wee (10 Replies)
Discussion started by: lweegp
10 Replies

4. Shell Programming and Scripting

remove : lines from file

A small question I have a test.txt file I have contents as: a:google b:yahoo : c:facebook : d:hotmail How do I remove the line with : my output should be a:google b:yahoo c:facebook d:hotmail (5 Replies)
Discussion started by: aronmelon
5 Replies

5. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

6. Shell Programming and Scripting

Remove lines from file

Hey Gang- I have a list of servers. I want to exclude servers that begin with and end with certain characters. Is there an easy command to do this? Example wvm1234dev wvm1234pro uvm1122dev uvm1122bku uvm1344dev I want to exclude any lines that start with "wvm" OR "uvm" AND end... (7 Replies)
Discussion started by: idiotboy
7 Replies

7. UNIX for Dummies Questions & Answers

Want to remove all lines but not latest 50 lines from a file

Hi, I have a huge file which has Lacs of lines. File system got full. I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
Discussion started by: prashant2507198
28 Replies

8. Shell Programming and Scripting

Remove lines in file

I have a file that contains the following: Party_Id1;Party_id2;Party_id3; 1;2;3; 0 0 4;5;6; 0 7;8;9; How can I adjust the file so it looks like this: Party_Id1;Party_id2;Party_id3; 1;2;3; 4;5;6; 7;8;9; I Think the '0' is something like a carriage return, I don't know. But how... (2 Replies)
Discussion started by: katled
2 Replies

9. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

10. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies
drslib()																  drslib()

NAME
drs_tool - Command-line interface to DRSLIB SYNPOSIS
DESCRIPTION
USAGE
Usage Usage: drs_tool [command] [options] [drs-pattern] command: list list publication-level datasets todo show file operations pending for the next version upgrade make changes to the selected datasets to upgrade to the next version mapfile make a mapfile of the selected dataset history list all versions of the selected dataset init initialise CMIP5 product detection data OPTIONS
--help display this help message and exit --version output version information and exit -h, --help show this help message and exit -R ROOT, --root=ROOT Root directory of the DRS tree -I INCOMING, --incoming=INCOMING Incoming directory for DRS files. Defaults to <root>/output -a ACTIVITY, --activity=ACTIVITY Set DRS attribute activity for dataset discovery -p PRODUCT, --product=PRODUCT Set DRS attribute product for dataset discovery -i INSTITUTE, --institute=INSTITUTE Set DRS attribute institute for dataset discovery -m MODEL, --model=MODEL Set DRS attribute model for dataset discovery -e EXPERIMENT, --experiment=EXPERIMENT Set DRS attribute experiment for dataset discovery -f FREQUENCY, --frequency=FREQUENCY Set DRS attribute frequency for dataset discovery -r REALM, --realm=REALM Set DRS attribute realm for dataset discovery -v VERSION, --version=VERSION Force version upgrades to this version -P FILE, --profile=FILE Profile the script exectuion into FILE --detect-product Automatically detect the DRS product of incoming data REQUIRES
FILES
drs_tool uses metaconfig to configure and store configuration information. Its configuration may be found then in a metaconfig.conf file, in /etc/metaconfig/metaconfig.conf on Debian systems, which may be overriden by $HOME/.metaconfig.conf or ./metaconfig.conf ENVIRONMENT VARIABLES
The variable METACONF_CONF VERSION
0.3.0a3 BUGS
To query and report bugs in drs_tool, please use reportbug on Debian systems. SEE ALSO
http://esgf.org/esgf-drslib-site/index.html AUTHOR
DRSLIB was written by Stephen Pascoe, STFC.SC.UK. This manual page was written by Alastair McKinstry. drslib()
All times are GMT -4. The time now is 01:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy