04-29-2008
concatenate all duplicate line in a file.
Hi All,
i have a zip file like the format
794051400123|COM|24|0|BD|R|99.98
794051413727|COM|11|0|BD|R|28.99
794051415622|COM|23|0|BD|R|28.99
883929004676|COM|0|0|BD|R|28.99
794051400123|MOM|62|0|BD|R|99.98
794051413727|MOM|4|0|BD|R|28.99
794051415622|MOM|80|0|BD|R|28.99
883929004676|MOM|0|0|BD|R|28.99
883929017164|MOM|0|0|BD|R|39.99
794051400123|RNO|73|0|BD|R|99.98
794051413727|RNO|8|0|BD|R|28.99
794051415622|RNO|84|0|BD|R|28.99
883929004676|RNO|0|0|BD|R|28.99
794051400123|SOM|25|0|BD|R|99.98
794051415622|SOM|80|0|BD|R|28.99
883929004676|SOM|0|0|BD|R|28.99
883929017164|SOM|0|0|BD|R|39.99
.................................
i need concate all duplicate line like
794051400123|COM|24|MOM|62|SOM|25|RNO|73
794051413727|COM|11||MOM|4|RNO|8
............
...
the file size is nearly 30 MB.So it takes lot of time.I have to do it with in 15 min.
please help me.
Thanks!
vaskar
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
i would like to scan file in for duplicate lines, and print the duplicates to another file,
oh and it has to be case insensitive.
example
line1
line2
line2
line3
line4
line4
outputfile:
line2
line4
any ideas (5 Replies)
Discussion started by: nixguy
5 Replies
2. Shell Programming and Scripting
Hi, I have a noob question . Can someone help me how to concatenate line by line using this variables?
var1:
Apple|
Banana|
var2:
Red
Yellow
then how can I concatenate both line by line? in which the result would be:
Apple|Red
Banana|Yellow
just to generate a row result i was... (6 Replies)
Discussion started by: hagdanan
6 Replies
3. Shell Programming and Scripting
Hi guys, I'm really happy to find this forum
I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated?
I already... (2 Replies)
Discussion started by: wax_light
2 Replies
4. Shell Programming and Scripting
I have a file a.txt having content like
deepak
ram
sham
deepram
sita
kumar
I Want to delete the first line containing "deep" ...
I tried using...
grep -i 'deep' a.txt
It gives me 2 rows...I want to delete the first one..
+ need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies
5. Shell Programming and Scripting
I have several hundreds of tiny files which need to be concatenated into one single line and all those in a single file. Some files have several blank lines. Tried to use this script but failed on it.
awk 'END { print r } r && !/^/ { print FILENAME, r; r = "" }{ r = r ? r $0 : $0 }' *.txt... (8 Replies)
Discussion started by: sdf
8 Replies
6. Shell Programming and Scripting
I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size).
Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies
7. Shell Programming and Scripting
Hello to all,
I'm new to perl, I have input file that contains the string below:
315350535ff450000014534130101ff4500ff45453779ff450ff45545f01ff45ff453245341ff4500000545000This string has as line separator "ff45". So, I want to print each line but the code below is not working.
perl -pe '... (2 Replies)
Discussion started by: Ophiuchus
2 Replies
8. Shell Programming and Scripting
I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code:
awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies
9. Shell Programming and Scripting
Hi,
Do anybody experience how to concatenate multiple line into one line by using awk or perl command?
Input file:
>set1
QAWEQRQ@EWQEASED
ASDAEQW
QAWEQRQTQ
ASRFQWRGWQ
From the above Input file, it got 5 lines
Desired output file:
>set1... (6 Replies)
Discussion started by: perl_beginner
6 Replies
10. UNIX for Dummies Questions & Answers
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
LEARN ABOUT DEBIAN
pbsnodes
pbsnodes(8B) PBS pbsnodes(8B)
NAME
pbsnodes - pbs node manipulation
SYNOPSIS
pbsnodes [-{a|x}] [-q] [-s server] [node|:property]
pbsnodes -l [-q] [-s server] [state] [nodename|:property ...]
pbsnodes [-{c|d|o|r}] [-q] [-s server] [-n] [-N "note"] [node|:property]
DESCRIPTION
The pbsnodes command is used to mark nodes down, free or offline. It can also be used to list nodes and their state. Node information is
obtained by sending a request to the PBS job server. Sets of nodes can be operated on at once by specifying a node property prefixed by a
colon.
Nodes do not exist in a single state, but actually have a set of states. For example, a node can be simultaneously "busy" and "offline".
The "free" state is the absence of all other states and so is never combined with other states.
In order to execute pbsnodes with other than the -a or -l options, the user must have PBS Manager or Operator privilege.
OPTIONS
-a All attributes of a node or all nodes are listed. This is the default if no flag is given.
-x Same as -a, but the output has an XML-like format.
-c Clear OFFLINE from listed nodes.
-d Print MOM diagnosis on the listed nodes. Not yet implemented. Use momctl instead.
-o Add the OFFLINE state. This is different from being marked DOWN. OFFLINE prevents new jobs from running on the specified
nodes. This gives the administrator a tool to hold a node out of service without changing anything else. The OFFLINE state
will never be set or cleared automatically by pbs_server; it is purely for the manager or operator.
-p Purge the node record from pbs_server. Not yet implemented.
-r Reset the listed nodes by clearing OFFLINE and adding DOWN state. pbs_server will ping the node and, if they communicate
correctly, free the node.
-l List node names and their state. If no state is specified, only nodes in the DOWN, OFFLINE, or UNKNOWN states are listed.
Specifying a state string acts as an output filter. Valid state strings are "free", "offline", "down", "reserve", "job-
exclusive", "job-sharing", "busy", "time-shared", or "state-unknown".
-N Specify a "note" attribute. This allows an administrator to add an arbitrary annotation to the listed nodes. To clear a
note, use -N "" or -N n.
-n Show the "note" attribute for nodes that are DOWN, OFFLINE, or UNKNOWN. This option requires -l.
-q Supress all error messages.
-s Specify the PBS server's hostname or IP address.
SEE ALSO
pbs_server(8B) and the PBS External Reference Specification
Local pbsnodes(8B)