In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows.
The file name in the folder will look like e.q:
The content of each file will consist of a header (that I want to keep in the new big file, and a set of tabulator separated fields. In addition a last column should be added, where the date of the soruce file is added (either from the file name or from the header of the spesific files). Example (for only two of the files):
MissingData_2014-08-25_09-30-18.txt:
MissingData_2014-08-26_09-30-12.txt:
RESULTS IN ONE BIG FILE:
I've tried to make a script by using 'cat' to merge the files, but that doesn't help me avoiding duplicates or truncate the header and add dates after each row:
Last edited by Bergans; 09-04-2014 at 07:13 AM..
Reason: code tags for data samples
Hi
Just wondering whether or not I can remove duplicated lines without sort
For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal.
Normally, I would do
who | cut -d" " -f1 |... (6 Replies)
hi all,
i have a file contain multicolumns, this file is sorted by col2 and col3.
i want to remove the duplicated columns if the col2 and col3 are the same in another line.
example
fileA
AA BB CC DD
CC XX CC DD
BB CC ZZ FF
DD FF HH HH
the output is
AA BB CC DD
BB CC ZZ FF... (6 Replies)
Hi,
I am trying to use awk to remove all rows where the first 4 fields are duplicates. e.g. in the following data lines 6-9 would be removed, leaving one copy of the duplicated row (row 5)
Borgarhraun FH9822 ol24 FH9822_ol24_m20 ol Deformed c
Borgarhraun FH9822 ol24 ... (3 Replies)
Hi,
I have two files that I would like to merge and think that there should be a solution using awk. The files look something like this:
file 1
IDX1 IDY1
IDX2 IDY2
IDX3 IDY3
file 2
IDY1 dataA data1
IDY2 dataB data2
IDY3 dataC data3
Desired output
IDX1 IDY1 dataA data1
IDX2 ... (5 Replies)
Hi,
I would like to get some advice here. Basically I have 2 diff files, I need to add/comment FileA (old) contents as compared with FileB (new). The file has almost 100k lines so it would be great if someone can advise me on the easiest way out.
For example
FileB has this parameter but... (1 Reply)
Hi,
I have two files A (2190 rows) and file B (1100 rows). I want to merge the contents of two files based on common field, also I need the unmatched rows from file A
file A:
ABC
XYZ
PQR
file B:
>LMN|chr1:11000-12456:
>ABC|chr15:176578-187678:
>PQR|chr3:14567-15866:
output... (3 Replies)
Hello,
I am trying to eliminate rows where the first field is duplicated, leaving the row where the last field is "NET".
Data file:
345234|22.34|LST
546543|55.33|LST
793929|98.23|LST
793929|64.69|NET
149593|49.22|LST
Desired output:
345234|22.34|LST
546543|55.33|LST... (2 Replies)
Hello.
I have a server with 2 boot disk but in the bootlist there are 5 paths of one disk but no path of the other.
How can I remove paths from one disk to insert paths from the other disk?
Thanks in advance.
server074:root:/# bootlist -om normal
hdisk0 blv=hd5 pathid=0
hdisk0... (7 Replies)