The first part:
reads the first line from file into the shell variable A (read A) and the second line from file into the shell variable B (read B), reprints those two lines unchanged (printf '%s\n' "$A" "$B" and sorts the remaining lines from file (sort). This sorts the data in file while keeping the headers (unsorted) at the start of the output.
All of the output from the first part is then piped (|) into the second part:
which creates an array indexed by the first field on each input line (T[$1]) and sets the value of that array to the number of times that index has been seen (after using the previous value of the variable). In awk, statements consist of a condition section and an action section. In this case the condition is !T[$1]++ and there is no action section. When there is no action section, the default action (print the line) is performed if, and only if, the condition evaluates to a non-zero numeric value or a non-empty string value. Since undefined elements of an array are treated as a zero or an empty string (depending on context), the condition !T[$1]++ is evaluated as follows the first time a line is seen with a given string in the first field:
T[$1] is seen as a zero value
!T[$1] yields a value of 1 for the condition
T[$1]++ increments the value of the array element to 1
and since the condition's value is non-zero, the line is printed. On subsequent lines with the same first field the evaluation is:
T[$1] is seen as a non-zero value (a count of the number of times this value has been seen before)
!T[$1] yields a value of 0 for the condition
T[$1]++ increments the value of the array element
and since the condition's value is zero, the line is not printed.
Note that the sort in the first part grouped all lines with the same 1st field together and (if both an "add" and a "delete" appeared for that value, the line(s) with "add" for that 1st field value will come before any line(s) with "delete".
Nice, simple, straight-forward magic. The only likely problem with this approach is that if the data in your file contains a 1st field value that is identical to the 1st field in one of the header lines, those data lines will be lost. But, as long as you don't have a user named "user", that shouldn't be a problem for you.
These 2 Users Gave Thanks to Don Cragun For This Post:
Hello,
I have what is probably a simple task in text manipulation, but I just can't wrap my brain around it.
I have a text file that looks something like the following. Note that some have middle initials in the first field and some don't.
john.r.smith:john.smith@yahoo.com... (4 Replies)
Hello,
say suppose i am processing an file emp.dat the field of which are
deptno empno empname etc
now say suppose i want to change the file to emp.lst then how can i do it? Here i what i attempted but in vain
BEGIN{
system("sort emp.dat > emp.lst")
FILENAME="emp.lst"
}
{
print... (2 Replies)
Hello,
I'm running a script on AIX to process lines in a file. I need to enclose the second column in quotation marks and write each line to a new file. I've come up with the following:
#!/bin/ksh
filename=$1
exec >> $filename.new
cat $filename | while read LINE
do
echo $LINE | awk... (2 Replies)
I am trying to process file which has following data
#23456789012345
ACNASPSA13N0N0
ACNAPCPA05N0N0
ACNAFATS11N0N0
I want to take out each line from the file and what to put in the file by name which if part of the line starting from offset 10 to 15. It means I want to create three file... (3 Replies)
I have a 3-column data file, for which I wish to print certain parts of $3
PHI PSI A(x)
-177.5 -177.5 1.0625
-177.5 -172.5 0.55
-177.5 -167.5 0.0478125
-177.5 -162.5 0
-177.5 -157.5 0.284375
-177.5 -152.5 0.187188
-177.5 -147.5 0.236875
-177.5 -142.5 0.383438
-177.5 ... (3 Replies)
Input File:
1234, 2345,abc
1,24141,gw
222,rff,sds
2232145,sdsd,121
Output file to be generated:
000001234,2345,abc
000000001,24141,gw
000000222,rff,sds
002232145,sdsd,121
i.e; the first column is padded to get 9 digits.
I tried with following: (1 Reply)
Input File:
1234, 2345,abc
1,24141,gw
222,rff,sds
2232145,sdsd,121
Output file to be generated:
000001234,2345,abc
000000001,24141,gw
000000222,rff,sds
002232145,sdsd,121
i.e; the first column is padded to get 9 digits.
I tried with following: (3 Replies)
Hello,
I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this :
This is the output of ls command : I stored the output in a file filelist
1.1M... (5 Replies)
I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this?
... (12 Replies)
Hi - I want to interrogate information about my poker hands, sessions are all recorded in a text file in a particular format. Each hand starts with the string <PokerStars> followed by a unique hand reference and other data like date/time. There is then all the information about each hand. My first... (5 Replies)