I am trying to remove the duplicate records from all the .txt files in a folder using the below script I got from the forum..the script is giving a error as do is not matching can any body tell me where I am doing it wrong.
Code:
find /root/neb/src/ -name '*.txt' | while read FILENAME
do
awk 'x[$0]++ == 2' < "$FILENAME" > "${FILENAME}.out"
Pls help me to understand the script $0 means we are considering the entire record I am not able to understand the x before the [ and the ++ signs pls tell me what they signify
--Mora
Last edited by Franklin52; 05-04-2011 at 09:42 AM..
Reason: Please use code tags
To get all the non duplicate records from .txt files
Code:
find /root/neb/src/ -name '*.txt' | while read FILENAME
do
awk 'x[$0]++{next} {print $0}' < "$FILENAME" > "${FILENAME}.out"
done
where ' x[$0]++ { next} ' stands for go to next record if the count is more than 1 ( which mean the record is already processed )
print $0 will print all the records ( provided the first condition is failed; which means it's not a duplicate entry )
To get ONLY the duplicate records from all .txt files
Code:
find /root/neb/src/ -name '*.txt' | while read FILENAME
do
awk 'x[$0]++' < "$FILENAME" > "${FILENAME}.out"
done
Hi,
I am trying to pass awk field to a command line executed within awk (need to convert a timestamp into formatted date).
All my attempts failed this far.
Here's an example.
It works fine with timestamp hard-codded into the command
echo "1381653229 something" |awk 'BEGIN{cmd="date -d... (4 Replies)
below is the output xml string from some other command and i will be parsing it using awk
cat /tmp/alerts.xml
<Alert id="10102" name="APP-DS-ds_ha-140018-componentFailure-S" alertDefinitionId="13982" resourceId="11427" ctime="1359453507621" fixed="false" reason="If Event/Log Level(ANY) and... (2 Replies)
Hi,
I have a situation to compare one file, say file1.txt with a set of files in directory.The directory contains more than 100 files.
To be more precise, the requirement is to compare the first field of file1.txt with the first field in all the files in the directory.The files in the... (10 Replies)
Hello all,
Here is what my bash script does: sums number columns, saves the tot in new column, outputs if tot >= threshold val:
> cat getnon0file.sh
#!/bin/bash
this="getnon0file.sh"
USAGE=$this"
InFile="xyz.38"
Min="0.05"
#
awk '{sum=0; for(n=2; n<=NF; n++){sum+=$n};... (4 Replies)
I have a script problem that I am not able to solve due my very limited understanding of unix/awk.
This is the contents of test.sh
awk '{print $1}'
From the prompt if I enter:
./test.sh Hello World
I would expect to see "Hello" but all I get is a blank line. Only then if I enter "Hello... (2 Replies)
Sorry for the duplicate thread this one is similar to the one in
https://www.unix.com/shell-programming-scripting/88132-awk-sed-script-read-values-parameter-files.html#post302255121
Since there were no responses on the parent thread since it got resolved partially i thought to open the new... (4 Replies)
Hi ,
I have a piece of code ...wherein I need to assign the following ...
1) A command line argument to a variable
e.g origCount=ARGV
2) A unix command to a variable
e.g result=`wc -l testFile.txt`
in my awk shell script
When I do this :
print "origCount" origCount --> I get the... (0 Replies)
