I am not sure what i am doing wrong with your code but it does not seem to work like your example.
Code:
awk -F '\t' '
NR==FNR {id[$0]
next
}
{for (SP in id) if ($0 ~ SP)
{print > "'"$PAD/removed_woord.csv"'" #positive
next
}
}
{print > "'"$PAD/raw1.csv"'" #negative
}
' filters/woord_COM.csv $PAD/raw.csv
i get the positive file completly filled with all lines even though it should only add several in there while the negative file does not even get a single line. I added negative and positive in above code to should which is which.
I just noticed i had 2 small errors in my copied code. I removed them but the same problem is still present. the positive file with all the matches contains all lines and the negative one is not even created.
---------- Post updated at 04:32 PM ---------- Previous update was at 12:59 PM ----------
I did some more fiddling and i think i know where the error is and how to solve it.
I took the original code:
Code:
awk -F ';' ' NR==FNR {id[$0] next } {for (SP in id) if ($0 ~ SP) {print > "Positive" next } } {print > "Negative" } ' file1 file2
the only thing i changed on this was the names of file1 and file2. To be sure i used the example data i provided earlier. When i executed it the code worked fine and created both files.
Then i changed file1 to be the original filter file. Now it did not work anymore. I only recieved the Positive file with all lines. To be sure i tested it the other way around also but this worked fine. So i found out the problem was at the filter file. After some thinking i remembered that i copied over the data straight from a excel file and when i used notepad as in between it solved the problem.
So with a new filter file it works fine. Thank you for helping.
The desired two-filter filtering can be done in one go. Does your data file have space field sparators or e.g. <TAB> separators?
That is the annoying part. Some actually have TAB and some have semicolon etc. What is strange though is that for another file where i was testing i tried setting the delimiter to ; but it just ignored the whole code and threw all in the positive file. After i changed it to TAB en also made the delimiter TAB it worked just fine.
It is a bit annoying but i can work around that for at least the time untill i actually start to understand the code some more. I would even go as far as saying that the current code could be done in about half the lines but at least i understand what it does atm
I am always open for updated code if you would be happen to know it but it is no biggy to do it in 2 runs .
If you provide samples that match the requirements (word filter, EAN filter, and data) and post possible results, we could give it a go.
Well to make it actually understandable i will post the whole file although it has been shortened a bit.
Code:
## Download the new list
curl -o $PAD/pricing.csv http://website/pricing.csv
## Backup original file
cp $PAD/pricing.csv $ARCHIEF/origineel.$TIJDDATUM.csv
## Remove all lines with 7% in colomn 6
awk -F"\t" '$6 != "7.00"' $PAD/pricing.csv > $PAD/raw.csv
## Put all lines that are in the word_filter_file in a seperated file and the rest in the raw2 for further processing
awk -F"\t" '
NR==FNR {id[$0]
next
}
{for (SP in id) if ($0 ~ SP) {print > "'"$PAD/removed_woord.csv"'"
next
}
}
{print > "'"$PAD/raw1.csv"'"
}
' $VOEG/filters/woord_COM.csv $PAD/raw.csv
## Remove all colomns except for the ones we need.
cut -f1,4,7-10 $PAD/raw1.csv > $PAD/raw2.csv
## Put all columns in the correct order and use TAB as delimiter
awk 'BEGIN { FS="\t"; OFS="\t"; } {print $4,$1,$5,$6,$2,$3}' $PAD/raw2.csv > $PAD/raw3.csv
## Remove all lines with a blank first column
awk '!/^\t/' $PAD/raw3.csv > $PAD/raw4.csv
## Remove all > in the complete file
sed 's/>//g' $PAD/raw4.csv > $PAD/raw5.csv
## Remove the first line
awk '{if (NR!=1) {print}}' $PAD/raw5.csv > $PAD/raw6.csv
## Limit the chars behind the . (only for column 5 needed)
awk 'BEGIN { FS="\t"; OFS="\t"; } {printf("%s\t%s\t%s\t%s\t%.2f\t%s\n",$1,$2,$3,$4,$5,$6)}' $PAD/raw6.csv > $PAD/raw7.csv
## Make the delimiter ;
awk 'BEGIN { FS="\t"; OFS=";"; } {print $1,$2,$3,$4,$5,$6}' $PAD/raw7.csv > $PAD/raw8.csv
## Put all lines that are in the ean_filter_file in a seperated file and the rest in the raw9 for further processing
awk 'FNR==NR{A[$1]=$1;next} ($1 in A){print >> "'"$PAD/removed_EAN.csv"'"} !($1 in A){print >> "'"$PAD/raw9.csv"'"}' $VOEG/filters/niet_gebruiken_ean.csv FS=";" $PAD/raw8.csv
## Remove line if column 5 contains -
awk -F';' '!($5 ~ "-")' $PAD/raw8.csv > $PAD/raw9.csv
## Add code as last column
awk '$0=$0" ;CMT"' $PAD/raw9.csv > $PAD/clean.csv
## Create backup of the cleaned file
cp $PAD/clean.csv $ARCHIEF/clean.$TIJDDATUM.csv
As mentioned before the code works fine but it is probarly not the best code.
Main sample data (the original file is far bigger)
All files are snippets of the full files as they are too big to put them here. With the code from above it should be understandable what happens.
Each filter file gives a seperate file as output and a raw one which continues the processing. At the end i have a tar command which packs those files together (not the raw ones) and sends them per email and also uploads them to the correct server.
The only real disadvantage that i have is that every supplier has its own layout but this above is one of the longest scripts for this. Most others are smaller but not less important. I know most of the code what i copied here so i should be able to make it work for those if needed.
Not sure if the fields selected and their order is correctly inferred; an output sample to compare to is missing. removed_woord is empty as all eligible lines were already discarded by the 7.00% VAT criterion.
I added the main code so you could see the output because of the double filters but this just butchered my code (in a good way that is).
Even though i am extremly gratefull for the complete rewrite of the code, i am not sure if i will/can use it since i cannot read it completly yet. The main purpose was mainly to add both filters together :P and to create 3 files (2 filters and 1 raw).
I will definatly check it out some more and see if i can actually understand some more of it to change the code to suit the other files also.
In the awk piped to sed below I am trying to format file by removing the odd xxxx_digits and whitespace after, then move the even xxxx_digit to the line above it and add a space between them. There may be multiple lines in file but they are in the same format. The Filename_ID line is the last line... (4 Replies)
In the awk below I am trying to remove all lines above and including the pattern Test or Test2. Each block is seperated by a newline and Test2 also appears in the lines to keep but it will always have additional text after it. The Test to remove will not. The awk executed until the || was added... (2 Replies)
Hi,
I'd be grateful for your help with the following. I have a file (file.txt) with 10 columns and about half a million lines, which in simplified form looks like this:
ID Col1 Col2 Col3....
a 4 2 8
b 5 6 1
c 8 4 1
d... (4 Replies)
I am trying to remove each line in which $2 is FP or RFP. I believe the below will remove one instance but not both. Thank you :).
file
12
123 FP
11
10 RFP
awk
awk -F'\t' '
$2 != "FP"' file
desired output
12
11 (6 Replies)
I am trying to remove lines in the target.txt file if $5 before the - in that file matches sorted_list. I have tried grep and awk. Thank you :).
grep
grep -v -F -f targets.bed sort_list
grep -vFf sort_list targets
awk
awk -F, '
> FILENAME == ARGV {to_remove=1; next}
> ! ($5 in... (2 Replies)
I have a file like this - I want to remove the 2015 (or any four digit #) from column $4 so I can get:
Nov 05 1997 /ifs/inventory2/
for example. Im not sure how. Should I use an if statement with awk?
Jan 16 2015 23:45 /ifs/sql_file
Jan 16 2015 23:45 /ifs/sql_file
Nov 05 2015 1997... (4 Replies)
I need to send email to receipient in each block of data in a file which has the sender address under TO and just send that block of data where it ends as COMPANY.
I tried to work this out by getting line numbers of the string HELLO but unable to grab the next block of data to send the next... (5 Replies)
Hi,
I need to get specific parts in a large file.
I need to:
Get a line containing an IP address, and read from there to another line saying ***SNMP-END***
So, I have the start and the end well defined, but the problem is that apparently the awk command using the -F option doesn't work... (17 Replies)
hi,,
i hav a file with many lines.i need to remove all lines before a line begginning with a specific pattern from the file because these lines are not required.
Can u help me out with either a perl script or shell script
example:-
if file initially contains lines:
a
b
c
d
.1.2
d
e
f... (2 Replies)
I want to search a file for a string and then if the string is found I need the line that the string is on - but also the previous two lines from the file (that the pattern will not be found in)
This is on solaris
Can you help? (2 Replies)
02-11-2016
.