Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Conditional awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Conditional awk
# 1  
Old 07-17-2013
Conditional awk
Hello All,

I have a file like this:

Code:
bash-3.00$ cat 1.txt 
201112091147|0|1359331220|1025
201112091147|0|1359331088|1024
201112091144|0|1359331172|1025
201112091147|0|1359331220|1021
201112091149|0|1359331088|1027
201112091144|0|1359331172|1029

and a list of MSISDNs in another file where there is only one column which is MSISDN:

Code:
bash-3.00$ cat 2.txt 
201112091149
201112091140

I need to print out the records from first file 1.txt where MSISDN from second file 2.txt equals to the first column of 1.txt. so output should like this (at the end there must be filename too):


Code:
201112091149|0|1359331088|1027,1.txt


I have tried something like this but could not finalise and make it work, appreciate any help to write a report with the msisdns and the file names which they are in.

Code:
nawk 'FNR=NR { a[$1] = $1; next }{ printf ("%s\t%s\n", $0, FILENAME)}' 1.txt 2.txt

KR.
# 2  
Old 07-17-2013
Code:
nawk -F\| 'NR==FNR{A[$1];next}$1 in A{print $0,FILENAME}' OFS=, 2.txt 1.txt

This User Gave Thanks to Yoda For This Post:
EAGL€
# 3  
Old 07-17-2013
Code:
awk -F \| 'NR==FNR{A[$0]++;next}{if(A[$1]){print $0","FILENAME}}' 2.txt 1.txt

This User Gave Thanks to pamu For This Post:
EAGL€
# 4  
Old 07-17-2013
thanks,

if i need to check if MSISDN column of 1.txt ($1] has correct length which must have 12 Numbers then should i use length($1) function with a if sentence?

Besides please advise how fast this one liner will be when there are tousands of files like 1.txt and tousands of rows in each file?
# 5  
Old 07-17-2013
Yes, if(length($1) == 2) { ... } ought to work.

awk is designed to process big flatfiles, it should do well here. Thousands of files of thousands of lines comes to millions of lines, which is not unfeasible.

There's always ways to make things faster, of course. Will you be doing lots of things like
Code:
awk '{...}' file1 file2 > output
awk '{...}' file1 file3 >> output
awk '{...}' file1 file4 >> output

If so, you could do awk '{...}' file1 file2 file3 file4 ... > output
...which would save a lot of time since file1 wouldn't need to be read thousands of times, just once.

If you need different files for each one that's still possible, but would need changes to the code.
Last edited by Corona688; 07-17-2013 at 12:37 PM..
# 6  
Old 07-17-2013
Quote:
Originally Posted by Corona688
Yes, if(length($1) == 2) { ... } ought to work.

awk is designed to process big flatfiles, it should do well here. Thousands of files of thousands of lines comes to millions of lines, which is not unfeasible.

There's always ways to make things faster, of course. Will you be doing lots of things like
Code:
awk '{...}' file1 file2 > output
awk '{...}' file1 file3 >> output
awk '{...}' file1 file4 >> output

If so, you could do awk '{...}' file1 file2 file3 file4 ... > output
...which would save a lot of time since file1 wouldn't need to be read thousands of times, just once.

If you need different files for each one that's still possible, but would need changes to the code.

Hello Corona,

First of all thanks a lot,
Let me make the description of files spesific :
Code:
file 1.txt will have lets say 2000 rows, and there will be a few tousands of files like it. Besides file 2.txt is just one file with only one column but with 10.000 lines.

So would it work well? or should i divide file 2.txt to lets say 100 files and in a for-do loop check things?
# 7  
Old 07-17-2013
Splitting into more or less files wouldn't change the actual amount of work. If anything, bigger files and fewer loops would be more efficient -- awk is at its most efficient when you give it a big job and let it do its thing, instead of restarting it for tiny jobs all the time.

It's not like it's keeping those 10,000 lines around in memory anyway, nothing's getting overloaded. (And even if it were, 10,000 lines is not a lot for awk!)

Your specific description doesn't tell me much about anything except how big your files are, you haven't told me what you will be doing with them.
Last edited by Corona688; 07-17-2013 at 01:14 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Conditional Arithmetic in [g]awk

I am having a difficult time getting an awk one-liner to work correctly that runs a mathematical operation upon values in a field when matching a given criteria. I would like to subtract 1 from every value in field $6 that is greater than 12. In this particular case it is only a constant of... (3 Replies)
Discussion started by: jvoot
3 Replies

2. Shell Programming and Scripting

[awk] conditional printing

Heya I'm trying to get to know awk a bit better. So i'm trying to get used to calls saving me a grep invocation just to get a specific part of a single line. This said, i want to get the current screen resolution according to xrandr's output. Screen 0: minimum 8 x 8, current 1920 x 1080,... (1 Reply)
Discussion started by: sea
1 Replies

3. Shell Programming and Scripting

awk conditional output

Hello, How can I use a conditional to produce an output file that varies with respect to the contents of column #4 in the data file: Data file: 9780020080954 9.95 0.49 AS 23.3729 9780020130857 9.95 0.49 AS 23.3729 9780023001406 22.20 0.25 AOD ... (12 Replies)
Discussion started by: palex
12 Replies

4. Shell Programming and Scripting

awk conditional find

Hi, I have a file in the following format: aabbba 25.31806899 baaabb 38.21808852 cccccu 1.31819523 552258121.31818253 ffddybb 5.41815555 almcamc87561812689 223aqas5.661828345 adacaaaaaaa1821285 adacaaaaaaa1821286 smckaa 3.81828756 ada2512510c1821287 ada2522511c1821328... (4 Replies)
Discussion started by: alex2005
4 Replies

5. Shell Programming and Scripting

conditional statement in awk

Hi all, I have a file containing the values that would be use as the basis for printing the lines of another set of files using awk. What I want to do is something like the one below: stdev.txt 0.21 0.42 0.32 0.25 0.15 file1.txt file2.txt file3.txt ..filen.txt 0.45 0.23 ... (4 Replies)
Discussion started by: ida1215
4 Replies

6. Shell Programming and Scripting

AWK conditional addition

I have a column of numbers $2, I would like to add 360 to all numbers that are negative. This method seems a bit convoluted, and does not work (outputs 0): BEGIN { A=sprintf("%d", $2); if(A<0) A=A+360; BIN++; } END { for(A in BIN) print... (5 Replies)
Discussion started by: chrisjorg
5 Replies

7. Shell Programming and Scripting

awk conditional assignment

Dear all, I want to use awk to read the three columns in a file called "test" and change them to ( 1.5 1.5 1.5) if any element is found to be greater than three. A part of the file is shown below: (0.478318 0.391032 -0.14054) (0.45562 0.392523 -0.121685) (0.437204 0.392811 -0.106158)... (3 Replies)
Discussion started by: mfmohdyasin
3 Replies

8. Shell Programming and Scripting

Awk Conditional

Hi Guys, i have this files: xyz20080716.log opqrs20080716.log abcdef20080716.log xyz20080717.log oprs20080717.log abcde20080717.log currentdate: 20080717.log I want to make script to zip the file for past day. Can anyone help for this? i've just learn awk scripting & still confused with... (3 Replies)
Discussion started by: icy_blu_blu
3 Replies

9. Shell Programming and Scripting

AWK - conditional cause

Hello guys, I want to make a conditional cause in the following file using awk: awk '{ if ($2 != 0) print $1, $2, $3}' test.csv > test2.csv FILE EXAMPLE = test.csv string,number,date abc,0,20050101 def,1,20060101 ghi,2,20040101 jkl,12,20090101 mno,123,20020101 ... (2 Replies)
Discussion started by: Rafael.Buria
2 Replies

10. Shell Programming and Scripting

awk conditional statement

how can i use awk or sed to do a conditional statement, so that HH:MM if MM not great than 30 , then MM=00 else MM=30 ie: 10:34 will display 10:30 10:29 will display 10:00 a=$(echo 10:34 | awk ......) Thanks in advance (10 Replies)
Discussion started by: 3Gmobile
10 Replies
Login or Register to Ask a Question