Need advise/tip if there is more efficient way of doing this cut/paste/awk after changing a field


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Need advise/tip if there is more efficient way of doing this cut/paste/awk after changing a field
# 1  
Old 02-24-2020
Need advise/tip if there is more efficient way of doing this cut/paste/awk after changing a field

Hi,

This is the script currently and it is working as required. Just thought maybe there is a better or easier way of doing what I am trying to do.

Code:
$ cat x.ksh
#!/bin/ksh
#

cut -d"|" -f1 x.txt > x1.txt
cut -d"|" -f2 x.txt | awk -F"=" '{ print "USER="tolower($2) }' > x2.txt
cut -d"|" -f3- x.txt > x3.txt

paste -d "|" x1.txt x2.txt x3.txt | sort | uniq > x4.txt

cat x.txt
echo
cat x4.txt
echo

Below is an excerpt of the file that I want to change. This is x.txt, the original file that I want to run this on is about 1000+ lines. Basically, these files are from several log files merged into one and I am wanting to change the USER=<username> field so that <username> is in lower case. I am working on the assumption that USERNAME=<username> is always field2.


Code:
PROGRAM=JDBC Thin Client|USER=MICKEY|HOST=11.123.12.123|testmachine.xyz.com.zz
PROGRAM=JDBC Thin Client|USER=mickey|HOST=11.123.12.123|testmachine.xyz.com.zz

Sample run of the script below:


Code:
$ ./x.ksh
PROGRAM=JDBC Thin Client|USER=MICKEY|HOST=11.123.12.123|testmachine.xyz.com.zz
PROGRAM=JDBC Thin Client|USER=mickey|HOST=11.123.12.123|testmachine.xyz.com.zz

PROGRAM=JDBC Thin Client|USER=mickey|HOST=11.123.12.123|testmachine.xyz.com.zz

I could simply do
Code:
sort x.txt | tr [:upper:] [:lower:] | uniq

but for 'clarity' I prefer to only change USER=<username> to USER=<lowercase_username> and leave the rest of the line as it is. I can't work out the awk or sed command options to use to achieve what I wanted, hence I ended up with a shell script instead. Maybe there is an awk one-liner that can do what I am trying to achieve Smilie

Please advise. Thanks in advance.
# 2  
Old 02-24-2020
Try
Code:
$ awk -F\| '{split ($2, T, "="); $2 = T[1] "=" tolower(T[2])} !a[$0]++' OFS=\| file
PROGRAM=JDBC Thin Client|USER=mickey|HOST=11.123.12.123|testmachine.xyz.com.zz



It converts just the username to lower case, and prints out only the first occurrence of the resulting line. No sort done.
This User Gave Thanks to RudiC For This Post:
# 3  
Old 02-26-2020
Hi RudiC


Thanks the one liner works like a charm. I won't be able to figure out for ages that this will do what I want Smilie It doesn't even requires any intermediate files.

Code:
awk -F\| '{split ($2, T, "="); $2 = T[1] "=" tolower(T[2])} !a[$0]++' OFS=\| file

Trying real hard to understand what's happening though.

Code:
{split ($2, T, "=")

this splits $2 and assign it T I believe and then some more processing happens.

Then
Code:
$2 = T[1] "=" tolower(T[2])} !a[$0]++

change $2 to be
Code:
USER=<lower_case_of_username_string>

if it is not a[$0]? Is that what
Code:
!a[$0]++

means and this is the one that prevents the duplicates? Do I understand it correctly?
# 4  
Old 02-27-2020
Yes, $2 (USER=MICKEY) is split at the equals sign into array T , and then rebuilt with the lower case username in T[2].


The !a[$0]++ is a trick (independent of $2): a[$0] evaluates to FALSE if it equals zero or doesn't exist or is created on first reference, so its negation is TRUE and triggers the default action: print. Then it is "post incremented" and will never (OK, not until reaching / crossing MAX_INT) trigger again. So any further occurrences of $0 are suppressed.
# 5  
Old 02-28-2020
A more detailed explanation follows.
The main awk code runs for each input line.
!a[$0]++ is ultra-condensed, quick and dirty.
A bit more explicit is !($0 in A) { A[$0]; print }:
If not $0 in array A (A[$0] not defined) then define A[$0] (no A[$0]=value needed here) and print $0.
The array A is associative (string-addressed). So if the same $0 will occur in another input line it will see a defined A[$0] and won't print.
If there is a pre-condition and no { action code } following then the default for a true condition is { print }, and print without arguments defaults to print $0.

Now to the quick and dirty !A[$0]++:
Define A[$0] with value 0 if undefined, if the negated value is non-zero (true) then default-print. Also post-increment A[$0].
If the same $0 will occur then the A[$0] value will be 1, negated 0 (false), won't print, but post-incremented.
If the same $0 will occur then the A[$0] value will be 2, negated 0 (false), won't print, but post-incremented.
...
This User Gave Thanks to MadeInGermany For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to cut the last field without using awk

i have file as with the below content aaa.bbb.cc.dd aaa.fff.bb yyyyy.rrrrr.ggggg.iii wwww.w.r.ty i want the o/p as below dd bb iii ty but i dont want to use awk. is there any other way to do this ? (5 Replies)
Discussion started by: anandgodse
5 Replies

2. Shell Programming and Scripting

need help with cut and paste command

I have a file which contains 3 fields separated by tabs example andrew kid baker I need to swap kid and baker using cut and paste commands how is this to be done? Thanks (3 Replies)
Discussion started by: drew211
3 Replies

3. UNIX for Advanced & Expert Users

Which cut command is more efficient?

Hi, I've got a query regarding which of the following is more efficient & why - cat <filename>|cut -d'*' -f2- > <newfilename> or cut -d'*' -f2- <filename> > <newfilename> Thanks. (17 Replies)
Discussion started by: sumoka
17 Replies

4. Shell Programming and Scripting

Using a combination of sort/cut/grep/awk/join/paste/sed

I have a file and need to only select users that have a shell of “/bin/bash” in the line using awk or sed please help (4 Replies)
Discussion started by: boyboy1212
4 Replies

5. Shell Programming and Scripting

awk,cut fields by change field format

Hi Everyone, # cat 1.txt 1321631,77770132976455,19,20091001011859,20091001011907 1321631,77770132976455,19,20091001011859,20091001011907 1321631,77770132976455,19,20091001011859,20091001011907 # cat 1.txt | awk -F, '{OFS=",";print $1,$3,$4,$5}' 1321631,19,20091001011859,20091001011907... (7 Replies)
Discussion started by: jimmy_y
7 Replies

6. Shell Programming and Scripting

cut and paste

Hi, Need a help with shell script. I have to search for a string in one of the file, if match found, copy the line to a new file and delete the line from the exisiting file. eg: 83510000000000063800000.1800000.1600000.1600000.2400000.1800000.2000000.21... (6 Replies)
Discussion started by: gpaulose
6 Replies

7. UNIX for Advanced & Expert Users

Printing Field with Delimiter in AWK/cut

Hello, I had posted earlier about printing fields using AWK, but now I have a slightly different problem. I have text files in the format: 1*2,3,4,5 and wish to print the first, third, and fifth fields, including the asterisk and commas. In other words, after filtering it should look... (1 Reply)
Discussion started by: Jahn
1 Replies

8. Shell Programming and Scripting

cut and paste?

hi, I have a file with content like this for an employee: EmployeeID 101 Day_type, day vacation,1/2/2009 sick day, 3/2/2009 personal day, 4/5/2009 jury duty day, 5/5/2009 how do I make the result to show: EmployeeID,Day_type,day 101,vacation,1/2/2009 101,sick day,... (6 Replies)
Discussion started by: jbchen
6 Replies

9. Shell Programming and Scripting

cut and paste using awk

Hi i need a favour i have a file which has some trillions of records. The file is like this 11111000000000192831840914000000000000000000000000000 45789899090000000000000000011111111111111111111111111 I want to cut specific postions in each line like cut1-3 and assisgn it to a variable and... (5 Replies)
Discussion started by: richa2.m
5 Replies

10. UNIX for Dummies Questions & Answers

cut and paste columns using awk

Hi, Let's say that I have a file called table, I know that if I need to see a the second column for exampls I use: awk ' {print $2}' table.txt Is there anyway to use awk to actually cut a column and put it somewhere else in the table?:confused: (8 Replies)
Discussion started by: cosmologist
8 Replies
Login or Register to Ask a Question