Take always the last x fields


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Take always the last x fields
# 1  
Old 07-15-2010
Take always the last x fields

Hi,

I have a bash script that converts .csv files to sql inserts and so far has worked great. It takes the values and does it's magic.
The thing is that now the .csv files have fields that I don't need, and they are always in the beggining of the line. For example:
One .cvs will have all the lines like:
Code:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10

And another one wil have:
Code:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

And so on, but I will always only need the last 5 fields and discard the rest.

Haw can I achieve this?

Thank you!


PS: This is my current full script
Code:
#!/bin/bash


################################
# Extensión a buscar
EXT="*.csv"

# Ruta completa hacia la carpeta de origen Lotes
IN="/root/Script/In/"

# Ruta completa hacia la carpeta de destino Lotes
OUT="/root/Script/Out/"

# Ruta completa hacia la carpeta de destino SQLs
SQL="/root/Script/SQLs/"

# Ruta completa hacia la carpeta de logs
LOG="/root/Script/Logs/procesador.log"

# Tiempo de espera entre búsquedas en segundos
SEC="3"

################################
Tomar_datos()
{
local tmp=`mktemp temp.uno`
cat $nombre > $2
cat $2 > $tmp
mv $tmp $2
}
################################


################################
Reemplazar_separador()
{
local cnt=`wc -l $nombre | cut -f1 -d" "`
local tmp=`mktemp temp.dos`

cnt=$((cnt - 1))
tail -n $cnt temp.datos > $tmp
mv $tmp temp.datos
}
################################

################################
Armar_inserts()
{
local tmp=`mktemp temp.tres`
cat > $tmp <<-EOF
{print "INSERT INTO $2 (APP_ID, APP_ISS_ID, APPCRD_TSN, CRD_SNR,CRD_INTSNR, DATAFILE_ID,DEV_ID, DEV_STAT, DEV_TSN,DEV_TYPE, EMPL_ID, EMPL_TYPE, ERROR_CODE, ERROR_INTCODE,TRX_FAREVALUE, GROUP_FNAME, GROUP_ID, LINE_ID, PLACE_ID, PROVIDER_ID, PROVIDER_ROLE, PURSE_AMOUNT, PURSE_ID, TRX_ID,TRX_DATETIME, REG_SEQ, CARDNUMBER, ID_LOTE, FLG_MULTPURSE, CARD_BALANCE, FARE_RULE) values(" \$1 ", "\$2 ", "\$3 ", "\$4 ", "\$5 ", "\$6 ", "\$7 ", "\$8 ", "\$9 ", "\$10 ", "\$11 ", "\$12 ", "\$13 ", "\$14 ", "\$15 ", '"\$16 "', "\$17 ", "\$18 ", "\$19 ", "\$20 ", "\$21 ", "\$22 ", "\$23 ", "\$25 ", TO_DATE( '"\$26 ":segundos', 'DD/MM/YYYY HH24:MI:SS'), "\$28 ", "\$29 ", "\$33 ", "\$27 ", "\$24 ", "\$15 "); "}
EOF
cat temp.datos | awk -F, -f $tmp > "$nombre.sql"
rm $tmp
echo "" >> "$nombre.sql"
echo "commit;" >> "$nombre.sql"
}
################################


################################
main()
{
# Copiando registros
local tmp=`mktemp temp.datos`

# Tomando registros
Tomar_datos $nombre $tmp

# Reemplazando separador de registros
Reemplazar_separador $tmp
#sed -i 's/,/./g' temp.datos #Reemplazando separador decimal
sed -i 's/;/,/g' temp.datos

# Armando los inserts
Armar_inserts $tmp $2
rm $tmp
#rm $nombre.sql
}
################################


################################
Tomar_lote ()
{
if [ "$nombre" == "*.csv" ]; then
   unset nombre
   echo "Nada para procesar. Volviendo a buscar en $SEC segundos." >> $LOG
   sleep $SEC
else
   FILE="$nombre"
   # Asegurarse que el archivo exista
   if [ ! -f $FILE ]; then 
	echo "Nada para procesar. Volviendo a buscar en $SEC segundos." >> $LOG
  	sleep 3
	Tomar_lote "$@"
   elif [ ! -r $FILE ]; then
   echo fail 
   fi
   START=$(date +%s)
   echo Procesando Lote $nombre >> $LOG
   main "$@"
   mv $nombre $OUT #Mueve el lote a la carpeta de destino
   mv $nombre.sql $SQL #Mueve el sql a la carpeta de destino
   echo Lote $nombre procesado en $(expr $(date +%s) - $START) segundos  >> $LOG
fi
}
################################

cd $IN
while [ 1 = 1 ]
do
	for nombre in $EXT; 
	do Tomar_lote "$@"
	done
	date echo "Nada para procesar. Volviendo a buscar en $SEC segundos." >> $LOG
	sleep $SEC
done

# 2  
Old 07-15-2010
Quote:
Originally Posted by Tr0cken
I will always only need the last 5 fields and discard the rest.
Something like this?
Code:
awk -F", " '{for(i=NF-4;i<NF;i++){printf("%s, ", $i)}print $NF}' file

# 3  
Old 07-15-2010
Quote:
Originally Posted by Tr0cken
Code:
1, 2, 3, 4, 5, 6, 7, 8, 9, 10
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

but I will always only need the last 5 fields and discard the rest
Code:
[house@leonov] cat data.file | awk -F ", " '{print $(($NF-4)),$(($NF-3)),$(($NF-2)),$(($NF-1)),$NF}'
6 7 8 9 10
9 10 11 12 13

# 4  
Old 07-15-2010
Quote:
Originally Posted by Franklin52
Something like this?
Code:
awk -F", " '{for(i=NF-4;i<NF;i++){printf("%s, ", $i)}print $NF}' file

This did the trick.

Thank you very much!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Is there a UNIX command that can compare fields of files with differing number of fields?

Hi, Below are the sample files. x.txt is from an Excel file that is a list of users from Windows and y.txt is a list of database account. $ head -500 x.txt y.txt ==> x.txt <== TEST01 APP_USER_PROFILE USER03 APP_USER_PROFILE TEST02 APP_USER_EXP_PROFILE TEST04 APP_USER_PROFILE USER01 ... (3 Replies)
Discussion started by: newbie_01
3 Replies

2. Shell Programming and Scripting

Print . in blank fields to prevent fields from shifting

The below code works great, kindly provided by @Don Cragun, the lines in bold print the current output. Since some of the fields printed can be blank some of the fields are shifted. I can not seem too add . to the blank fields like in the desired output. Basically, if there is nothing in the field... (10 Replies)
Discussion started by: cmccabe
10 Replies

3. Shell Programming and Scripting

awk sort based on difference of fields and print all fields

Hi I have a file as below <field1> <field2> <field3> ... <field_num1> <field_num2> Trying to sort based on difference of <field_num1> and <field_num2> in desceding order and print all fields. I tried this and it doesn't sort on the difference field .. Appreciate your help. cat... (9 Replies)
Discussion started by: newstart
9 Replies

4. Shell Programming and Scripting

awk - compare 1st 15 fields of record with 20 fields

I'm trying to compare 2 files for differences in a selct number of fields. When differnces are found it will write the whole record of the second file including appending '|C' out to a delta file. Each record will have 20 fields, but only want to do comparison of 1st 15 fields. The 1st field of... (7 Replies)
Discussion started by: sljnk
7 Replies

5. Shell Programming and Scripting

How to print 1st field and last 2 fields together and the rest of the fields after it using awk?

Hi experts, I need to print the first field first then last two fields should come next and then i need to print rest of the fields. Input : a1,abc,jsd,fhf,fkk,b1,b2 a2,acb,dfg,ghj,b3,c4 a3,djf,wdjg,fkg,dff,ggk,d4,d5 Expected output: a1,b1,b2,abc,jsd,fhf,fkk... (6 Replies)
Discussion started by: 100bees
6 Replies

6. Shell Programming and Scripting

Join fields comparing 4 fields using awk

Hi All, I am looking for an awk script to do the following Join the fields together only if the first 4 fields are same. Can it be done with join function in awk?? a,b,c,d,8,,, a,b,c,d,,7,, a,b,c,d,,,9, a,b,p,e,8,,, a.b,p,e,,9,, a,b,p,z,,,,9 a,b,p,z,,8,, desired output: ... (1 Reply)
Discussion started by: aksijain
1 Replies

7. UNIX for Dummies Questions & Answers

Set variables from fields in fields

Hi, This is my first post here and I am a newbie. :) I have a file that looks like this : Introduction:Intro_123.html Product definition:Prod_def.html System Setup:SSetup-64bit.html Setting up user accounts:Set_user_acc.html I tried to create a script that would output "The filename... (3 Replies)
Discussion started by: Joq
3 Replies

8. Shell Programming and Scripting

Add fields in different files only if some fields between them match

Hi everybody (first time posting here) I have a file1 that looks like > 1,101,0.1,0.1 1,26,0.1,0.1 1,3,0.1,0.1 1,97,0.5,0.5 1,98,8.1,0.218919 1,99,6.2,0.248 2,101,0.1,0.1 2,24,3.1,0.147619 2,25,23.5,0.559524 2,26,34,0.723404with 762 lines.. I have another 'similar' file2 > ... (10 Replies)
Discussion started by: murpholinox
10 Replies

9. Shell Programming and Scripting

awk sed cut? to rearrange random number of fields into 3 fields

I'm working on formatting some attendance data to meet a vendors requirements to upload to their system. With some help on the forums here, I have the data close. But they've since changed what they want. The vendor wants me to submit three fields to them. Field 1 is the studentid field,... (4 Replies)
Discussion started by: axo959
4 Replies

10. Shell Programming and Scripting

combining fields in two text fields

Can someone tell me how to do this using sed, awk, or any other basic shell scripting? Basically I have two text files with the following contained in each file: File A: a b c d e f g h i File B: 1 2 3 I want the final outcome to look like this: a b c 1 d e f 2 g h i 3 How... (3 Replies)
Discussion started by: shocker
3 Replies
Login or Register to Ask a Question