Cvs manipule.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Cvs manipule.
# 1  
Old 09-25-2015
Cvs manipule.

Hello all,

What i need to do is manipulate a squid log file.
I have many milions of file in this format:
Code:
1442814667.478     76 4.3.2.1 TCP_MISS/200 31845 GET http://pippo.com/inde.html - DIRECT/1.2.3.4 text/css

What i need to do is transform field 7 from Pippo.com - 404 File Not Found in [url]http://pippo.com and write the new file like:
Code:
1442814667.478,76,4.3.2.1,TCP_MISS/200,31845,GET, http://pippo.com,-, DIRECT/1.2.3.4,text/css

I can convert file in cvs with awk but have no idea how to modify the field 7.
Can you please help me?
Thanks in advance a.k.a

---------- Post updated at 02:25 AM ---------- Previous update was at 02:13 AM ----------

what i have try is a silly shell code but it's damn slow Smilie
Code:
[root@site a]# cat log.log |while read a b c d e f g h i l;
do
PIPPO=`echo $g |cut -f1 -d"/"`;
PLUTO=`echo $a |cut -f1 -d"."`;
echo "$PLUTO,$b,$c,$d,$e,$f,$PIPPO,$h,$i,$l" >>sasha.csv
done
[root@site a]# -


Last edited by Don Cragun; 09-25-2015 at 04:19 AM.. Reason: Add CODE and ICODE tags.
# 2  
Old 09-25-2015
I don't understand.

Why is your output field separator sometimes a comma and sometimes a space and a comma? Why aren't all of the spaces removed from the output?

There is no "Pippo.com - 404 File Not Found" in your input file (in field 7 nor anywhere else). Are you just saying that you want to remove the last slash character (/) and everything that following from field 7?

Furthermore, your sample code removes the period and everything following it from the 1st field; but your desired output shows no change at all to field 1 AND your description of your problem says nothing about changing field 1.

Please be more clear in your explanation of what you are trying to do.
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 09-25-2015
Making some wild guesses, here are a couple of ways to do what you seem to want: one using an awk script and one just using shell built-ins (assuming that you are using a shell that performs basic POSIX shell parameter expansions):
Code:
#!/bin/ksh
awk -v OFS=, '
{	sub("[.][^.]*$", "", $1)
	sub("/[^/]*$", "", $7)
	$1=$1
}
1' log.log >>sasha.csv

while read a b c d e f g h i j
do	printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n' "${a%.*}" "$b" "$c" "$d" "$e" \
	    "$f" "${g%/*}" "$h" "$i" "$j"
done < log.log >> sasha.csv

As always, if you want to use awk on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.

With your sample input, both of these produce the output:
Code:
1442814667,76,4.3.2.1,TCP_MISS/200,31845,GET,http://pippo.com,-,DIRECT/1.2.3.4,text/css

# 4  
Old 09-25-2015
Quote:
Originally Posted by Don Cragun
I don't understand.

Why is your output field separator sometimes a comma and sometimes a space and a comma? Why aren't all of the spaces removed from the output?

There is no "Pippo.com - 404 File Not Found" in your input file (in field 7 nor anywhere else). Are you just saying that you want to remove the last slash character (/) and everything that following from field 7?

Furthermore, your sample code removes the period and everything following it from the 1st field; but your desired output shows no change at all to field 1 AND your description of your problem says nothing about changing field 1.

Please be more clear in your explanation of what you are trying to do.
Sorry, will try to explain it better.

1. what i have is a common squid log forrmat.
2. what i need is to load in a database this log file.

What is my flow:

1. convert log file separated by space in a CSV file.
2. trim the destination url field cause i'm not intersted in full url but only in destination, so http://www.blablabla.com/bla.html have to be transformed in
www.blablabla.com.
(btw, sorry i forget to uncheck "Automatically parse links in text")

What i say is that for me is simple create a CSV file, and with the shell code trim the file but shell code is very very slow. What i need is understand how to make a sed/awk script that will "clean" only the filed 7 (url)

Thank in advance and sorry for the poor and confudes information in first post.
# 5  
Old 09-25-2015
Did you look at post #3 in this thread? If shows you how to use awk and how to use a shell script to trim fields 1 & 7 in ways that should be considerably faster than your current shell script. It should be easy for you to remove a sub() statement from the awk script or remove a parameter expansion from the shell script if you don't want to modify field 1.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

CVS Automation

Hi All, I am just looking for CVS automation for SQL scripts. Normally Devs will check in new sql scripts or they will update the existing sql scripts with new query. We will take the scripts from CVS and run in DB. I am thinking to automate that process like whenever a new script is checked in... (1 Reply)
Discussion started by: pvmanikandan
1 Replies

2. AIX

Using cvs on AIX

Hello. I am using CVS on my AIX 6.1 box. I have gotten the repository setup and everything is working from command line. Now I want to use CVS on my Windows 7 machine using the remote repository on my AIX box. Has anyone else tried this? Why can't I get this to work? I have tried using... (3 Replies)
Discussion started by: hpodhrad
3 Replies

3. UNIX for Dummies Questions & Answers

Can you automate CVS?

Currently we have a load of files which we manually edit and then commit back into CVS ready for whoever else to edit. I have now made a script which auto-populates these files, however the powers that be still want them accessible via CVS. Is there a way I can automatically commit these files... (7 Replies)
Discussion started by: JayC89
7 Replies

4. Red Hat

CVS Configuration Help

Hello, I have read a great deal of documentation on CVS and I hope I have not overlooked what I need but I have certain issues with CVS that I cannnot resolve. The setup for the /etx/xinetd.d file I have is as follows: # default: off # description: The CVS service can record the history... (2 Replies)
Discussion started by: mojoman
2 Replies

5. Red Hat

CVS on redhat

Hi all, i am trying to set up a CVS server on linux and to remote access the repository using WinCVS. I am facing some problem and i am unsure whether is it the client or the server not set up properly. In my winCVS client, i clicked Admin ->login and i specify my CVSROOT to be ":... (5 Replies)
Discussion started by: new2ss
5 Replies

6. Debian

What is CVS...?

Hi everyone... Could one of you kind Linux experts please let me know what CVS is In return I will kindly give you a thumbs up :b: a good trade I feel!! (1 Reply)
Discussion started by: TonyChapman
1 Replies

7. Solaris

cvs error

dear all I'm one of the CVS administrators here at my company and i have this cvs error in the messages Sep 18 07:20:37 dev cvs: Dying gasps received from client. can any one help me about this error ............. thanks murad jaber (1 Reply)
Discussion started by: murad.jaber
1 Replies

8. UNIX for Advanced & Expert Users

Cvs Cr-lf

I've experienced a problem with CVS when I've checked out sh script. When new build was created sources were checked out from CVS under Windows. Later this build was deployed under Linux und I recieved error from shell becouse of CR-LF EOL in file. I've tryed command dos2unix and become script... (7 Replies)
Discussion started by: Hitori
7 Replies

9. UNIX for Dummies Questions & Answers

CVS on UNIX

Hey guys, I am trying to setup CVS to run with an SSH connection, but am hitting a brick wall. I seem to be getting CVS to login correctly, but when I attempt to check out a module I am getting the following error: ksh: cvs: not found This is kinda implying to me that its not loading... (2 Replies)
Discussion started by: LiquidChild
2 Replies

10. Shell Programming and Scripting

CVS, Perl and VI

I am trying to write a Perl script to cover a few CVS commands (setup specifically), however the VI screen that pops up and requests a comment, what is the best way of entering the text automatically (will be a variable name) and then saving that cvs notification from my Perl program? Help is... (2 Replies)
Discussion started by: Shakey21
2 Replies
Login or Register to Ask a Question