Visit Our UNIX and Linux User Community


Getting rid of a field in a big file - Urgent


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Getting rid of a field in a big file - Urgent
# 1  
Old 09-30-2009
Getting rid of a field in a big file - Urgent

Hi Unix Gurus,

I have a file with 100,000 lines with lines in the format:

1,35518,35518,1,2,1,72,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,78973,78973,1,2,2,91,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,82108,82108,1,2,1,402,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,83406,83406,1,2,1,1684,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,89373,89373,1,2,1,402,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,90924,90924,1,2,2,402,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,91225,91225,1,2,5,1119,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,91721,91721,1,2,1,3305,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,94631,94631,1,2,6,402,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445
1,102766,102766,1,2,6,1119,1253889535,1253889775,240,30,152.163.141.8,US,0,00,-g 1253888723-g 1253889445

I need to get rid of the part "1253888723-g" from every line. Basically in the last field of every line, "-g 1253888723-g 1253889445" I need to get rid of the center part which is "1253888723-g" and have the last field like this: "-g 1253889445" . Is there any easy way of doing this using perl or awk. Please help me out. Your help is really appreciated.

Thanks,
Toms
# 2  
Old 09-30-2009
Code:
sed -i.bak 's/-g [0-9]\{10\}//' your_bigFile

assuming the field is always '-g followed by 10 digits'.
This will also keep a copy of your file at your_bigFile.bak.
# 3  
Old 09-30-2009
One more question please...

Thanks a lot for your quick response. That seemed to work pretty well Smilie

I would also like to get a solution to actually get rid of the entire last field including the last comma from every line. Basically the entire field ",-g 1253888723-g 1253889445"

Toms
# 4  
Old 09-30-2009
Code:
sed 's/\(.*\),.*$/\1/'


Previous Thread | Next Thread
Test Your Knowledge in Computers #362
Difficulty: Medium
The sed command first appears in Version 5 AT&T UNIX.
True or False?

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

URGENT Reading a file and assessing the syntax shell script URGENT

I am trying to write a shell script which takes an input file as an arguement in the terminal e.g. bash shellscriptname.sh input.txt. I would like for the file to be read line by line each time checking if the .txt file contains certain words or letters(validating the syntax). If the line being... (1 Reply)
Discussion started by: Gurdza32
1 Replies

2. UNIX for Dummies Questions & Answers

Any way to get rid of ^M characters in a text file using pr?

When I use vi to see what's in the file I get this: int add1(int x) {^M return x + 1;^M} ^Mint subtract1(int x) {^M return x - 1;^M} ^Mint double_it(int x) {^M return x * 2;^M} ^Mint halve_it(int x) {^Mreturn x / 2;^M} ^Mint main() {^M int myint;^M int result;^M ... (2 Replies)
Discussion started by: Nonito84
2 Replies

3. UNIX for Dummies Questions & Answers

How big is too big a config.log file?

I have a 5000 line config.log file with several "maybe" errors. Any reccomendations on finding solvable problems? (2 Replies)
Discussion started by: NeedLotsofHelp
2 Replies

4. Shell Programming and Scripting

awk get rid of space in end of field

Hello. I'm using a file to "grep" in a 2nd one (with awk) cat file1 2 first user 9 second user 1 third user (with a space after user) I want to get the line except the 1st field so I do : field=$(gawk '{$1 =""; print $0}' file | sed 's/^ //') It works but it deletes... (5 Replies)
Discussion started by: xanthos
5 Replies

5. Shell Programming and Scripting

how to get rid of ^M in the file

Hi I have a file wich contains ^M characters, looks like these are from DOS, Is there a way to get rid of them? Thanks -A (3 Replies)
Discussion started by: aoussenko
3 Replies

6. Shell Programming and Scripting

How can I get rid of the ` character from input file?

A sed command works most of the time however it fails sometimes. I put each line (record) I read of a file through the following command data=$(cat file | sed 's///g' | sed 's|.*ex:Msg\(.*\)ex:Msg.*|\1|' ) When I run the script I get a message that states that there is an invalid format... (6 Replies)
Discussion started by: gugs
6 Replies

7. UNIX for Advanced & Expert Users

Get rid of junk character in a file

I have a file with one of the following lines, when opened with vi 33560010686GPT¬£120600GBPGBP10082007DS In the above line, I want to get rid of the junk character before the £ (pound sysmbol). When I tried copying £ from windows and copy in unix vi, it prints as ¬£ and I tried pattern replace... (2 Replies)
Discussion started by: nskworld
2 Replies

8. UNIX for Dummies Questions & Answers

How to view a big file(143M big)

1 . Thanks everyone who read the post first. 2 . I have a log file which size is 143M , I can not use vi open it .I can not use xedit open it too. How to view it ? If I want to view 200-300 ,how can I implement it 3 . Thanks (3 Replies)
Discussion started by: chenhao_no1
3 Replies

Featured Tech Videos