Sponsored Content
Top Forums Shell Programming and Scripting How to parse a huge 600MB zipped file? Post 302668753 by alister on Tuesday 10th of July 2012 02:26:30 AM
Old 07-10-2012
Quote:
Originally Posted by PikK45
Our advice would be, use awk commands like below in the script.
Code:
column1=$(bzcat test.test.bz2| awk '{FS=","} {print $1}' | uniq | wc -l)
                echo $column1
column2=$(bzcat test.test.bz2 | awk '{FS=","} {print $2}' | uniq | wc -l)
                echo $column2

My simple suggestion was to do the following...
Code:
column1=$(printf "%s" "$line" | awk '{FS=","} {print $1}' | uniq | wc -l)
                echo $column1
column2=$(printf "%s" "$line" | awk '{FS=","} {print $2}' | uniq | wc -l)
                echo $column2

... but that wouldn't work. Woops.

The most efficient way to accomplish this that comes to mind is to do it all with a single awk invocation:

Code:
bzcat test.test.bz2 |
awk -F, '$1 != o1 {x++} $2 != o2 {y++} {o1 = $1; o2 = $2} END {print "column1="x; print "column2="y}'

Regards,
Alister
This User Gave Thanks to alister For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

zipped or unzipped file

Is there a way you can tell if a file is still zipped or it's unzipped I have a file called ssss.zip and I would like to know if this file is still zipped or if it's unzipped? I'm on IBM AIX/RS6000 (3 Replies)
Discussion started by: ted
3 Replies

2. UNIX for Dummies Questions & Answers

sendind a zipped file via email

Hi, I was not sure if I can do this. Suppose I have a file under /tmp Suppose the file is called any_11_52.txt Fisrt QUESTION??? If I zip this file using gzip will the user be able to unzip it , if I send it as an attachment in an email. Secondly is there a command by which we can... (2 Replies)
Discussion started by: rooh
2 Replies

3. Shell Programming and Scripting

How to search a pattern inside a zipped file ie (.gz file) with out unzipping it

How to search a pattern inside a zipped file ie (.gz file) with out unzipping it? using grep command.. Bit urgent.. pls..help me (2 Replies)
Discussion started by: senraj01
2 Replies

4. UNIX for Dummies Questions & Answers

how to check if file is zipped

I have a script that grabs files from directory , zips and moves them somewhere else. It works fine except the case when files it grabs are already zipped. Then it trys to zip it again which does not make sence. How can I check before zipping if file is already zipped? thanks in advance (3 Replies)
Discussion started by: arushunter
3 Replies

5. UNIX for Dummies Questions & Answers

reading a zipped file without unzipping it?

Dear all, I would like to ask how i can read a zipped file (file.gz) without actually unzipping it? i think there is a way to do so but i can't remember it.. can anyone help? thanks in advance.. (1 Reply)
Discussion started by: marwan
1 Replies

6. UNIX for Dummies Questions & Answers

Zipped tar file is corrupt

Hello, I am currently dumping 30-40 reports on a Unix folder located here /home/apps/reports/prode/excel I use K-shell to do this task. In that, I use the gzip command to compress these files. I want to be able to use a tar command to first load the entire directory into one file then gzip that... (2 Replies)
Discussion started by: Pramodini Rode
2 Replies

7. Solaris

How can I tell if a file is zipped or not?

SunOS xxxxxx 5.10 Generic_142900-15 sun4v sparc SUNW,T5240 We receive files that are sometimes zipped, but the file may not have the .gz or other extention that would indicated that the file is zipped. Is there a unix "test" command that I could use or something similar? Thanks in advance (7 Replies)
Discussion started by: Harleyrci
7 Replies

8. Shell Programming and Scripting

FTP'ing the zipped file

Hi, I need to have a shell script that FTP's a zipped file from a particular location. I have some path and inside that path i will have folders like x_timestamp and inside x_timestamp there may many folders based upon events like y_111,y_222,y_333.Inside each event there will be another... (3 Replies)
Discussion started by: weknowd
3 Replies

9. Shell Programming and Scripting

awk to parse huge files

Hello All, I have a situation as below: (1) Read a source file (a single file of 1.2 million rows in it ) (2) Read Destination files one by one and replace the content ( few fields in it ) with the corresponding matching field from source file. I tried as below: ( please note I am not... (4 Replies)
Discussion started by: panyam
4 Replies

10. Shell Programming and Scripting

Work with huge Zipped files

Hello dear members, I have one general and one specific question which I will be very grateful if you could help me with them. Let's start with my general question: 1. I am working on cluster computer shared with other people and I need to manipulate a big zipped text file of 13 GB. There is... (1 Reply)
Discussion started by: Homa
1 Replies
IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME
igawk - gawk with include files SYNOPSIS
igawk [ all gawk options ] -f program-file [ -- ] file ... igawk [ all gawk options ] [ -- ] program-text file ... DESCRIPTION
Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1). AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like @include getopt.awk in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path. OPTIONS
See gawk(1) for a full description of the AWK language and the options that gawk supports. EXAMPLES
cat << EOF > test.awk @include getopt.awk BEGIN { while (getopt(ARGC, ARGV, "am:q") != -1) ... } EOF igawk -f test.awk SEE ALSO
gawk(1) Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995. AUTHOR
Arnold Robbins (arnold@skeeve.com). Free Software Foundation Nov 3 1999 IGAWK(1)
All times are GMT -4. The time now is 06:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy