Work with huge Zipped files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Work with huge Zipped files
# 1  
Old 11-08-2013
Ubuntu Work with huge Zipped files

Hello dear members,

I have one general and one specific question which I will be very grateful if you could help me with them. Let's start with my general question:

1. I am working on cluster computer shared with other people and I need to manipulate a big zipped text file of 13 GB. There is no possibility of unzipping it, if I could do that, I would have any problem but my problem is that working with zipped files using Awk or Linux commands become very difficult. I would be very grateful if you could give me some directions in this regard.

2. The second question is that being in that 13 GB zipped, I want to select one part of the file to make an example file to write my code to be able to later apply them to the whole file. When I display the file on shell by the command:

Code:
zcat zippedfile | less

and I copy a section just by clicking and I paste it in a text editor, the spacing, number of fields, everything gets destroyed. How could I select a part of the file by command line without unzipping it?

Thank you
# 2  
Old 11-11-2013
Hello Homa,
  • How do you know which section you are after?
  • How many lines is the total file and how many lines do you want to extract?
  • What operating system and version are you using?



Thanks, in advance,
Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1... (4 Replies)
Discussion started by: ariesto
4 Replies

2. Shell Programming and Scripting

Grabing the same timestamp from files that are ZIPPED

Hi, I am zipping more than 20 files that has same timestamp in all of them. I need to create the zip file with the same timestamp as in the files that are zipped. So I have files: Dummytest_20140601W110515_file1.txt Dummytest_20140601W110515_file2.txt ....... .......... (5 Replies)
Discussion started by: Saanvi1
5 Replies

3. Shell Programming and Scripting

Renaming all files inside a zipped file

Hi, To all the Unix gurus this should be a simple task, but as a newbie I'm finding it hard to crack this. Any help is highly appreciated... Scenario: Step 1 : Move zip file from FTP folder to WORK folder Step 2: Unzip the file "Sample_YYYYMMDDHHMMSS.tar.gz" which contains many file... (10 Replies)
Discussion started by: asandy1234
10 Replies

4. Shell Programming and Scripting

How to parse a huge 600MB zipped file?

I'm new to Unix, trying to parse a huge 600MB zipped file... I need to bzcat this file once and do some calculations (word count) on the lines based on certain criteria (see script) the correct result/output should be: column1=6 column2=4 the problem is that I'm getting column2=0 (see... (16 Replies)
Discussion started by: DeltaComp
16 Replies

5. Shell Programming and Scripting

Copy all zipped files from one folder to another

Hi everyone, when I try to copy *.gz files run cp within the correct source folder it works as follow: Source folder = C:/Documents and Settings/user/Recent papers/2771/ Destination folder = C:/Documents and Settings/user/My documents/1532/temp cp *.gz "C:/Documents and Settings/user/My... (2 Replies)
Discussion started by: cgkmal
2 Replies

6. Shell Programming and Scripting

Difference between 2 zipped text files.

Hi, I have below two zipped files - file1.gz 023384148,1,,,02077301961,R,02077301961,N,0,02077301961,N,0,0,8010,02077300518,U,N,,02077300518,U,20100501011732,20100501011732,0,20100501011815,00000430,16,16,10,N;... (8 Replies)
Discussion started by: ravigupta2u
8 Replies

7. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

8. Linux

grep thru zipped files

Hi All, I need to check the logs and grep it. The problem is that the previous days are zipped with *.gz. How do grep on the *.gz? Thank you in advance. (1 Reply)
Discussion started by: itik
1 Replies

9. Shell Programming and Scripting

Greping from zipped files without unzipping

I have more than 500 gzipped files in a directory. I have one lookup file in the same directory with 200 key values. I need to get the name of the gzipped file which have any of these 200 key values. Here my criteria is do not unzip the files due to space constraint. Any suggestion? (3 Replies)
Discussion started by: kanu_kanu
3 Replies

10. UNIX for Dummies Questions & Answers

uploading Zipped files get 553 error

I am trying to upload .zip files to Unix server and get the error 553 qmerev2002.zip: Permission denied, what is my problem?? I am able to load other files and folders fine. (3 Replies)
Discussion started by: CoastGuard1970
3 Replies
Login or Register to Ask a Question