Sponsored Content
Full Discussion: Work with huge Zipped files
Top Forums Shell Programming and Scripting Work with huge Zipped files Post 302872273 by Homa on Friday 8th of November 2013 04:57:52 AM
Old 11-08-2013
Ubuntu Work with huge Zipped files

Hello dear members,

I have one general and one specific question which I will be very grateful if you could help me with them. Let's start with my general question:

1. I am working on cluster computer shared with other people and I need to manipulate a big zipped text file of 13 GB. There is no possibility of unzipping it, if I could do that, I would have any problem but my problem is that working with zipped files using Awk or Linux commands become very difficult. I would be very grateful if you could give me some directions in this regard.

2. The second question is that being in that 13 GB zipped, I want to select one part of the file to make an example file to write my code to be able to later apply them to the whole file. When I display the file on shell by the command:

Code:
zcat zippedfile | less

and I copy a section just by clicking and I paste it in a text editor, the spacing, number of fields, everything gets destroyed. How could I select a part of the file by command line without unzipping it?

Thank you
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

uploading Zipped files get 553 error

I am trying to upload .zip files to Unix server and get the error 553 qmerev2002.zip: Permission denied, what is my problem?? I am able to load other files and folders fine. (3 Replies)
Discussion started by: CoastGuard1970
3 Replies

2. Shell Programming and Scripting

Greping from zipped files without unzipping

I have more than 500 gzipped files in a directory. I have one lookup file in the same directory with 200 key values. I need to get the name of the gzipped file which have any of these 200 key values. Here my criteria is do not unzip the files due to space constraint. Any suggestion? (3 Replies)
Discussion started by: kanu_kanu
3 Replies

3. Linux

grep thru zipped files

Hi All, I need to check the logs and grep it. The problem is that the previous days are zipped with *.gz. How do grep on the *.gz? Thank you in advance. (1 Reply)
Discussion started by: itik
1 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. Shell Programming and Scripting

Difference between 2 zipped text files.

Hi, I have below two zipped files - file1.gz 023384148,1,,,02077301961,R,02077301961,N,0,02077301961,N,0,0,8010,02077300518,U,N,,02077300518,U,20100501011732,20100501011732,0,20100501011815,00000430,16,16,10,N;... (8 Replies)
Discussion started by: ravigupta2u
8 Replies

6. Shell Programming and Scripting

Copy all zipped files from one folder to another

Hi everyone, when I try to copy *.gz files run cp within the correct source folder it works as follow: Source folder = C:/Documents and Settings/user/Recent papers/2771/ Destination folder = C:/Documents and Settings/user/My documents/1532/temp cp *.gz "C:/Documents and Settings/user/My... (2 Replies)
Discussion started by: cgkmal
2 Replies

7. Shell Programming and Scripting

How to parse a huge 600MB zipped file?

I'm new to Unix, trying to parse a huge 600MB zipped file... I need to bzcat this file once and do some calculations (word count) on the lines based on certain criteria (see script) the correct result/output should be: column1=6 column2=4 the problem is that I'm getting column2=0 (see... (16 Replies)
Discussion started by: DeltaComp
16 Replies

8. Shell Programming and Scripting

Renaming all files inside a zipped file

Hi, To all the Unix gurus this should be a simple task, but as a newbie I'm finding it hard to crack this. Any help is highly appreciated... Scenario: Step 1 : Move zip file from FTP folder to WORK folder Step 2: Unzip the file "Sample_YYYYMMDDHHMMSS.tar.gz" which contains many file... (10 Replies)
Discussion started by: asandy1234
10 Replies

9. Shell Programming and Scripting

Grabing the same timestamp from files that are ZIPPED

Hi, I am zipping more than 20 files that has same timestamp in all of them. I need to create the zip file with the same timestamp as in the files that are zipped. So I have files: Dummytest_20140601W110515_file1.txt Dummytest_20140601W110515_file2.txt ....... .......... (5 Replies)
Discussion started by: Saanvi1
5 Replies

10. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1... (4 Replies)
Discussion started by: ariesto
4 Replies
DEBCONF-SET-SELECTIONS(1)					      Debconf						 DEBCONF-SET-SELECTIONS(1)

NAME
debconf-set-selections - insert new default values into the debconf database SYNOPSIS
debconf-set-selections file debconf-get-selections | ssh newhost debconf-set-selections DESCRIPTION
debconf-set-selections can be used to pre-seed the debconf database with answers, or to change answers in the database. Each question will be marked as seen to prevent debconf from asking the question interactively. Reads from a file if a filename is given, otherwise from stdin. WARNING
Only use this command to seed debconf values for packages that will be or are installed. Otherwise you can end up with values in the database for uninstalled packages that will not go away, or with worse problems involving shared values. It is recommended that this only be used to seed the database if the originating machine has an identical install. DATA FORMAT
The data is a series of lines. Lines beginning with a # character are comments. Blank lines are ignored. All other lines set the value of one question, and should contain four values, each separated by one character of whitespace. The first value is the name of the package that owns the question. The second is the name of the question, the third value is the type of this question, and the fourth value (through the end of the line) is the value to use for the answer of the question. Alternatively, the third value can be "seen"; then the preseed line only controls whether the question is marked as seen in debconf's database. Note that preseeding a question's value defaults to marking that question as seen, so to override the default value without marking a question seen, you need two lines. Lines can be continued to the next line by ending them with a "" character. EXAMPLES
# Force debconf priority to critical. debconf debconf/priority select critical # Override default frontend to readline, but allow user to select. debconf debconf/frontend select readline debconf debconf/frontend seen false OPTIONS
--verbose, -v verbose output --checkonly, -c only check the input file format, do not save changes to database SEE ALSO
debconf-get-selections(1) (available in the debconf-utils package) AUTHOR
Petter Reinholdtsen <pere@hungry.com> 2011-06-22 DEBCONF-SET-SELECTIONS(1)
All times are GMT -4. The time now is 06:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy