Sponsored Content
Top Forums Shell Programming and Scripting How to get a very big file sorted by contents of another variable list in one pass? Post 302459080 by methyl on Sunday 3rd of October 2010 02:40:13 PM
Old 10-03-2010
Is this a sort or a sift?
Do you need to sort the content of the 42 files? If so, what is the sort key?

Assuming no sort, sifting the main file into individual files with names based on the data characters 2-3 is relatively easy in a modern shell but it will not be quick with 17 million records.

Code:
For example:
cat filename | while read record
do
     category=`echo "${record}"|cut -c2-3`
     echo "${record}" >> "filename_${category}"
done

You can then do whatever processing you need to do in priority order.

For example:
cat reference_file | while read category
do
         ls -ald "filename_${category}"
done

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

getting the file name and pass as variable

Can any one suggest me how to check the file extension and pass the name based out of the filename within the folder. There would be always one latest file in the folder, but extension may vary... ie .csv, .CSV,.rpt,.xls etc what is best way to get the latest file name and pass as variable.... (1 Reply)
Discussion started by: u263066
1 Replies

2. Shell Programming and Scripting

how to read a value from a file and pass it to a variable

Hi, Some one please help me with this script. I have a file "sequence.txt" and contents of it look like below. 1 5 3 7 4 7 5 74 from my script i should be able to read every single and save the first columnbs to variable x1, x2, x3 and second column as y1, y2, y3 and so on. Then i can... (3 Replies)
Discussion started by: pragash_ms
3 Replies

3. Shell Programming and Scripting

Storing the contents of a file in a variable

There is a file named file.txt whose contents are: +-----------------------------------+-----------+ | Variable_name | Value | +-----------------------------------+-----------+ | Aborted_clients | 0 | | Aborted_connects | 25683... (6 Replies)
Discussion started by: proactiveaditya
6 Replies

4. Shell Programming and Scripting

How to read contents of a file into variable :(

My file is in this format : username : student information : default shell : student ID Eg : joeb:Joe Bennett:/bin/csh:1234 jerryd:Jerry Daniels:/bin/csh:2345 deaverm: Deaver Michelle:/bin/bash:4356 joseyg:Josey Guerra:/bin/bash:8767 michaelh:Michael Hall:/bin/ksh:1547 I have to... (1 Reply)
Discussion started by: dude_me5
1 Replies

5. Homework & Coursework Questions

How to read contents of a file into variable :(

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I have to read the contents of each field of a file creating user accounts. The file will be of format : ... (6 Replies)
Discussion started by: dude_me5
6 Replies

6. Red Hat

How to pass value of pwd as variable in SED to replace variable in a script file

Hi all, Hereby wish to have your advise for below: Main concept is I intend to get current directory of my script file. This script file will be copied to /etc/init.d. A string in this copy will be replaced with current directory value. Below is original script file: ... (6 Replies)
Discussion started by: cielle
6 Replies

7. Shell Programming and Scripting

Folder contents getting appended as strings while redirecting file contents to a variable

Hi one of the output of the command is as below # sed -n "/CCM-ResourceHealthCheck:/,/---------/{/CCM-ResourceHealthCheck:/d;/---------/d;p;}" Automation.OutputZ$zoneCounter | sed 's/$/<br>/' Resource List : <br> *************************** 1. row ***************************<br> ... (2 Replies)
Discussion started by: vivek d r
2 Replies

8. Shell Programming and Scripting

Inserting lines from one file to a sorted list

Hello friends! I am working a Psychology/Neuro* project where I am sorting inline citations by category. The final step of the process has me a little stuck. I need to take citations from a text list and sort them in another text file. Here is a file X example... (1 Reply)
Discussion started by: danbroz
1 Replies

9. Shell Programming and Scripting

How to pass a shellscript variable to a sql file?

Hi, i wan't to pass a shellscript variable to a sql file. a.sql select $field from dual; the way i am calling this is through sqlplus field_name="sysdate" sqlplus -s username/password@hostname:port/servicename <<EOF @a.sql $field_name EOF (4 Replies)
Discussion started by: reignangel2003
4 Replies

10. UNIX for Beginners Questions & Answers

Storing file contents to a variable

Hi All, I was trying a shell script. I was unable to store file contents to a variable in the script. I have tried the below but unable to do it. Input = `cat /path/op.diary` Input = $(<op.diary) I am using ksh shell. I want to store the 'op.diary' file contents to the variable 'Input'... (12 Replies)
Discussion started by: am24
12 Replies
Address(3pm)                                            User Contributed Perl Documentation                                           Address(3pm)

NAME
Palm::Address - Handler for Palm AddressBook databases SYNOPSIS
use Palm::Address; DESCRIPTION
The Address PDB handler is a helper class for the Palm::PDB package. It parses AddressBook databases. AppInfo block The AppInfo block begins with standard category support. See Palm::StdAppInfo for details. Other fields include: $pdb->{appinfo}{lastUniqueID} $pdb->{appinfo}{dirtyFields} I don't know what these are. $pdb->{appinfo}{fieldLabels}{name} $pdb->{appinfo}{fieldLabels}{firstName} $pdb->{appinfo}{fieldLabels}{company} $pdb->{appinfo}{fieldLabels}{phone1} $pdb->{appinfo}{fieldLabels}{phone2} $pdb->{appinfo}{fieldLabels}{phone3} $pdb->{appinfo}{fieldLabels}{phone4} $pdb->{appinfo}{fieldLabels}{phone5} $pdb->{appinfo}{fieldLabels}{phone6} $pdb->{appinfo}{fieldLabels}{phone7} $pdb->{appinfo}{fieldLabels}{phone8} $pdb->{appinfo}{fieldLabels}{address} $pdb->{appinfo}{fieldLabels}{city} $pdb->{appinfo}{fieldLabels}{state} $pdb->{appinfo}{fieldLabels}{zipCode} $pdb->{appinfo}{fieldLabels}{country} $pdb->{appinfo}{fieldLabels}{title} $pdb->{appinfo}{fieldLabels}{custom1} $pdb->{appinfo}{fieldLabels}{custom2} $pdb->{appinfo}{fieldLabels}{custom3} $pdb->{appinfo}{fieldLabels}{custom4} $pdb->{appinfo}{fieldLabels}{note} These are the names of the various fields in the address record. $pdb->{appinfo}{country} An integer: the code for the country for which these labels were designed. The country name is available as $Palm::Address::countries[$pdb->{appinfo}{country}]; $pdb->{appinfo}{misc} An integer. The least-significant bit is a flag that indicates whether the database should be sorted by company. The other bits are reserved. Sort block $pdb->{sort} This is a scalar, the raw data of the sort block. Records $record = $pdb->{records}[N]; $record->{fields}{name} $record->{fields}{firstName} $record->{fields}{company} $record->{fields}{phone1} $record->{fields}{phone2} $record->{fields}{phone3} $record->{fields}{phone4} $record->{fields}{phone5} $record->{fields}{address} $record->{fields}{city} $record->{fields}{state} $record->{fields}{zipCode} $record->{fields}{country} $record->{fields}{title} $record->{fields}{custom1} $record->{fields}{custom2} $record->{fields}{custom3} $record->{fields}{custom4} $record->{fields}{note} These are scalars, the values of the various address book fields. $record->{phoneLabel}{phone1} $record->{phoneLabel}{phone2} $record->{phoneLabel}{phone3} $record->{phoneLabel}{phone4} $record->{phoneLabel}{phone5} Most fields in an AddressBook record are straightforward: the "name" field always gives the person's last name. The "phoneN" fields, on the other hand, can mean different things in different records. There are five such fields in each record, each of which can take on one of eight different values: "Work", "Home", "Fax", "Other", "E-mail", "Main", "Pager" and "Mobile". The $record->{phoneLabel}{phone*} fields are integers. Each one is an index into @Palm::Address::phoneLabels, and indicates which particular type of phone number each of the $record->{phone*} fields represents. $record->{phoneLabel}{display} Like the phone* fields above, this is an index into @Palm::Address::phoneLabels. It indicates which of the phone* fields to display in the list view. $record->{phoneLabel}{reserved} I don't know what this is. METHODS
new $pdb = new Palm::Address; Create a new PDB, initialized with the various Palm::Address fields and an empty record list. Use this method if you're creating an Address PDB from scratch. new_Record $record = $pdb->new_Record; Creates a new Address record, with blank values for all of the fields. The AppInfo block will contain only an "Unfiled" category, with ID 0. "new_Record" does not add the new record to $pdb. For that, you want "$pdb->append_Record". SOURCE CONTROL
The source is in Github: http://github.com/briandfoy/p5-Palm/tree/master AUTHOR
Alessandro Zummo, "<a.zummo@towertech.it>" Currently maintained by brian d foy, "<bdfoy@cpan.org>" SEE ALSO
Palm::PDB(3) Palm::StdAppInfo(3) BUGS
The new() method initializes the AppInfo block with English labels and "United States" as the country. perl v5.10.1 2010-02-23 Address(3pm)
All times are GMT -4. The time now is 01:22 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy