Sponsored Content
Top Forums UNIX for Beginners Questions & Answers sed awk: split a large file to unique file names Post 302980154 by MadeInGermany on Wednesday 24th of August 2016 01:42:55 PM
Old 08-24-2016
I like the shell solution. It can become a little more I/O efficient (matters when the output files are written to a network file system):
Code:
p_col1=""
while read col1 rest
do
  if [ "$col1" != "$p_col1" ]
  then
    p_col1=$col1
    exec 3>"$col1".txt
  fi
  echo "$col1 $rest" >&3
done < Input_file
exec 3>&-

The input file must be sorted on col1 (otherwise: remove previous output files and append with exec 3>>"$col1".txt)

Last edited by MadeInGermany; 08-24-2016 at 04:42 PM.. Reason: Removed comment about awk - close() releases the file descriptors!
This User Gave Thanks to MadeInGermany For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies

2. UNIX for Dummies Questions & Answers

split a file with unique sets

This may sound like a trivial problem, but I still need some help: I have a file with ids and I want to split it 'n' ways (could be any number) into files: 1 1 1 2 2 3 3 4 5 5 Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may... (8 Replies)
Discussion started by: ChicagoBlues
8 Replies

3. Shell Programming and Scripting

extract unique pattern from large text file

Hi All, I am trying to extract data from a large text file , I want to extract lines which contains a five digit number followed by a hyphen , like 12345- , i tried with egrep ,eg : egrep "+" text.txt but which returns all the lines which contains any number of digits followed by hyhen ,... (19 Replies)
Discussion started by: shijujoe
19 Replies

4. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies

5. UNIX for Dummies Questions & Answers

Get List of Unique File Names

I have a large directory of web pages. I am doing a search through the web pages using grep and would like to get a list of unique file names of search results. The following command works fine to give me a list of file names where term appears: grep -l term *.html However, since these are... (3 Replies)
Discussion started by: rjulich
3 Replies

6. Shell Programming and Scripting

How to split a data file into separate files with the file names depending upon a column's value?

Hi, I have a data file xyz.dat similar to the one given below, 2345|98|809||x|969|0 2345|98|809||y|0|537 2345|97|809||x|544|0 2345|97|809||y|0|651 9685|98|809||x|321|0 9685|98|809||y|0|357 9685|98|709||x|687|0 9685|98|709||y|0|234 2315|98|809||x|564|0 2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies

7. Shell Programming and Scripting

Split File by Pattern with File Names in Source File... Awk?

Hi all, I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example: ... (2 Replies)
Discussion started by: cul8er
2 Replies

8. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

9. Shell Programming and Scripting

sed and awk not working on a large record file

Hi All, I have a very large single record file. abc;date||bcd;efg|......... pqr;stu||record_count;date when i do wc -l on this file it gives me "0" records, coz of missing line feed. my problem is there is an extra pipe that is coming at the end of this record like... (6 Replies)
Discussion started by: Gurkamal83
6 Replies

10. Linux

Split a large textfile (one file) into multiple file to base on ^L

Hi, Anyone can help, I have a large textfile (one file), and I need to split into multiple file to break each file into ^L. My textfile ========== abc company abc address abc contact ^L my company my address my contact my skills ^L your company your address ========== (3 Replies)
Discussion started by: fspalero
3 Replies
OCI_EXECUTE(3)															    OCI_EXECUTE(3)

oci_execute - Executes a statement

SYNOPSIS
bool oci_execute (resource $statement, [int $mode = OCI_COMMIT_ON_SUCCESS]) DESCRIPTION
Executes a $statement previously returned from oci_parse(3). After execution, statements like INSERT will have data committed to the database by default. For statements like SELECT, execution per- forms the logic of the query. Query results can subsequently be fetched in PHP with functions like oci_fetch_array(3). Each parsed statement may be executed multiple times, saving the cost of re-parsing. This is commonly used for INSERT statements when data is bound with oci_bind_by_name(3). PARAMETERS
o $statement - A valid OCI statement identifier. o $mode - An optional second parameter can be one of the following constants: Execution Modes +----------------------+---------------------------------------------------+ | Constant | | | | | | | Description | | | | +----------------------+---------------------------------------------------+ | | | |OCI_COMMIT_ON_SUCCESS | | | | | | | Automatically commit all outstanding changes for | | | this connection when the statement has succeeded. | | | This is the default. | | | | | | | | OCI_DESCRIBE_ONLY | | | | | | | Make query meta data available to functions like | | | oci_field_name(3) but do not create a result set. | | | Any subsequent fetch call such as | | | oci_fetch_array(3) will fail. | | | | | | | | OCI_NO_AUTO_COMMIT | | | | | | | Do not automatically commit changes. Prior to PHP | | | 5.3.2 (PECL OCI8 1.4) use OCI_DEFAULT which is | | | equivalent to OCI_NO_AUTO_COMMIT. | | | | +----------------------+---------------------------------------------------+ Using OCI_NO_AUTO_COMMIT mode starts or continues a transaction. Transactions are automatically rolled back when the connection is closed, or when the script ends. Explicitly call oci_commit(3) to commit a transaction, or oci_rollback(3) to abort it. When inserting or updating data, using transactions is recommended for relational data consistency and for performance reasons. If OCI_NO_AUTO_COMMIT mode is used for any statement including queries, and oci_commit(3) or oci_rollback(3) is not subsequently called, then OCI8 will perform a rollback at the end of the script even if no data was changed. To avoid an unnecessary rollback, many scripts do not use OCI_NO_AUTO_COMMIT mode for queries or PL/SQL. Be careful to ensure the appropriate transactional consis- tency for the application when using oci_execute(3) with different modes in the same script. RETURN VALUES
Returns TRUE on success or FALSE on failure. EXAMPLES
Example #1 oci_execute(3) for queries <?php $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'SELECT * FROM employees'); oci_execute($stid); echo "<table border='1'> "; while ($row = oci_fetch_array($stid, OCI_ASSOC+OCI_RETURN_NULLS)) { echo "<tr> "; foreach ($row as $item) { echo " <td>" . ($item !== null ? htmlentities($item, ENT_QUOTES) : "&nbsp;") . "</td> "; } echo "</tr> "; } echo "</table> "; ?> Example #2 oci_execute(3) without specifying a mode example <?php // Before running, create the table: // CREATE TABLE MYTABLE (col1 NUMBER); $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (123)'); oci_execute($stid); // The row is committed and immediately visible to other users ?> Example #3 oci_execute(3) with OCI_NO_AUTO_COMMIT example <?php // Before running, create the table: // CREATE TABLE MYTABLE (col1 NUMBER); $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (:bv)'); oci_bind_by_name($stid, ':bv', $i, 10); for ($i = 1; $i <= 5; ++$i) { oci_execute($stid, OCI_NO_AUTO_COMMIT); // use OCI_DEFAULT for PHP <= 5.3.1 } oci_commit($conn); // commits all new values: 1, 2, 3, 4, 5 ?> Example #4 oci_execute(3) with different commit modes example <?php // Before running, create the table: // CREATE TABLE MYTABLE (col1 NUMBER); $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (123)'); oci_execute($stid, OCI_NO_AUTO_COMMIT); // data not committed $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (456)'); oci_execute($stid); // commits both 123 and 456 values ?> Example #5 oci_execute(3) with OCI_DESCRIBE_ONLY example <?php $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'SELECT * FROM locations'); oci_execute($s, OCI_DESCRIBE_ONLY); for ($i = 1; $i <= oci_num_fields($stid); ++$i) { echo oci_field_name($stid, $i) . "<br> "; } ?> NOTES
Note Transactions are automatically rolled back when connections are closed, or when the script ends, whichever is soonest. Explicitly call oci_commit(3) to commit a transaction. Any call to oci_execute(3) that uses OCI_COMMIT_ON_SUCCESS mode explicitly or by default will commit any previous uncommitted transaction. Any Oracle DDL statement such as CREATE or DROP will automatically commit any uncommitted transaction. Note Because the oci_execute(3) function generally sends the statement to the database, oci_execute(3) can identify some statement syn- tax errors that the lightweight, local oci_parse(3) function does not. SEE ALSO
oci_parse(3). PHP Documentation Group OCI_EXECUTE(3)
All times are GMT -4. The time now is 03:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy