Visit Our UNIX and Linux User Community

String editing using sed? awk?

Thread Tools Search this Thread
Homework and Emergencies Homework & Coursework Questions String editing using sed? awk?
# 8  
Old 11-12-2009
Originally Posted by peage1475
Thanks alot, and I would much rather learn myself than be spoon fed since I like this kind of stuff.
Excellent! This is the right attitude.

Originally Posted by peage1475
Ok im already having trouble. I can create a file that lists all the users;
Ok, but where do you get the information? In your original find-command you have searched certain directories and this will give you the users only if you know for sure that all the users you search have their home directories there. Is there a data structure in a Unix system which lists all the users and their home directories?

Ok, here is a tip: if in troubles, consult the "man-pages". Everybody does it (Unix expertise is about to know where to look rather than knowing everything.) There is especially one command you might need in your future career:

man -k <keyword>       searches all the man-articles for <keyword>
                        and lists the articles where <keyword> was found

I know that I should use cmp to tell if the files are the same, but I do not know how to set up the while loop
No problem, we will eventually get to that. You are correct, "cmp" is a correct tool for that purpose.

Why "a correct" and not "the correct"? Because in Unix there are usually several ways to do something - in this case i had "diff" in mind when i asked you, but "cmp" is also correct, so use this. (This - the possibility to do things in several ways - often leads to what Unixers call "religious wars". Some people will be convinced that there is only "one true way" and usually half of them will think variant A as this one true way and the other half will think variant B as the one true way. Exorcism ensues and this is why Unix is said to give people a purpose in life. ;-) )

I know I have to use a variable, but how do I get the while loop to transverse the file line by line?
Well, this is easy and i will tell you because it can't be researched easily:

Suppose you have some process, which output is a list, like. You can set up a loop with the elements of the list as variable content using the following construct. It is called a "pipeline":

process | while read variable ; do
   do_something_with $variable

Here is an example. Try out "ls -1" as single command first to see what "goes into" the pipeline, only then try the whole script. It should be pretty self-explanatory:


ls -1 | while read filename ; do
     print - "This is file: $filename"

Ok, report back when you have all the necessary information and we will continue with the last part of the script.

# 9  
Old 11-12-2009
ok here is what I am trying but it does not seem to work correcty
cat test1 | while read path
if cmp -s $path ./public_html/index.html
echo $path >> files
echo "working..."

from my understanding cmp -s will produce either a 0, if the files are identical, or a 1 if the files are different. So my if statment says that if the files are different add $path to files, else prin working. why is this only producing working... when I know for a fact that there are at least 5 that are modified different then the one it is comparing too?
# 10  
Old 11-12-2009
Originally Posted by peage1475
from my understanding cmp -s will produce either a 0, if the files are identical, or a 1 if the files are different. So my if statment says that if the files are different add $path to files, else prin working. why is this only producing working... when I know for a fact that there are at least 5 that are modified different then the one it is comparing too?
The reason is you have a simple syntactic error, but lets first go over what you have done correctly, because there are some good points.

First, you have found a fundamental structure in Unix: you can use the return code of a program to determine how it has operated. In this case you - correctly - try to use the return code of "cmp". Very good.

Second, you have found out another fundamental thing: like in the C programming language throughout Unix a value of 0 is considered to be a logical TRUE while every other value is considered to mean a logical FALSE.

Now to your error: the correct syntax of the if-statement is:

if [ :TRUE: ] ; then

Consider ":TRUE:" to be some process (or set of processes in a pipeline) or condition which either results in a "0" (TRUE) or another integer (FALSE).

In the historical Unix there was a command called "[", which was a link to the command "/usr/bin/test", which is still there. I suggest you read to man page of "test" to find out which possibilities it has. Basically "test" gets some logical expression and like any other program returns "0" (if the logical expression results in TRUE) or "1" (FALSE).

Because "[" was a normal program (most shells have implemented it as an internal command in the meantime, but Unix is adamant about being backwards compatible) it is clear why before and after it there have to be spaces: the rest of the line up to "]" are commandline arguments to "test" and therefore there have to be spaces like "ls-l" is wrong and "ls -l" is right.

Try the following two scripts:

if 0 ; then
     echo TRUE
     echo FALSE

if [ 1 = 1 ] ; then
     echo TRUE
     echo FALSE

In the first script then change the "0" to "1" or another integer and try again - notice what happens in the light of what i did say.

In the second script change the (obviously true) expression to another expression and experiment. Watch the output.

Bonus question: the expression "1 = 1" is working, but in fact the expression is not suited for comparing integer values ("=" is for comparing strings). How should it be phrased as true integer comparison? Hint: consult the man page!

You should now be able to change your script to work. That leaves two questions:

1) You now feed some file content to the while-loop, but this is hardly an up-to-date user directory. It is a good development device, though, so i am very satisfied that you had this idea to "simulate having this list" for the moment. It is a clever way of solving one problem after the other - i do exactly the same when writing scripts. Still, you have to solve the problem of "how to obtain a list of all defined users in a machine" and - "how to get the subset of this list i am interested in"? Any ideas?

2) Notice, that you output only "$path" to an output file, and not, what you want to stand there. If you prepare the output file this way it will be intermediary in nature and this is not, what you want. Is there a way to manipulate strings in the shell and - if this is possible - which commands might be useful for this purpose? Hint: consult your classes written material, the recommended book or whatever you got in this class rather than the "man-page" first - this is something the man-pages are poorly suited to research. Once you have a good idea of string-manipulating commands read their man pages of course.

All in all I'm quite satisfied with your progress so far - you might not know it but you are close to the solution. Keep up the good work.

I hope this helps.

# 11  
Old 11-13-2009
Yeah that does help, and thank you so much for helping me. However I think that another reason it is not working is that I am not using the cat | read correctly. I created a file that contains a path to some file then I used cat on it and piped it to read like so:
cat tmp1 | read testvar
echo $testvar

Nothing was printed by echo, I am thinking that my syntax is off or something and that is another reason why this is not working for me. I have tried finding out how to do this but I cannot find anything nor can I figure it out.
# 12  
Old 11-13-2009
Carefully analyze the difference between what you have done before and what you are doing now and you will understand whats wrong yourself:

cat test1 | while read line


cat test1 | read line

"cat test1" is a process, yes? It has some output, which you can observe by issuing this command, it will print the content of the file, line by line.

Now you pipe the output of this process into another process, which is set up by "while ... do .... done". To the outside this counts as one process, even if it contains several other processes inside.

In the second case you pipe the output into a completely different process, a command called "read", which tries to fill a variable with content. In the first case "read" is only invoked by the while-loop as a sub-process and is presented one line after the other which it dutifully assigns to the variable $line. In the second case it is fed the whole output of the first process at once. What is poor "read" supposed to do, hm?

# 13  
Old 11-13-2009
Ok so I finally got it all working except for the part on how to read from the command line. I do not know what I am supposed to do, or even begin to start on how to do it. Can you help please? thanks
# 14  
Old 11-15-2009
If you don't know what to do i suggest you read the thread and answer the questions i posed. If you got the script working post it here and i can tell you what is correct and what isn't.

Up to now you have never mentioned any command line arguments you have to read so i'm a bit astonished to hear from them now. The problem as you have presented it doesn't need any input from the command line at all.


Previous Thread | Next Thread
Test Your Knowledge in Computers #315
Difficulty: Easy
To bypass US Munitions Export Laws, the creator of the PGP published all the source code in a binary format.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replace string of a file with a string of another file for matches using grep,sed,awk

I have a file comp.pkglist which mention package version and release . In 'version change' and 'release change' line there are two versions 'old' and 'new' Version Change: --> Release Change: --> cat comp.pkglist Package list: nss-util-devel-3.28.4-1.el6_9.x86_64 Version Change: 3.28.4 -->... (1 Reply)
Discussion started by: Paras Pandey
1 Replies

2. Shell Programming and Scripting

Replace string in XML file with awk/sed with string from another

Sorry for the long/weird title but I'm stuck on a problem I have. I have this XML file: </member> <member> <name>TransactionID</name> <value><string>123456789123456</string></value> </member> <member> <name>Number</name> ... (9 Replies)
Discussion started by: cozzin
9 Replies

3. Shell Programming and Scripting

sed awk to remove the , in a string

Dear All, Can anyone help to remove the , bewteen "" in a string by using sed or awk? e.g. input : 1,4,5,"abcdef","we,are,here",4,"help hep" output:1,4,5,"abcdef","wearehere",4,"help hep" Thanks, Mimi (5 Replies)
Discussion started by: mimilaw
5 Replies

4. Shell Programming and Scripting

sed or awk command to replace a string pattern with another string based on position of this string

here is what i want to achieve... consider a file contains below contents. the file size is large about 60mb cat dump.sql INSERT INTO `table1` (`id`, `action`, `date`, `descrip`, `lastModified`) VALUES (1,'Change','2011-05-05 00:00:00','Account Updated','2012-02-10... (10 Replies)
Discussion started by: vivek d r
10 Replies

5. Shell Programming and Scripting

editing file with awk cut and sed

HI All, I am new to unix. I have a file would like to do some editing by using awk, cut and sed. Could anyone help? This file contain 100 lines. There are one line for example: 2,"102343454",5060,"579668","579668","579668","SIP",,,"825922","035885221283026",1,268,"00:59:00.782 APR 17... (2 Replies)
Discussion started by: mimilaw
2 Replies

6. Shell Programming and Scripting

match string exactly with awk/sed

Hi all, I have a list that I would like to parse with awk/sed. The list is contains entries such as: JournalTitle: Biochemistry JournalTitle: Biochemistry and cell biology = Biochimie et biologie cellulaire JournalTitle: Biochemistry and experimental biology JournalTitle: Biochemistry and... (6 Replies)
Discussion started by: euval
6 Replies

7. Shell Programming and Scripting

Line/Variable Editing for Awk sed Cut

Hello, i have a file, i open the file and read the line, i want to get the first item in the csv file and also teh third+6 item and wirte it to a new csv file. only problem is that using echo it takes TOO LONG: please help a newbie. below is my code: WorkingDir=$1 FileName=`cut -d ',' -f... (2 Replies)
Discussion started by: limamichelle
2 Replies

8. Shell Programming and Scripting

sed or awk editing help

Hi all I use aix (sadly). I've got a file consisting of fields separated by commas, I need a sed or awk command that will delete all spaces between two commas as long as there are only spaces between the commas. eg ,abc, ,sd , ,dr at would become ,abc,,sd ,,dr at I have... (53 Replies)
Discussion started by: mychmose
53 Replies

9. Shell Programming and Scripting

awk/sed - getting string instead of number

Hi! I am writing a script handling downloading list of files and I have to check whether file is present locally and if not finished than continue downloading. To do so I have to compare sizes of remote file and local file. To check remote file size I have to parse something like this: ... (2 Replies)
Discussion started by: hrwath
2 Replies

10. Shell Programming and Scripting

Editing File using awk/sed

Hello Awk Gurus, Can anyone of you help me with the below problem. I have got a file having data in below format pmFaultyTransportBlocks ----------------------- 9842993 pmFrmNoOfDiscRachFrames ----------------------- NULL pmNoRecRandomAccSuccess -----------------------... (4 Replies)
Discussion started by: Mohammed
4 Replies

Featured Tech Videos