Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

5
votes
1answer
371 views

merge that works at word granularity instead of line

The old diff, patch, and merge programs, and their modern descendents in the form of version control systems work great for managing source code that can be edited concurrently by multiple people. ...
3
votes
5answers
2k views

How to replace a list of placeholders in a text file?

I have a text (configuration) file, but the program that reads the file unfortunately doesn't allow using any kind of variables. So I'd like to use a preprocessor that replaces a set of placeholders ...
0
votes
1answer
1k views

Convert text to image in centos from bash

I want to create an image [jpg, png, doesn't matter] from shell but I don't have convert command. The image is very simple - white background with a medium-size black text in the middle. Is it ...
8
votes
1answer
3k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
4
votes
4answers
1k views

How can I remove a specific string from a file only if there are other lines in the file?

How can I remove a specific string from a file ONLY if there are other lines in the file? For example, don't touch this file: cat file.txt ASDF but remove "ASDF" from this file: cat file.txt ASDF ...
3
votes
6answers
6k views

Converting DOS text files to Unix

I have a bunch of PHP scripts that are in DOS format (vi displayed [noeol] [dos] in the status line) and as expected they weren't usable. I ran dos2unix over them to fix the line endins: dos2unix ...
2
votes
1answer
2k views

Enable line breaks when assigning multi line outputs to var in bash

I'm using the following very basic shell script I copied off the net to list the contents of my database backup folder and mail them regularily: MYSQLLIST=$(ls -lhG /var/backups/mysql/daily/) ...
6
votes
2answers
4k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...
13
votes
5answers
48k views

How to strip multipe spaces to one using sed?

sed on AIX is not doing what I think it should. I'm trying to replace multiple spaces with a single space in the output of IOSTAT: # iostat System configuration: lcpu=4 drives=8 paths=2 vdisks=0 ...
7
votes
2answers
1k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
45
votes
4answers
23k views

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
27
votes
6answers
12k views

How to obtain inverse behavior for `tail` and `head`?

Is there a way to head/tail a document and get the reverse output; because you don't know how many lines there are in a document? I.e. I just want to get everything but the first 2 lines of foo.txt ...
29
votes
17answers
66k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
6
votes
3answers
4k views

Command like `column -t` that instead keeps separators in output

I'm editing a simple table. I would like to have it nicely formatted. While I could use tbl, latex, or similar, this seems overkill -- plain text really is sufficient. As it's simple I might as ...
2
votes
2answers
743 views

How to temporarily disable the Escape key in a Terminal?

I am parsing keyboard input via read -n 1 and I haven't found a way to distinguish between an actual key-press of the Escape key and the first Escape character (byte) of a control sequence generated ...
4
votes
3answers
2k views

Read and process a string, char by char, yet allow user to simple line edit the input

I want to incrementally read a line of input from the terminal, and allow the user some basic line editing functionality; INS, DEL, RIGHT, LEFT HOME, END, BACKSPACE Each time the string is ...
8
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
2
votes
1answer
333 views

rotating buffer type file logging utility

I'am trying to debug a server that throws a lot of logs on stdout/stderr. I need to redirect only last N lines to a file. Something like a rotating buffer feature there in tcpdump's -C & -W flags. ...
8
votes
3answers
2k views

Is it possible to nest a 'find -exec' within another 'find -exec'?

Something like the following is what I what I'm after, but my code doesn't work, no matter how I escape {} and + ; find ./ -maxdepth 1 -type d -name '.*' -exec \ find {} -maxdepth 1 -type f ...
3
votes
2answers
6k views

Replace all occurrences of a character in a variable using korn script

Part of the Korn script I am writing requires that I replace all occurrences of the ' character with two occurrences (''). I am trying to log some SQL that I am generating in this script to a column ...
6
votes
2answers
156 views

What is the connection between a gedit bug and a Unix-&-Linux Q/A href?

While answering a Unix-&-Linux question, I observed that Gedit and two other editors, Leafpad and Medit (I tested 12 editors altogether) exhibit the a certain bug. As it turns out, the bug is ...
8
votes
5answers
2k views

Why does shell Command Substitution gobble up a trailing newline char?

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p # ...
10
votes
2answers
2k views

Where has the trailing newline char gone from my command substitution?

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 ...
2
votes
1answer
242 views

Rename beginning of file name using specific text in the file itself

I have many files in directory (file1.txt, file2.txt, ...) Every files contains bunch of lines. One of the line is specific text thetext followed by random number just like thetext56, thetext21, etc. ...
7
votes
1answer
324 views

Change date “July 29th, 2011” to “20110729”

I have lots of HTML files contains date with format July 29th, 2011 I want to change date format July 29th, 2011 to 20110729, December 9th, 2010 to 20101209, etc. I think sed may works but still ...
2
votes
4answers
6k views

Deleting Lines Containing a Specified String in Different Files

Suppose there are two files in web/ named foo.php and bar.php. The 1st line of foo.php is "sdajgeSTRINGdsad" and the 10th line of bar.php is "gdfhu98324STRING". The task is to first locate these two ...
13
votes
6answers
27k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...
4
votes
2answers
6k views

Show only text between 2 matching pattern

I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed If param1 is line matching pattern from other file using sed, below command will works ...
2
votes
1answer
150 views

Is there a way to get the effect of combining grep -v with grep -A?

Is there a way, either with grep or something else, to (in effect) combine grep -v and grep -A? That is, rather than include everything for x lines after a string, exclude everything. Quick reminder ...
4
votes
2answers
1k views

Where has my `uniq` or `sort -u` line gone, with some unicode characters

What is going on in the following code snippet? I'm not getting my expected output. I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something ...
8
votes
9answers
20k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
1
vote
2answers
585 views

Text processing: convert text of table of content into djvu bookmark format?

Usually in the table of content of a book, each line is for a chapter or section and ended with its page number, for example, 1 first chapter 10 1.1 first section 11 1.1.1 first subsection 12 1.2 ...
3
votes
3answers
2k views

In awk, how to print a range of fields with the original $0 delimiters intact?

How to extract a substring of $0, in awk, when that substring's position and length is based on the positions of fields in the original $0? It effectively equates to removing the first n.lead fields ...
3
votes
2answers
289 views

pulling strings out of mixed file of strings and binary

I'm trying to get my nokia backup file into an acceptable vcard format. It has valid vcard strings that appear in between binary data. For example: !@#JUNKCHARACTERS!@#!@BEGIN:VCARD VERSION:2.1 ...
3
votes
4answers
456 views

Shell script or command to Strip text and Binary files excerpts from large Log Files

I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the ...
6
votes
2answers
3k views

awk: forcing a return status?

This is a followup to my earlier question. I am validating the number of fields in /etc/passwd using this handy snippit. In the following example, the users 'fieldcount1' and 'fieldcount2' have the ...
3
votes
2answers
750 views

Remove slashes/parent paths from filenames inside CSS and Javascript content

I need to go through all my css and js files and if there is a filename referenced that has any slashes (/) at all then the slash should be removed. What I want is: if any files referenced are ...
2
votes
1answer
314 views

Verify that /etc/passwd has correct number of fields on each line?

I'm looking for an easy, portable way to validate the number of fields in /etc/passwd, /etc/shadow, /etc/group and /etc/master.passwd. This would run on FreeBSD, Linux and other Un*xes. pwck does ...
3
votes
1answer
454 views

reading input in sed

i'm new to shell scripting. I'm learning little by little. I'm more familiar with php which is working toward my disadvantage right now. Anyway, i'm writing a shell script and i read in input with ...
20
votes
6answers
37k views

combine text files column-wise

I have two text files. The first one has content: Languages Recursively enumerable Regular while the second one has content: Minimal automaton Turing machine Finite I want to combine them into ...
3
votes
1answer
160 views

Add <script> to the bottom of a html file?

I have a bunch of HTML files, and I need to add <script src="ga.js"></script> before each search instance of "</body></html>". How would I add that to the bottom of all these ...
3
votes
1answer
146 views

Sample a few lines from an strace printout

I've got a process that I want to inspect with strace, but since it's printing out a lot of stuff, I would like to sample only a few lines every X seconds from it. Is there any way of doing that with ...
12
votes
4answers
1k views

How did they manage to drive a Unix computer before mice and copy & paste?

Since Unix is 40 years old, Unix is older than the invention of the computer mouse. (actually only 3 years if Unix is from 1969 and the mouse from 1972) How in the world did a new user do anything ...
2
votes
1answer
1k views

remove carriage return character from large binary file

I have a large binary file ( nearly binary file ! ) which has many \r\n in it [ gets Generated in windows env and doesnt have controle over generating application ]. I want to remove \r from this ...
3
votes
1answer
513 views

Writing a bash script for apache conf files

I'm trying to automate creating vhosts on my computer. This is merely a learning experience for bash scripting. I'm currently novice. I'm trying to learn more about awk, sed. No, this isn't homework. ...
2
votes
5answers
695 views

How to get this OUTPUT? perl/awk/sed? How?

How can I generate from this INPUT in "general" INPUT (/proc/net/ip_conntrack) udp 17 0 src=192.168.1.128 dst=91.120.112.125 sport=29249 dport=39802 packets=3 bytes=408 [UNREPLIED] ...
2
votes
2answers
583 views

parsing through log file and printing out regex backreferences

I want to parse my /var/log/maillog to print out the email addresses in "from=<>" and "to=<>" (mainly to do a quick check for false positives in the DNS RBLs that I've configured) A maillog ...
3
votes
2answers
731 views

How do remove identical lines in one file from another, using sed?

I have two files, one being a superset of the other. I want to remove the identical lines in the smaller files from the larger file. One possible complication is that the lines contain backslashes. ...
2
votes
1answer
60 views

Replacing the end of a few hundred files

I need to change the way a few hundred Perl modules end. Currently they end as 1; However I need to modify this so that they all have this on their last few lines. BEGIN { # VERSION } 1; # ...
1
vote
3answers
347 views

Sort data on multiple columns

INPUT (unsorted, bad) "XYZ-ZTE-43255 serverB618 agreed","XYZ-ZTE-44432" ,"XYZ-ZTE-43255" "XYZ-ZTE-52775 serverB110 agreed", "XYZ-ZTE-79213 - serverB688 agreed", "XYZ-ZTE-77323 serverB617 agreed", ...