Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

10
votes
1answer
4k views

reformatting output with aligned columns [duplicate]

Possible Duplicate: A shell tool to “tablify” input data I have a program outputting something like this: abc defgh ijklm nopqr stu vw xyza bcde fghi which I'd like ...
2
votes
3answers
6k views

How to check if a file is tab-delimited and has 8 columns?

I have two things I need to check a text file for: If it is tab-delimited If it has exactly 8 columns How can I check those things and show an error message if either is false?
8
votes
3answers
6k views

How do I grep without leading whitespaces?

I'm greping through a large codebase, and leading whitespaces and tabulation seem to be quite annoying. Is there any way to get rid of it? grep -R "something" ./ E.g, instead of: foo/bar.cpp: ...
2
votes
1answer
378 views

Possible to extract title and pagenum of each page in a pdf file

I was wondering if there are some ways to extract title and pagenum of each page in a pdf file? Either by some applications, or by programming in some programming language with some pdf libraries? ...
2
votes
1answer
579 views

Change the order of displayed fields of arbitrary command output

How could I change the order of fields from a given command? As an example take the output of ls -l. By default the file name is displayed as the last field of the output. What could I pipe that ...
6
votes
4answers
1k views

Compare an old file and new file, but ignore lines which only exist in new file?

I have two files: oldlist - This contains a list of files and a md5 hash for each file. This was generated one year ago. newlist - This also contains a list of files and a md5 hash for each file. ...
9
votes
2answers
1k views

Why does a bash here-string add a trailing newline char?

The following examples show that a newline is added to a here-string. Why is this done? xxd -p <<<'a' # output: 610a xxd -p <<<'a ' # output: 610a0a
3
votes
1answer
183 views

Help improving my AWK skills

This program works as I intended but I feel like I used some clumsy methods to get the out put I desired(especially with my use of print commands and variable declarations) Could some one improve ...
5
votes
1answer
363 views

merge that works at word granularity instead of line

The old diff, patch, and merge programs, and their modern descendents in the form of version control systems work great for managing source code that can be edited concurrently by multiple people. ...
3
votes
5answers
2k views

How to replace a list of placeholders in a text file?

I have a text (configuration) file, but the program that reads the file unfortunately doesn't allow using any kind of variables. So I'd like to use a preprocessor that replaces a set of placeholders ...
0
votes
1answer
1k views

Convert text to image in centos from bash

I want to create an image [jpg, png, doesn't matter] from shell but I don't have convert command. The image is very simple - white background with a medium-size black text in the middle. Is it ...
8
votes
1answer
3k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
4
votes
4answers
1k views

How can I remove a specific string from a file only if there are other lines in the file?

How can I remove a specific string from a file ONLY if there are other lines in the file? For example, don't touch this file: cat file.txt ASDF but remove "ASDF" from this file: cat file.txt ASDF ...
3
votes
6answers
5k views

Converting DOS text files to Unix

I have a bunch of PHP scripts that are in DOS format (vi displayed [noeol] [dos] in the status line) and as expected they weren't usable. I ran dos2unix over them to fix the line endins: dos2unix ...
2
votes
1answer
2k views

Enable line breaks when assigning multi line outputs to var in bash

I'm using the following very basic shell script I copied off the net to list the contents of my database backup folder and mail them regularily: MYSQLLIST=$(ls -lhG /var/backups/mysql/daily/) ...
6
votes
2answers
4k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...
12
votes
5answers
45k views

How to strip multipe spaces to one using sed?

sed on AIX is not doing what I think it should. I'm trying to replace multiple spaces with a single space in the output of IOSTAT: # iostat System configuration: lcpu=4 drives=8 paths=2 vdisks=0 ...
7
votes
2answers
1k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
43
votes
4answers
22k views

Why is `while IFS= read` used so often, instead of `IFS=; while read..`?

It seems that normal practice would put the setting of IFS outside the while loop in order to not repeat setting it for each iteration... Is this just a habitual "monkey see, monkey do" style, as it ...
26
votes
5answers
11k views

How to obtain inverse behavior for `tail` and `head`?

Is there a way to head/tail a document and get the reverse output; because you don't know how many lines there are in a document? I.e. I just want to get everything but the first 2 lines of foo.txt ...
28
votes
17answers
63k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
6
votes
3answers
3k views

Command like `column -t` that instead keeps separators in output

I'm editing a simple table. I would like to have it nicely formatted. While I could use tbl, latex, or similar, this seems overkill -- plain text really is sufficient. As it's simple I might as ...
2
votes
2answers
712 views

How to temporarily disable the Escape key in a Terminal?

I am parsing keyboard input via read -n 1 and I haven't found a way to distinguish between an actual key-press of the Escape key and the first Escape character (byte) of a control sequence generated ...
4
votes
3answers
2k views

Read and process a string, char by char, yet allow user to simple line edit the input

I want to incrementally read a line of input from the terminal, and allow the user some basic line editing functionality; INS, DEL, RIGHT, LEFT HOME, END, BACKSPACE Each time the string is ...
8
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
2
votes
1answer
327 views

rotating buffer type file logging utility

I'am trying to debug a server that throws a lot of logs on stdout/stderr. I need to redirect only last N lines to a file. Something like a rotating buffer feature there in tcpdump's -C & -W flags. ...
8
votes
3answers
2k views

Is it possible to nest a 'find -exec' within another 'find -exec'?

Something like the following is what I what I'm after, but my code doesn't work, no matter how I escape {} and + ; find ./ -maxdepth 1 -type d -name '.*' -exec \ find {} -maxdepth 1 -type f ...
3
votes
2answers
6k views

Replace all occurrences of a character in a variable using korn script

Part of the Korn script I am writing requires that I replace all occurrences of the ' character with two occurrences (''). I am trying to log some SQL that I am generating in this script to a column ...
6
votes
2answers
155 views

What is the connection between a gedit bug and a Unix-&-Linux Q/A href?

While answering a Unix-&-Linux question, I observed that Gedit and two other editors, Leafpad and Medit (I tested 12 editors altogether) exhibit the a certain bug. As it turns out, the bug is ...
8
votes
5answers
2k views

Why does shell Command Substitution gobble up a trailing newline char?

As per the following example, and as in my recent question In bash, where has the trailing newline char gone?, I want to know "why" it happens x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p # ...
10
votes
2answers
1k views

Where has the trailing newline char gone from my command substitution?

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 ...
2
votes
1answer
235 views

Rename beginning of file name using specific text in the file itself

I have many files in directory (file1.txt, file2.txt, ...) Every files contains bunch of lines. One of the line is specific text thetext followed by random number just like thetext56, thetext21, etc. ...
7
votes
1answer
324 views

Change date “July 29th, 2011” to “20110729”

I have lots of HTML files contains date with format July 29th, 2011 I want to change date format July 29th, 2011 to 20110729, December 9th, 2010 to 20101209, etc. I think sed may works but still ...
2
votes
4answers
6k views

Deleting Lines Containing a Specified String in Different Files

Suppose there are two files in web/ named foo.php and bar.php. The 1st line of foo.php is "sdajgeSTRINGdsad" and the 10th line of bar.php is "gdfhu98324STRING". The task is to first locate these two ...
13
votes
6answers
26k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...
4
votes
2answers
6k views

Show only text between 2 matching pattern

I'm going to submit form using cURL, where some of the contents is come from other file, selected using sed If param1 is line matching pattern from other file using sed, below command will works ...
2
votes
1answer
147 views

Is there a way to get the effect of combining grep -v with grep -A?

Is there a way, either with grep or something else, to (in effect) combine grep -v and grep -A? That is, rather than include everything for x lines after a string, exclude everything. Quick reminder ...
4
votes
2answers
1k views

Where has my `uniq` or `sort -u` line gone, with some unicode characters

What is going on in the following code snippet? I'm not getting my expected output. I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something ...
6
votes
7answers
19k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
1
vote
2answers
546 views

Text processing: convert text of table of content into djvu bookmark format?

Usually in the table of content of a book, each line is for a chapter or section and ended with its page number, for example, 1 first chapter 10 1.1 first section 11 1.1.1 first subsection 12 1.2 ...
3
votes
3answers
1k views

In awk, how to print a range of fields with the original $0 delimiters intact?

How to extract a substring of $0, in awk, when that substring's position and length is based on the positions of fields in the original $0? It effectively equates to removing the first n.lead fields ...
3
votes
2answers
285 views

pulling strings out of mixed file of strings and binary

I'm trying to get my nokia backup file into an acceptable vcard format. It has valid vcard strings that appear in between binary data. For example: !@#JUNKCHARACTERS!@#!@BEGIN:VCARD VERSION:2.1 ...
3
votes
4answers
438 views

Shell script or command to Strip text and Binary files excerpts from large Log Files

I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the ...
5
votes
1answer
3k views

awk: forcing a return status?

This is a followup to my earlier question. I am validating the number of fields in /etc/passwd using this handy snippit. In the following example, the users 'fieldcount1' and 'fieldcount2' have the ...
3
votes
2answers
733 views

Remove slashes/parent paths from filenames inside CSS and Javascript content

I need to go through all my css and js files and if there is a filename referenced that has any slashes (/) at all then the slash should be removed. What I want is: if any files referenced are ...
2
votes
1answer
310 views

Verify that /etc/passwd has correct number of fields on each line?

I'm looking for an easy, portable way to validate the number of fields in /etc/passwd, /etc/shadow, /etc/group and /etc/master.passwd. This would run on FreeBSD, Linux and other Un*xes. pwck does ...
3
votes
1answer
419 views

reading input in sed

i'm new to shell scripting. I'm learning little by little. I'm more familiar with php which is working toward my disadvantage right now. Anyway, i'm writing a shell script and i read in input with ...
20
votes
6answers
33k views

combine text files column-wise

I have two text files. The first one has content: Languages Recursively enumerable Regular while the second one has content: Minimal automaton Turing machine Finite I want to combine them into ...
3
votes
1answer
159 views

Add <script> to the bottom of a html file?

I have a bunch of HTML files, and I need to add <script src="ga.js"></script> before each search instance of "</body></html>". How would I add that to the bottom of all these ...
3
votes
1answer
145 views

Sample a few lines from an strace printout

I've got a process that I want to inspect with strace, but since it's printing out a lot of stuff, I would like to sample only a few lines every X seconds from it. Is there any way of doing that with ...