Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

3
votes
1answer
144 views

Sample a few lines from an strace printout

I've got a process that I want to inspect with strace, but since it's printing out a lot of stuff, I would like to sample only a few lines every X seconds from it. Is there any way of doing that with ...
12
votes
4answers
1k views

How did they manage to drive a Unix computer before mice and copy & paste?

Since Unix is 40 years old, Unix is older than the invention of the computer mouse. (actually only 3 years if Unix is from 1969 and the mouse from 1972) How in the world did a new user do anything ...
1
vote
1answer
1k views

remove carriage return character from large binary file

I have a large binary file ( nearly binary file ! ) which has many \r\n in it [ gets Generated in windows env and doesnt have controle over generating application ]. I want to remove \r from this ...
3
votes
1answer
473 views

Writing a bash script for apache conf files

I'm trying to automate creating vhosts on my computer. This is merely a learning experience for bash scripting. I'm currently novice. I'm trying to learn more about awk, sed. No, this isn't homework. ...
2
votes
5answers
690 views

How to get this OUTPUT? perl/awk/sed? How?

How can I generate from this INPUT in "general" INPUT (/proc/net/ip_conntrack) udp 17 0 src=192.168.1.128 dst=91.120.112.125 sport=29249 dport=39802 packets=3 bytes=408 [UNREPLIED] ...
2
votes
2answers
535 views

parsing through log file and printing out regex backreferences

I want to parse my /var/log/maillog to print out the email addresses in "from=<>" and "to=<>" (mainly to do a quick check for false positives in the DNS RBLs that I've configured) A maillog ...
3
votes
2answers
719 views

How do remove identical lines in one file from another, using sed?

I have two files, one being a superset of the other. I want to remove the identical lines in the smaller files from the larger file. One possible complication is that the lines contain backslashes. ...
2
votes
1answer
60 views

Replacing the end of a few hundred files

I need to change the way a few hundred Perl modules end. Currently they end as 1; However I need to modify this so that they all have this on their last few lines. BEGIN { # VERSION } 1; # ...
1
vote
3answers
345 views

Sort data on multiple columns

INPUT (unsorted, bad) "XYZ-ZTE-43255 serverB618 agreed","XYZ-ZTE-44432" ,"XYZ-ZTE-43255" "XYZ-ZTE-52775 serverB110 agreed", "XYZ-ZTE-79213 - serverB688 agreed", "XYZ-ZTE-77323 serverB617 agreed", ...
1
vote
3answers
2k views

Get line numbers for empty lines

I have a large pipe-delimited file where I need to find the line number of all lines where a certain field is empty. I can use cut -d \| -f 6 filename.txt to output just that column. What is a ...
20
votes
6answers
11k views

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
3
votes
2answers
657 views

Cut on both characters and field

I have a record like : 2011-05-29 17:51:34 => 'HS|CMGC|RN431|CI13950|CH7-4a37-afe2-acabfc9d262d|DA110529|TI175133|' I want my final output to be something like: 2011-05-29 17:51:34 CI13950 I ...
2
votes
1answer
408 views

Search and replace text within multiple files

I am migrating my blog, and all the posts have metadata at the top of the file. As a result I have a group of nearly 100 files in a directory where I want to do a search and replace for: ^permalink: ...
3
votes
1answer
651 views

Search and replace in multiple files based on condition

I am porting a Visual Studio C++ project to Linux, and would like to find all lines in all .h and .cpp files that start with #include and in any of those lines would like to replace the backslash with ...
3
votes
2answers
744 views

Search & remove text in files from a source file

I want to search for and remove a block text from several files. The block of text to be matched against is in file, say /home/user/myblock.txt I want to parse the directory /home/user/rep and ...
12
votes
3answers
7k views

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
10
votes
5answers
7k views

How do I recursively grep through compressed archives?

I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files ...
45
votes
4answers
27k views

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
5
votes
1answer
115 views

Search for a text in Linux but don't display the complete line in results

I'm using grep to do this, like: grep someText file.log But this returns the complete line in results. If someText is found in the middle of a line I don't want to ouput the whole line, just the ...
2
votes
2answers
168 views

Need a script that generates files from words within a single text files

I have the following scenario and my lack of linux knowledge doesn't help a lot. I have a text file that contains YML values, in the following matters: coolregion: min: {z: -99613.0, y: 45.0, x: ...
7
votes
2answers
969 views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
14
votes
6answers
8k views

Best way to follow a log and execute a command when some text appears in the log

I have a server log that outputs a specific line of text into its log file when the server is up. I want to execute a command once the server is up, and hence do something like the following: tail -f ...
19
votes
4answers
7k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...
3
votes
3answers
663 views

Sed: how to replace nextline \n symbol in text files?

I need to fix an error and to replace the second tag </time> with </tags> in an XML file with the following structure: <time>20260664</time> <tags>substancesummit ...
22
votes
6answers
11k views

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
4
votes
2answers
1k views

Split a line into columns with some of the columns right-justified?

Is there an easy way to split comma-delimited lines of text into columns with some of the columns right-justified? As a bonus, it would be nice to format numbers, but this probably isn't too hard ...
4
votes
2answers
12k views

Replace all white spaces with commas in a text file

I need to replace all white spaces inside my text with commas. I'm currently using this line but it doesn't work: I get as output a text file which is exactly the same of the original one: sed ...
2
votes
1answer
460 views

Single command to move a domain from cPanel's localdomains to remotedomains file?

I find myself needing to do this a few times daily and would love an easier way to do it. The domain example.com's email is handled by a third party, so the server needs to be told this by listing ...
0
votes
1answer
55 views

Writing to picture which is scanned document

I have a scanned contract and I need to change only a few names and dates in the contract. It's easy to scan the document but impossible to ocr the document and open in *.doc format. Is there an ...
8
votes
2answers
18k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same file in Jedit, it warns me of invalid UTF-8 encoding... The html meta tag states ...
1
vote
1answer
656 views

How to convert <a href=“http://xy.com”>XY</a> to [http://xy.com XY]?

Just for editing Wiki's.. So i have this input [from a larget text]: TEXT TEXT BLA-BLA RANDOMSTRINGS BLA-BLA TEXT TEXT BLA-BLA RANDOMSTRINGS BLA-BLA TEXT TEXT <a ...
3
votes
5answers
2k views

Replacing text from a list of replacements. Added complication: backslashes

I have a file A that contains pairs of strings, one per line: \old1 \new1 \old2 \new2 ..... I would like to iterate over file A, and for each line perform the replacement (e.g. "\old1 -> \new1") ...
4
votes
1answer
3k views

Why is 'while.. read.. <<EOL' performing variable expansion, yet <file and | don't?

Aside from the variable expansion mentioned in the question's heading, I also experience another alarming issue when reading inline data to <<EOL... When the data contains a backtick `, it ...
3
votes
2answers
793 views

How can I count lines with title-case words of certain length?

How can I create a shell script in unix/linux to find the number of lines in a text file that have words which are 5 to 10 character long and begin with a capital letter?
35
votes
8answers
6k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
16
votes
3answers
12k views

How do I remove certain lines (using line numbers) in a file?

There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
7
votes
6answers
2k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
30
votes
7answers
6k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
18
votes
5answers
22k views

Shell: How to read the bytes of a binary file and print as hexadecimal?

In shell, how can I read the bytes of a binary file I have, and print the output as hexadecimal numbers?
8
votes
4answers
18k views

How to use bash script to read binary file content?

I want to read a character and then a fixed length of string (the string is not null terminated in the file, and its length is given by the preceding character). How can I do this in a bash script? ...
4
votes
2answers
4k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
4
votes
1answer
3k views

How to numerical sort by last column?

I have this input: sdkxyosl 1 safkls 2 asdf--asdfasxy_asd 5 dkd8k jasd 29 sdi44sw 43 asasd afsdfs 10 rklyasd 4 I need this output: sdi44sw 43 dkd8k jasd 29 asasd afsdfs 10 asdf--asdfasxy_asd 5 ...
4
votes
1answer
283 views

sed : insert sth before subsequent lines that begin the same, but are not the same

I have a LaTeX file with one glossary entry per line: ... \newglossaryentry{ajahn}{name=Ajahn,description={\textit{(Thai)} From the Pali \textit{achariya}, a Buddhist monk's preceptor: `teacher'; ...
1
vote
3answers
336 views

Switch from grep to sed

I have a files students.txt, with lines of the form: Surname, Forename: Day.Month.Year: Degree For example: Smith, John: 15.01.1986: MSc IT Taylor, Susan: 04.05.1987: MSc IT Thomas, Steve: ...
2
votes
3answers
3k views

How can I loop through lines of a file and find files matching each line?

In a BASH shell, I would like to take the lines of a file (eg pattern.txt) and find the files on my system whose names contain the patterns in each line of my file. So, I have the following for loop ...
7
votes
4answers
327 views

Getting text from last marker to EOF in POSIX.2

I have a text with marker lines like: aaa --- bbb --- ccc I need to get a text from the last marker (not inclusive) to EOF. In this case it will be ccc Is there an elegant way within POSIX.2? ...
15
votes
3answers
18k views

How to find unmatched brackets in a text file?

Today I learned that I can use perl -c filename to find unmatched curly brackets {} in arbitrary files, not necessarily Perl scripts. The problem is, it doesn't work with other types of brackets () [] ...
27
votes
3answers
12k views

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
6
votes
5answers
22k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
1
vote
1answer
857 views

Keep newlines in output of cut

Let's imagine we have a file: foo bar foo1 bar1 I want to cut every second word in every line, so I want file to look like this: foo foo1 I am trying to use cut -d ' ' -f -1 but what I get is: ...