Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

3
votes
4answers
481 views

Shell script or command to Strip text and Binary files excerpts from large Log Files

I need to remove a large binary file (PDF file) from a large log file which is generated daily.This is seriously hogging space on our servers.I need to remove the large PDF from the logs to make the ...
6
votes
2answers
3k views

awk: forcing a return status?

This is a followup to my earlier question. I am validating the number of fields in /etc/passwd using this handy snippit. In the following example, the users 'fieldcount1' and 'fieldcount2' have the ...
3
votes
2answers
771 views

Remove slashes/parent paths from filenames inside CSS and Javascript content

I need to go through all my css and js files and if there is a filename referenced that has any slashes (/) at all then the slash should be removed. What I want is: if any files referenced are ...
2
votes
1answer
326 views

Verify that /etc/passwd has correct number of fields on each line?

I'm looking for an easy, portable way to validate the number of fields in /etc/passwd, /etc/shadow, /etc/group and /etc/master.passwd. This would run on FreeBSD, Linux and other Un*xes. pwck does ...
3
votes
1answer
493 views

reading input in sed

i'm new to shell scripting. I'm learning little by little. I'm more familiar with php which is working toward my disadvantage right now. Anyway, i'm writing a shell script and i read in input with ...
22
votes
6answers
42k views

combine text files column-wise

I have two text files. The first one has content: Languages Recursively enumerable Regular while the second one has content: Minimal automaton Turing machine Finite I want to combine them into ...
3
votes
1answer
161 views

Add <script> to the bottom of a html file?

I have a bunch of HTML files, and I need to add <script src="ga.js"></script> before each search instance of "</body></html>". How would I add that to the bottom of all these ...
3
votes
1answer
149 views

Sample a few lines from an strace printout

I've got a process that I want to inspect with strace, but since it's printing out a lot of stuff, I would like to sample only a few lines every X seconds from it. Is there any way of doing that with ...
12
votes
4answers
1k views

How did they manage to drive a Unix computer before mice and copy & paste?

Since Unix is 40 years old, Unix is older than the invention of the computer mouse. (actually only 3 years if Unix is from 1969 and the mouse from 1972) How in the world did a new user do anything ...
2
votes
1answer
1k views

remove carriage return character from large binary file

I have a large binary file ( nearly binary file ! ) which has many \r\n in it [ gets Generated in windows env and doesnt have controle over generating application ]. I want to remove \r from this ...
3
votes
1answer
542 views

Writing a bash script for apache conf files

I'm trying to automate creating vhosts on my computer. This is merely a learning experience for bash scripting. I'm currently novice. I'm trying to learn more about awk, sed. No, this isn't homework. ...
2
votes
5answers
705 views

How to get this OUTPUT? perl/awk/sed? How?

How can I generate from this INPUT in "general" INPUT (/proc/net/ip_conntrack) udp 17 0 src=192.168.1.128 dst=91.120.112.125 sport=29249 dport=39802 packets=3 bytes=408 [UNREPLIED] ...
3
votes
2answers
639 views

Parsing through log file and printing out regex backreferences

I want to parse my /var/log/maillog to print out the email addresses in from=<> and to=<> (mainly to do a quick check for false positives in the DNS RBLs that I've configured) A maillog ...
3
votes
2answers
744 views

How do remove identical lines in one file from another, using sed?

I have two files, one being a superset of the other. I want to remove the identical lines in the smaller files from the larger file. One possible complication is that the lines contain backslashes. ...
2
votes
1answer
60 views

Replacing the end of a few hundred files

I need to change the way a few hundred Perl modules end. Currently they end as 1; However I need to modify this so that they all have this on their last few lines. BEGIN { # VERSION } 1; # ...
1
vote
3answers
353 views

Sort data on multiple columns

INPUT (unsorted, bad) "XYZ-ZTE-43255 serverB618 agreed","XYZ-ZTE-44432" ,"XYZ-ZTE-43255" "XYZ-ZTE-52775 serverB110 agreed", "XYZ-ZTE-79213 - serverB688 agreed", "XYZ-ZTE-77323 serverB617 agreed", ...
1
vote
3answers
3k views

Get line numbers for empty lines

I have a large pipe-delimited file where I need to find the line number of all lines where a certain field is empty. I can use cut -d \| -f 6 filename.txt to output just that column. What is a ...
22
votes
6answers
14k views

Removing control chars (including console codes / colours) from script output

I can use the "script" command to record an interactive session at the command line. However, this includes all control characters and colour codes. I can remove control characters (like backspace) ...
3
votes
2answers
800 views

Cut on both characters and field

I have a record like : 2011-05-29 17:51:34 => 'HS|CMGC|RN431|CI13950|CH7-4a37-afe2-acabfc9d262d|DA110529|TI175133|' I want my final output to be something like: 2011-05-29 17:51:34 CI13950 I ...
2
votes
1answer
411 views

Search and replace text within multiple files

I am migrating my blog, and all the posts have metadata at the top of the file. As a result I have a group of nearly 100 files in a directory where I want to do a search and replace for: ^permalink: ...
3
votes
1answer
727 views

Search and replace in multiple files based on condition

I am porting a Visual Studio C++ project to Linux, and would like to find all lines in all .h and .cpp files that start with #include and in any of those lines would like to replace the backslash with ...
4
votes
2answers
789 views

Search & remove text in files from a source file

I want to search for and remove a block text from several files. The block of text to be matched against is in file, say /home/user/myblock.txt I want to parse the directory /home/user/rep and ...
13
votes
3answers
9k views

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
10
votes
5answers
7k views

How do I recursively grep through compressed archives?

I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files ...
16
votes
6answers
10k views

How can you combine all lines that end with a backslash character?

Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash? For example, given the file: foo bar \ bash \ baz dude \ happy ...
52
votes
4answers
38k views

Can grep output only specified groupings that match?

Say I have a file: # file: 'test.txt' foobar bash 1 bash foobar happy foobar I only want to know what words appear after "foobar", so I can use this regex: "foobar \(\w\+\)" The parenthesis ...
5
votes
1answer
119 views

Search for a text in Linux but don't display the complete line in results

I'm using grep to do this, like: grep someText file.log But this returns the complete line in results. If someText is found in the middle of a line I don't want to ouput the whole line, just the ...
2
votes
2answers
171 views

Need a script that generates files from words within a single text files

I have the following scenario and my lack of linux knowledge doesn't help a lot. I have a text file that contains YML values, in the following matters: coolregion: min: {z: -99613.0, y: 45.0, x: ...
7
votes
2answers
1k views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
17
votes
6answers
11k views

Best way to follow a log and execute a command when some text appears in the log

I have a server log that outputs a specific line of text into its log file when the server is up. I want to execute a command once the server is up, and hence do something like the following: tail -f ...
21
votes
4answers
10k views

How to get only the unique results without having to sort data?

$ cat data.txt aaaaaa aaaaaa cccccc aaaaaa aaaaaa bbbbbb $ cat data.txt | uniq aaaaaa cccccc aaaaaa bbbbbb $ cat data.txt | sort | uniq aaaaaa bbbbbb cccccc $ The result that I need is to display ...
3
votes
3answers
695 views

Sed: how to replace nextline \n symbol in text files?

I need to fix an error and to replace the second tag </time> with </tags> in an XML file with the following structure: <time>20260664</time> <tags>substancesummit ...
24
votes
6answers
13k views

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
4
votes
2answers
2k views

Split a line into columns with some of the columns right-justified?

Is there an easy way to split comma-delimited lines of text into columns with some of the columns right-justified? As a bonus, it would be nice to format numbers, but this probably isn't too hard ...
4
votes
2answers
14k views

Replace all white spaces with commas in a text file

I need to replace all white spaces inside my text with commas. I'm currently using this line but it doesn't work: I get as output a text file which is exactly the same of the original one: sed ...
2
votes
1answer
504 views

Single command to move a domain from cPanel's localdomains to remotedomains file?

I find myself needing to do this a few times daily and would love an easier way to do it. The domain example.com's email is handled by a third party, so the server needs to be told this by listing ...
0
votes
1answer
55 views

Writing to picture which is scanned document

I have a scanned contract and I need to change only a few names and dates in the contract. It's easy to scan the document but impossible to ocr the document and open in *.doc format. Is there an ...
10
votes
2answers
23k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same file in Jedit, it warns me of invalid UTF-8 encoding... The html meta tag states ...
1
vote
1answer
691 views

How to convert <a href=“http://xy.com”>XY</a> to [http://xy.com XY]?

Just for editing Wiki's.. So i have this input [from a larget text]: TEXT TEXT BLA-BLA RANDOMSTRINGS BLA-BLA TEXT TEXT BLA-BLA RANDOMSTRINGS BLA-BLA TEXT TEXT <a ...
4
votes
5answers
2k views

Replacing text from a list of replacements. Added complication: backslashes

I have a file A that contains pairs of strings, one per line: \old1 \new1 \old2 \new2 ..... I would like to iterate over file A, and for each line perform the replacement (e.g. "\old1 -> \new1") ...
4
votes
1answer
3k views

Why is 'while.. read.. <<EOL' performing variable expansion, yet <file and | don't?

Aside from the variable expansion mentioned in the question's heading, I also experience another alarming issue when reading inline data to <<EOL... When the data contains a backtick `, it ...
3
votes
2answers
831 views

How can I count lines with title-case words of certain length?

How can I create a shell script in unix/linux to find the number of lines in a text file that have words which are 5 to 10 character long and begin with a capital letter?
38
votes
8answers
8k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
16
votes
3answers
14k views

How do I remove certain lines (using line numbers) in a file?

There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
8
votes
6answers
3k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
31
votes
8answers
7k views

Is there a way to modify a file in-place?

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there ...
18
votes
5answers
28k views

Shell: How to read the bytes of a binary file and print as hexadecimal?

In shell, how can I read the bytes of a binary file I have, and print the output as hexadecimal numbers?
8
votes
4answers
20k views

How to use bash script to read binary file content?

I want to read a character and then a fixed length of string (the string is not null terminated in the file, and its length is given by the preceding character). How can I do this in a bash script? ...
4
votes
2answers
5k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
4
votes
1answer
4k views

How to numerical sort by last column?

I have this input: sdkxyosl 1 safkls 2 asdf--asdfasxy_asd 5 dkd8k jasd 29 sdi44sw 43 asasd afsdfs 10 rklyasd 4 I need this output: sdi44sw 43 dkd8k jasd 29 asasd afsdfs 10 asdf--asdfasxy_asd 5 ...