Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

4
votes
3answers
751 views

Delete the matching line and several more from a file

I have a text file called file_a.txt. My first command is grep -A 12 ".production =" file_a.txt The output is a few block. Each block of string contains 13 rows I specifically want to delete all ...
3
votes
3answers
207 views

Rearrange text file by prepending data rows with header row

I have a file whose content looks like this: 2009 150 0 0 0 0000 75.316 0.0400390625 0.00007 0.00000 0.8980 76.216 0.0400390625 0.00007 1.00000 0.9046 77.217 0.0400390625 ...
3
votes
4answers
150 views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
3
votes
2answers
230 views

tac command's option creates strange output [comprehension question]

Say I have this file, containing nothing but a b c b a Using tac --separator=a file in BASH [on a Debian based Linux], I get this: # empty line # empty line b c ...
2
votes
3answers
234 views

Remove spaces and headers from a dumped database table

I have dumped the database table data into a flat file and below is how the data looks like : (Kindly copy from below ;metier_code ; ;-------------------------; (0 rows affected) ...
2
votes
5answers
2k views

vlookup function in unix

How to make something similar to Excel's vlookup function in Unix? excerpt from office website, VLOOKUP The V in VLOOKUP stands for vertical. Use VLOOKUP instead of HLOOKUP when your comparison ...
2
votes
2answers
6k views

How can I replace text after a specific word using sed?

I have a file named .ignore. In need to replace the projdir. For example: ignore \..* ignore README projdir Snake I need to replace Snake with, for example, "PacMan". I read the man page, but I ...
1
vote
4answers
653 views

Replace text between brackets

I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like: SOMETHING [BAR, WIBBLE] SOMETHING [BAR] This gives the desired result of: SOMETHING [FOO, FOO] SOMETHING [FOO] ...
45
votes
6answers
45k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid off those messages?
20
votes
9answers
44k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
14
votes
13answers
2k views

How to read an IP address backwards?

If we have this string (IP address): 192.168.1.1 How can I derive the (DNS reverse record form) from this string, so it will be shown like 1.1.168.192.in-addr.arpa using a shell script?
12
votes
11answers
8k views

How can I split a text file into multiple text files?

I have a text file called entry.txt that contains the following: [ entry1 ] 1239 1240 1242 1391 1392 1394 1486 1487 1489 1600 1601 1603 1657 1658 1660 2075 2076 2078 2322 2323 2325 2740 2741 2743 ...
17
votes
3answers
3k views

echo a file without the first and last lines

Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
5
votes
5answers
18k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
12
votes
1answer
3k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
11
votes
6answers
19k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...
16
votes
2answers
11k views

How do I remove certain lines (using line numbers) in a file?

There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
14
votes
3answers
13k views

How to find unmatched brackets in a text file?

Today I learned that I can use perl -c filename to find unmatched curly brackets {} in arbitrary files, not necessarily Perl scripts. The problem is, it doesn't work with other types of brackets () [] ...
7
votes
5answers
4k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
14
votes
10answers
2k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
11
votes
1answer
331 views

How to do `head` and `tail` on null-delimited input in bash?

find command can output names of files as a null-delimited strings (if -print0 is provided), and xargs can consume them with -0 option turned on. But in between, it's hard to manipulate that ...
8
votes
4answers
4k views

Efficient in-place header removing for large files using sed?

The commands below may takes minutes depends on the file size. Is there any more effient method? sed -i 1d large_file
7
votes
6answers
2k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
6
votes
2answers
5k views

How do I insert a space every four characters in a long line?

I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...
10
votes
2answers
18k views

How do I delete the first n lines of an ascii file using shell commands?

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
10
votes
4answers
40k views

Finding text between two specific characters or strings

Say I have lines like this: *[234]* *[23]* *[1453]* where * represents any string (except a string of the form [number]). How can I parse these lines with a command line utility and extract the ...
10
votes
2answers
1k views

Where has the trailing newline char gone from my command substitution?

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 ...
9
votes
3answers
2k views

Process last line first using awk

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally. The ...
7
votes
1answer
347 views

Is there a way to make perl -i not clobber symlinks?

A friend of mine points out that if you do: perl -pi.bak -e 's/foo/bar/' somefile when "somefile" is actually a symlink, perl does just what the docs say it will do: It does this by renaming ...
7
votes
2answers
1k views

Parsing XML's , JSON's and newer data file formats in UNIX using command line utilities

Unix environment has some excellent tools for parsing text in various forms. However off late the data is not in the regular set of formats(CSV,TSV,Record based or some other delimiter based) it used ...
5
votes
5answers
5k views

Parsing a delimited text file in bash as command arguments

I have a text file split up like so: field1,field2,field3 xield1,xield2,xield3 dield1,dield2,dield3 gield1,gield2,gield3 Each of these columns will be a parameter to a program, and I would like ...
5
votes
4answers
5k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
4
votes
2answers
170 views

Printing lines from one file if part of them appears in another. Both files are millions of lines long

I have two files, let's call them 123.txt and 789.txt. 123.txt is 2.5M lines long, and 789.txt is 65M lines long. Is there any way to use grep or similar to keep any lines from 789.txt that contain ...
3
votes
1answer
73 views

Find and Replace Recursive

I want to modify every file in a directory recursively. I want to replace: <?php with <?php /** * * Copyright (C) MyCompany, Ltd. - All Rights Reserved * Unauthorized copying of ...
3
votes
5answers
270 views

evaluate multiple patterns from program output and write into pattern specific files

I have a script outputting some value/numbers and I want to split those into two files. I am looking at something like: ./runme.sh | grep 'ook' >> ook.out | grep 'eek' >> eek.out Where ...
3
votes
1answer
3k views

How to numerical sort by last column?

I have this input: sdkxyosl 1 safkls 2 asdf--asdfasxy_asd 5 dkd8k jasd 29 sdi44sw 43 asasd afsdfs 10 rklyasd 4 I need this output: sdi44sw 43 dkd8k jasd 29 asasd afsdfs 10 asdf--asdfasxy_asd 5 ...
13
votes
7answers
1k views

Replace string containing newline in huge file

Anyone know of a non-line-based tool to "binary" search/replace strings in a somewhat memory-efficient way? See this question too. I have a +2GB text file that I would like to process similar to what ...
10
votes
2answers
389 views

/usr/bin/ptx: Can you provide a use case or two?

I was going through the list of files included in coreutils and I was able to come up with an example of how I could personally use all of the commands provided except for ptx. Can you give one or two ...
10
votes
2answers
2k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
9
votes
3answers
446 views

Doing two things with output from a command

I have a program texcount that outputs the number of words in my LaTeX document. I can also pipe the output of this to sed to make the newlines TeX linebreaks and write this to a file which I can then ...
8
votes
4answers
2k views

How to prepend a license header recursively for all .h and .cpp files in a directory

I'm trying to add a license header to all header files and source files in a project directory using a for loop. This is not working, is there any other approach using sed?
8
votes
5answers
478 views

What's the best way to take a segment out of a text file?

What's a good way of extracting say, lines 20 -45 out of a huge text file. Non-interactively of course!
7
votes
3answers
7k views

How do I append text to the beginning and end of multiple text files in Bash?

I have a directory full of text files. My goal is to append text to the beginning and end of all of them. The text that goes at the beginning and end is the same for each file. Based on code I got ...
5
votes
4answers
439 views

How can I “lazily” read output from xrandr?

I have a bash script that I use to adjust my monitor brightness that uses xrandr --verbose to get the current brightness. It works fine, but using xrandr is kind of slow on my machine, as you can see ...
5
votes
4answers
78 views

Search a pattern and print preceding lines starting with another pattern

I need to grep for a pattern and once I find it, I need to look for another pattern preceding that and print all those lines in between those. Better example: 1 a 2 a 3 a 4 a 5 a 6 b 7 c 8 d 9 xyz ...
5
votes
4answers
1k views

Tool to create text files from a template

I have to regularly create 100+ text files from templates. I currently use an overly complicated shell script. I think there is a smarter way to handle this, but I don't know how. I have a "database" ...
4
votes
1answer
376 views

Save HTML from clipboard as markdown text

I want to be able to highlight a section of a web page and copy it to the clipboard then save it to my local disk as markdown. I need an efficient way to do that. My current cumbersome method is: ...
4
votes
4answers
1k views

A unix command to truncate each line of a file

I have a CSV file and I want to truncate it from the third semicolon. For example, if I have this file: 1;foo;bar;baz;x;y;z 2;foo;bar;baz;x;y;z 3;foo;bar;baz;x;y;z I want to get the following ...
4
votes
2answers
791 views

How to display TSV (csv) in console, when empty cells are missed by: `column -t -s $'\t' `

I have file with columns spearated with tab. I have file when some rows have empty cells (on begining, in middle). In such cases column -t -s $'\t' simply fails: Input: $ echo -e ...
4
votes
3answers
2k views

Inserting text at the beginning of a file with sed via the terminal in Linux [duplicate]

Possible Duplicate: How can I prepend a tag to the beginning of several files? How do I insert text at the beginning of a file via terminal?