Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

6
votes
4answers
2k views

Tool to create text files from a template

I have to regularly create 100+ text files from templates. I currently use an overly complicated shell script. I think there is a smarter way to handle this, but I don't know how. I have a "database" ...
2
votes
2answers
297 views

How to read by multiple lines instead of line-by-line

I see that in my code I can modify a line if the entire code is based on that line. Example: <p class="example">this is an example Happy Holidays</p>. However if the </p> tag is ...
3
votes
2answers
164 views

How to add a line in many files

I have many .html files and I need to add a meta tag after <head>'s start tag in each file. How I can do that? Can vim help me?
8
votes
4answers
3k views

How do I write a sed one-liner to add a character after every third character?

So, I have a string that looks like this: AUGGCCAUGGCGCCCAGAACUGAGAUCAAUAGUACCCGUAUUAACGGGUGA And I want to split the string into 3-character chunks delimited by a '+' sign. ...
9
votes
3answers
3k views

Process last line first using awk

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally. The ...
8
votes
7answers
358 views

Review changes in .tex files

I have two files paper.tex and paper_suggested_changes.tex the latter is a modified version of the former with corrections. I need to go through the two files and choose which changes to accept or ...
5
votes
1answer
478 views

How to remove all instances of a particular symbol from a text file?

I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going ...
2
votes
3answers
228 views

Indent like first line

How can I indent a file such as its first line? Example: A file containing x=1+2+3+4+ 5+6+7+8 +9+10+12 should be converted to x=1+2+3+4+ 5+6+7+8 +9+10+12 I need this inside ...
20
votes
1answer
1k views

How to implement a horizontal cat? [duplicate]

Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
5
votes
3answers
14k views

split long line on a delimiter

What command can I use to split input like this: foo:bar:baz:quux into this? foo bar baz quux I'm trying to figure out the cut command but it seems to only work with fixed amounts of input, like ...
8
votes
4answers
3k views

How to delete commands in history matching a given string?

I need to delete all commands in my history matching a string. I've tried: $ history | grep searchstring | cut -d" " -f2 | history -d -bash: history: -d: option requires an argument $ history | grep ...
2
votes
3answers
2k views

Substitute placeholders in template

Say I have a shell configuration file config like this: HOST=localhost PORT=8080 Now I have a template template like this: The host is <%= @HOST %> The port is <%= @PORT %> How do I ...
3
votes
3answers
688 views

Script matching literal pattern over multiple lines?

I have a multi-line string in the variable $PAT. $PAT must be searched for within the file $FILE. If $PAT is in $FILE, it needs to print the file with $PAT removed. If $PAT is not found, then print ...
6
votes
3answers
901 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
9
votes
2answers
4k views

How can I use column to delimit on tabs and not spaces?

I'd like to use Unix column to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I make column ...
8
votes
10answers
2k views

What's a good way to filter a text file to remove empty lines?

I have a .csv file (on a mac) that has a bunch of empty lines, e.g.: "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
2
votes
2answers
568 views

Deleting specific lines and duplicates from a 11gb wordlist text file

I have an 11gb wordlist file which is already sorted as each word is on its own line. I need to remove duplicates and lines starting from 077. I guess I need to run sed and sort -u together but I ...
3
votes
3answers
495 views

Getting Source as Flatfile and to remove the Carriage returns and \n …special charaters?

I have an issue where I am getting a flatfile source with a huge number of records with PIPE delimiters and one of the fields is getting carriage returns (in multiple lines) and starts with a newline ...
13
votes
10answers
16k views

How to print all lines after a match up to the end of the file?

Input file1 is: dog 123 4335 cat 13123 23424 deer 2131 213132 bear 2313 21313 I give the match the pattern from in other file ( like dog 123 4335 from file2). I match the pattern of the line ...
3
votes
2answers
439 views

Extract lines containing PAT1 but not PAT2 with sed

I want to extract the lines containing PAT1 but not PAT2 from a file. For example, with PAT1='dog', PAT2='cat', and the following input: 1 cat chicken 2 bird dog apple 3 dog orange cat 4 cat juice ...
7
votes
2answers
8k views

How to find difference between two timestamps up to milliseconds?

I am new to shell scripting. The heart of my script is to find the difference between two timestamps up to milliseconds. With me I have a file with content of timestamps only as 2012-09-13 ...
2
votes
2answers
698 views

How to traverse a file and extract words using their index?

The given file must not be stored in variable and then traversed due to memory size restrictions: Example: var=$(cat FILE) for i in $var do echo $i done How do you traverse all strings in a file ...
3
votes
2answers
232 views

add fields from different files

I have two files A and B with the same number of lines. Each of them has multiple columns. I want to add field 6 in file A with field 3 in file B for each line. e.g.: file A: 2769 RUT10EKl3fY 0 ...
4
votes
1answer
2k views

intersection of two files according to the first column

I have two files in file A, there are sequence_numbers in the other file B, there are many columns, and the first column is sequnce numbers, I want to get a files with all the lines in the B with the ...
22
votes
5answers
12k views

echo a file without the first and last lines

Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
5
votes
4answers
2k views

A unix command to truncate each line of a file

I have a CSV file and I want to truncate it from the third semicolon. For example, if I have this file: 1;foo;bar;baz;x;y;z 2;foo;bar;baz;x;y;z 3;foo;bar;baz;x;y;z I want to get the following ...
3
votes
2answers
610 views

Limit stdout from a continuously running process

I haven't had much luck finding an answer to my problem, but maybe I'm not asking for it correctly. I have a process I startup like the following: nohup ping 127.0.0.1 > log.txt >2>&1 & ...
0
votes
2answers
384 views

Easiest way to find/replace in a file using a list?

I have a file, foo.txt, and a regexp I want to find in that file. Each time I find the regexp, I want to take a line from another file, bar.txt, and substitute it in for the regexp match I found in ...
0
votes
1answer
607 views

Remove a block of lines between two patterns [duplicate]

Possible Duplicate: Show only text between 2 matching pattern In a bash script using sed how can I remove a block of lines beginning with -pattern a- and ending with -pattern b- where the ...
3
votes
4answers
698 views

Delete a block of lines with a condition on the last line

I have a log-file where at the end of a series of lines you can see if this block is relevant. Now I'm looking for a command like sed to delete de blocks ending with "Content-Length: 0" and beginning ...
4
votes
3answers
1k views

Find files that have words in common

What would be the best way to create a list of files that have common words with a given file. For example, if I had: $ ls mainFile file1 file2 file file4 $ cat mainFile exquisite malicious ...
2
votes
1answer
4k views

How to add header and footer to the flat file

I need to add a header and a footer to flat file - I just need to get values from the parameter like system date and report run date in the top line and for the footer I need to get the record count ...
6
votes
2answers
4k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
8
votes
4answers
1k views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
5
votes
4answers
12k views

How to do an IF statement from the result of a executed command

I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this? if [ "`netstat -lnp | ...
1
vote
4answers
1k views

Replace text between brackets

I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like: SOMETHING [BAR, WIBBLE] SOMETHING [BAR] This gives the desired result of: SOMETHING [FOO, FOO] SOMETHING [FOO] ...
4
votes
2answers
3k views

Skip first 3 byte of a file

I am using AIX 6.1 ksh shell I want to use one liner to do something like this cat A_FILE | skip-first-3-bytes-of-the-file I want to skip the first 3 bytes of the first line, is there a way to ...
16
votes
4answers
24k views

Delete last line from the file

I use sed to quickly delete lines with specific position as sed '1d' sed '5d' But, what if I want to delete the last line of the file and I don't know the count of lines (I know I can get that ...
6
votes
2answers
997 views

shell script/utility replace 2 byte unicode in binary file

I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains "C3 B9 C3 AB C3 B8 C3 B0" and I would like to ...
3
votes
3answers
294 views

Print only lines that are completely numeric

I'd like to filter through a text file and only print the lines where each column is a valid floating point number. For example: 3 6 2 -4.2 21.2 3 x 4.2 21.2 3 2 2.2.2 Only the first line would ...
2
votes
3answers
637 views

Extract integers enclosed in round brackets

I'm new to Linux. I've a file which contains list of movie names and their release year. I want to list out all release years. Specifically my file looks like below TDKR(2012) Vicky Cristina ...
16
votes
3answers
23k views

How to print only the duplicate values from a text file?

Suppose there is a column of numeric values like following: File1: 1 2 3 3 3 4 4 4 5 6 I want the output: 3 4 That is, only the repeated lines. Is there any command line tools to find this ...
1
vote
1answer
663 views

Join problem: throwing error, join extra operand

I want to join 3 files on a column which has sorted unique numeric values (those files have only one column of values though) and starts with same prefix for an example "usi". Now, while I am doing ...
1
vote
1answer
2k views

Text processing - Find text before/after given word

I would like to extract contents between Station Name and Data End of the file: Structure of text file for clarification: ---- Huge text -- not interested Station Name 100 - 200 lines Data End ---- ...
16
votes
7answers
12k views

Text processing - join every two lines with commas

I have more than 1000 lines in a file. The file starts as follows (line numbers added): 1. Station Name 2. Station Code 3. A N DEV NAGAR 4. ACND 5. ABHAIPUR 6. AHA 7. ABOHAR 8. ABS 9. ABU ROAD 10. ...
3
votes
1answer
6k views

Identify the number of unique values and then the number of occurrences of those values in the file

I have a data file which contains 15000 lines, but only 400 unique values. I am looking for a way to identify the number of unique values and then the number of occurrences of those values in the ...
4
votes
3answers
917 views

Delete the matching line and several more from a file

I have a text file called file_a.txt. My first command is grep -A 12 ".production =" file_a.txt The output is a few block. Each block of string contains 13 rows I specifically want to delete all ...
2
votes
2answers
232 views

How can I enhance the output of find and grep?

I really don't look forward to having to do find/grep because the output, as returned by find . -exec grep sometext {} \; -print is just not very easy to read even when you dump it in a file. What ...
0
votes
2answers
266 views

Is it against linux ethos to provide a progress indicator for running commands? [duplicate]

Possible Duplicate: Why doesn't cp have a progress bar like wget? Lately I have been dealing with rather large files in Linux. I often use commands like sed, cut, sort, uniq, awk, and ...
6
votes
1answer
179 views

Is there a command like head but for line truncation?

Linux's head selects a number of lines from the top of a file. Is there a command that selects a number of characters from the front of a line? The use case behind the question is this: I want to ...