Manipulation of text by programs, scripts, etc.
5
votes
4answers
249 views
Tool to create text files from a template
I have to regularly create 100+ text files from templates.
I currently use an overly complicated shell script. I think there is a smarter way to handle this, but I don't know how.
I have a "database" ...
2
votes
2answers
159 views
How to read by multiple lines instead of line-by-line
I see that in my code I can modify a line if the entire code is based on that line. Example: <p class="example">this is an example Happy Holidays</p>. However if the </p> tag is ...
3
votes
2answers
117 views
How to add a line in many files
I have many .html files and I need to add a meta tag after <head>'s start tag in each file.
How I can do that?
Can vim help me?
8
votes
4answers
395 views
How do I write a sed one-liner to add a character after every third character?
So, I have a string that looks like this:
AUGGCCAUGGCGCCCAGAACUGAGAUCAAUAGUACCCGUAUUAACGGGUGA
And I want to split the string into 3-character chunks delimited by a '+' sign.
...
9
votes
3answers
351 views
Process last line first using awk
I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally.
The ...
6
votes
6answers
265 views
Review changes in .tex files
I have two files paper.tex and paper_suggested_changes.tex the latter is a modified version of the former with corrections.
I need to go through the two files and choose which changes to accept or ...
5
votes
1answer
128 views
How to remove all instances of a particular symbol from a text file?
I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going ...
2
votes
3answers
149 views
Indent like first line
How can I indent a file such as its first line?
Example:
A file containing
x=1+2+3+4+
5+6+7+8
+9+10+12
should be converted to
x=1+2+3+4+
5+6+7+8
+9+10+12
I need this inside ...
16
votes
1answer
502 views
How to implement a horizontal cat?
Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
3
votes
3answers
464 views
split long line on a delimiter
What command can I use to split input like this:
foo:bar:baz:quux
into this?
foo
bar
baz
quux
I'm trying to figure out the cut command but it seems to only work with fixed amounts of input, like ...
3
votes
2answers
252 views
How to delete commands in history matching a given string?
I need to delete all commands in my history matching a string. I've tried:
$ history | grep searchstring | cut -d" " -f2 | history -d
-bash: history: -d: option requires an argument
$ history | grep ...
2
votes
3answers
240 views
Substitute placeholders in template
Say I have a shell configuration file config like this:
HOST=localhost
PORT=8080
Now I have a template template like this:
The host is <%= @HOST %>
The port is <%= @PORT %>
How do I ...
1
vote
3answers
178 views
Script matching literal pattern over multiple lines?
I have a multi-line string in the variable $PAT. $PAT must be searched for within the file $FILE. If $PAT is in $FILE, it needs to print the file with $PAT removed. If $PAT is not found, then print ...
6
votes
2answers
187 views
How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?
I've got a html file with a lot of %-encoded UTF-8 text in URLs.
For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian).
The task is to replace all ...
3
votes
2answers
118 views
How can I use column to delimit on tabs and not spaces?
I'd like to use Unix column to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I make column ...
7
votes
10answers
505 views
What's a good way to filter a text file to remove empty lines?
I have a .csv file (on a mac) that has a bunch of empty lines, e.g.:
"1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
1
vote
2answers
220 views
Deleting specific lines and duplicates from a 11gb wordlist text file
I have an 11gb wordlist file which is already sorted as each word is on its own line.
I need to remove duplicates and lines starting from 077.
I guess I need to run sed and sort -u together but I ...
2
votes
2answers
79 views
Getting Source as Flatfile and to remove the Carriage returns and \n …special charaters?
I have an issue where I am getting a flatfile source with a huge number of records with PIPE delimiters and one of the fields is getting carriage returns (in multiple lines) and starts with a newline ...
1
vote
4answers
897 views
How to print all lines after a match up to the end of the file?
Input file1 is:
dog 123 4335
cat 13123 23424
deer 2131 213132
bear 2313 21313
I give the match the pattern from in other file ( like dog 123 4335 from file2).
I match the pattern of the line ...
3
votes
2answers
142 views
Extract lines containing PAT1 but not PAT2 with sed
I want to extract the lines containing PAT1 but not PAT2 from a file. For example, with PAT1='dog', PAT2='cat', and the following input:
1 cat chicken
2 bird dog apple
3 dog orange cat
4 cat juice ...
5
votes
2answers
2k views
How to find difference between two timestamps up to milliseconds?
I am new to shell scripting. The heart of my script is to find the difference between two timestamps up to milliseconds. With me I have a file with content of timestamps only as
2012-09-13 ...
2
votes
2answers
220 views
How to traverse a file and extract words using their index?
The given file must not be stored in variable and then traversed due to memory size restrictions:
Example:
var=$(cat FILE)
for i in $var
do
echo $i
done
How do you traverse all strings in a file ...
2
votes
2answers
72 views
add fields from different files
I have two files A and B with the same number of lines. Each of them has multiple columns.
I want to add field 6 in file A with field 3 in file B for each line. e.g.:
file A:
2769 RUT10EKl3fY 0 ...
2
votes
1answer
189 views
intersection of two files according to the first column
I have two files
in file A, there are sequence_numbers
in the other file B, there are many columns, and the first column is sequnce numbers,
I want to get a files with all the lines in the B with the ...
11
votes
2answers
654 views
echo a file without the first and last lines
Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
4
votes
4answers
439 views
A unix command to truncate each line of a file
I have a CSV file and I want to truncate it from the third semicolon. For example, if I have this file:
1;foo;bar;baz;x;y;z
2;foo;bar;baz;x;y;z
3;foo;bar;baz;x;y;z
I want to get the following ...
3
votes
2answers
154 views
Limit stdout from a continuously running process
I haven't had much luck finding an answer to my problem, but maybe I'm not asking for it correctly.
I have a process I startup like the following:
nohup ping 127.0.0.1 > log.txt >2>&1 &
...
0
votes
2answers
159 views
Easiest way to find/replace in a file using a list?
I have a file, foo.txt, and a regexp I want to find in that file. Each time I find the regexp, I want to take a line from another file, bar.txt, and substitute it in for the regexp match I found in ...
0
votes
1answer
259 views
Remove a block of lines between two patterns [duplicate]
Possible Duplicate:
Show only text between 2 matching pattern
In a bash script using sed how can I remove a block of lines beginning with -pattern a- and ending with -pattern b- where the ...
3
votes
3answers
185 views
Delete a block of lines with a condition on the last line
I have a log-file where at the end of a series of lines you can see if this block is relevant.
Now I'm looking for a command like sed to delete de blocks ending with "Content-Length: 0" and beginning ...
4
votes
3answers
261 views
Find files that have words in common
What would be the best way to create a list of files that have common words with a given file. For example, if I had:
$ ls
mainFile file1 file2 file file4
$ cat mainFile
exquisite malicious ...
1
vote
1answer
619 views
How to add header and footer to the flat file
I need to add a header and a footer to flat file - I just need to get values from the parameter like system date and report run date in the top line and for the footer I need to get the record count ...
5
votes
2answers
311 views
Replacing missing value blank space with zero
I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt
...
7
votes
4answers
316 views
Creating a single output stream out of three other streams produced in parallel
I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format.
This Python script is slow and CPU-bound (to a ...
2
votes
4answers
188 views
How to do an IF statement from the result of a executed command
I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this?
if [ "`netstat -lnp | ...
1
vote
4answers
335 views
Replace text between brackets
I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like:
SOMETHING [BAR, WIBBLE]
SOMETHING [BAR]
This gives the desired result of:
SOMETHING [FOO, FOO]
SOMETHING [FOO]
...
3
votes
2answers
273 views
Skip first 3 byte of a file
I am using AIX 6.1 ksh shell
I want to use one liner to do something like this
cat A_FILE | skip-first-3-bytes-of-the-file
I want to skip the first 3 bytes of the first line, is there a way to ...
9
votes
4answers
1k views
Delete last line from the file
I use sed to quickly delete lines with specific position as
sed '1d'
sed '5d'
But, what if I want to delete the last line of the file and I don't know the count of lines (I know I can get that ...
6
votes
2answers
282 views
shell script/utility replace 2 byte unicode in binary file
I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains
"C3 B9 C3 AB C3 B8 C3 B0"
and I would like to ...
2
votes
3answers
77 views
Print only lines that are completely numeric
I'd like to filter through a text file and only print the lines where each column is a valid floating point number. For example:
3 6 2 -4.2 21.2
3 x 4.2 21.2
3 2 2.2.2
Only the first line would ...
2
votes
3answers
210 views
Extract integers enclosed in round brackets
I'm new to Linux. I've a file which contains list of movie names and their release year. I want to list out all release years. Specifically my file looks like below
TDKR(2012)
Vicky Cristina ...
8
votes
2answers
950 views
How to print only the duplicate values from a text file?
Suppose there is a column of numeric values like following:
File1:
1
2
3
3
3
4
4
4
5
6
I want the output:
3
4
That is, only the repeated lines. Is there any command line tools to find this ...
1
vote
1answer
153 views
Join problem: throwing error, join extra operand
I want to join 3 files on a column which has sorted unique numeric values (those files have only one column of values though) and starts with same prefix for an example "usi".
Now, while I am doing ...
1
vote
1answer
652 views
Text processing - Find text before/after given word
I would like to extract contents between Station Name and Data End of the file:
Structure of text file for clarification:
---- Huge text -- not interested
Station Name
100 - 200 lines
Data End
---- ...
6
votes
6answers
800 views
Text processing - join lines with commas
I have more than 1000 lines in a file. The file starts as follows (line numbers added):
1. Station Name
2. Station Code
3. A N DEV NAGAR
4. ACND
5. ABHAIPUR
6. AHA
7. ABOHAR
8. ABS
9. ABU ROAD
10. ...
3
votes
1answer
734 views
Identify the number of unique values and then the number of occurrences of those values in the file
I have a data file which contains 15000 lines, but only 400 unique values. I am looking for a way to identify the number of unique values and then the number of occurrences of those values in the ...
4
votes
3answers
248 views
Delete the matching line and several more from a file
I have a text file called file_a.txt.
My first command is
grep -A 12 ".production =" file_a.txt
The output is a few block.
Each block of string contains 13 rows
I specifically want to delete all ...
2
votes
2answers
98 views
How can I enhance the output of find and grep?
I really don't look forward to having to do find/grep because the output, as returned by
find . -exec grep sometext {} \; -print
is just not very easy to read even when you dump it in a file. What ...
0
votes
2answers
106 views
Is it against linux ethos to provide a progress indicator for running commands? [duplicate]
Possible Duplicate:
Why doesn't cp have a progress bar like wget?
Lately I have been dealing with rather large files in Linux. I often use commands like sed, cut, sort, uniq, awk, and ...
4
votes
1answer
79 views
Is there a command like head but for line truncation?
Linux's head selects a number of lines from the top of a file. Is there a command that selects a number of characters from the front of a line?
The use case behind the question is this: I want to ...
