Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

2
votes
2answers
4k views

Command to find file/path lengths that are too long for burning to DVD?

I'm trying to burn a DVD from Windows but it fails because the full path name length exceeds the limit of something like 255 characters. Our files are stored in Debian Linux (accessed by Windows ...
0
votes
1answer
99 views

Add column based on search term

I have a file a: aaa bbb ccc DDD ads zzz caa DDD aba bab cac DAD aad cca cas SFD and a file b: DDD 5 DAD 13 SFD 11 I want to add a fifth column to file a based on file ...
4
votes
5answers
13k views

removing the first and the last character of every line from command line

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
4
votes
3answers
120 views

Count appearance of a variable in another file

I do have the following situation: File1 looks like: A B C File2 looks like: chr1 \t 1234523 \t A chr3 \t 1234231 \t A chr6 \t 121233 \t A chr1 \t 1126685 \t B chr1 \t 15834523 \t B ...
5
votes
2answers
4k views

Is there a command line utility to transpose a csv-file?

Given a file like so First, Last, Age Cory, Klein, 27 John Jacob, Smith, 30 Is there a command line utility to transpose the contents so the output appears like so First, Cory, John Jacob Last, ...
6
votes
5answers
9k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
3
votes
3answers
15k views

How to use cut command to get the first and last elements of a row?

I've asked almost the same question already, but this time, I want to retrieve the X latest elements of a row of a CSV file. For example, with an input file as this one: 1;foo;bar;baz;x;y;z ...
11
votes
4answers
5k views

Is there a command line tool to insert line breaks into a long string?

Given a long string that needs to be displayed with limited text-width, is there a command line tool in *nix that converts the single-line string to a multi-line string with each line being no longer ...
5
votes
3answers
3k views

sed: multi-line replace of config block

I have some configuration files that basically look like (...content...) # BEGIN DYNAMIC BLOCK - DO NOT EDIT MANUALLY (... more content ...) # END DYNAMIC BLOCK (... even more content ...) Now, in ...
3
votes
1answer
279 views

Printing the line after or before a regular expression match in Bash

Given a string like the following: BASH - Mplayer Playlist Shuffle http://www.youtube.com/watch?v=IB7SCPtbJM8 metalx1000 BASH - Playing Music http://www.youtube.com/watch?v=S_-EGoLYCXY metalx1000 ...
6
votes
4answers
2k views

Tool to create text files from a template

I have to regularly create 100+ text files from templates. I currently use an overly complicated shell script. I think there is a smarter way to handle this, but I don't know how. I have a "database" ...
2
votes
2answers
301 views

How to read by multiple lines instead of line-by-line

I see that in my code I can modify a line if the entire code is based on that line. Example: <p class="example">this is an example Happy Holidays</p>. However if the </p> tag is ...
3
votes
2answers
165 views

How to add a line in many files

I have many .html files and I need to add a meta tag after <head>'s start tag in each file. How I can do that? Can vim help me?
8
votes
4answers
3k views

How do I write a sed one-liner to add a character after every third character?

So, I have a string that looks like this: AUGGCCAUGGCGCCCAGAACUGAGAUCAAUAGUACCCGUAUUAACGGGUGA And I want to split the string into 3-character chunks delimited by a '+' sign. ...
9
votes
3answers
3k views

Process last line first using awk

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally. The ...
8
votes
7answers
359 views

Review changes in .tex files

I have two files paper.tex and paper_suggested_changes.tex the latter is a modified version of the former with corrections. I need to go through the two files and choose which changes to accept or ...
5
votes
1answer
507 views

How to remove all instances of a particular symbol from a text file?

I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going ...
2
votes
3answers
230 views

Indent like first line

How can I indent a file such as its first line? Example: A file containing x=1+2+3+4+ 5+6+7+8 +9+10+12 should be converted to x=1+2+3+4+ 5+6+7+8 +9+10+12 I need this inside ...
20
votes
1answer
1k views

How to implement a horizontal cat? [duplicate]

Standard cat concatenates files line by line (row by row, if you will). I find myself needing a horizontal cat command more and more often recently; i.e. a command that takes a list of files and ...
5
votes
3answers
15k views

split long line on a delimiter

What command can I use to split input like this: foo:bar:baz:quux into this? foo bar baz quux I'm trying to figure out the cut command but it seems to only work with fixed amounts of input, like ...
8
votes
4answers
3k views

How to delete commands in history matching a given string?

I need to delete all commands in my history matching a string. I've tried: $ history | grep searchstring | cut -d" " -f2 | history -d -bash: history: -d: option requires an argument $ history | grep ...
2
votes
3answers
2k views

Substitute placeholders in template

Say I have a shell configuration file config like this: HOST=localhost PORT=8080 Now I have a template template like this: The host is <%= @HOST %> The port is <%= @PORT %> How do I ...
3
votes
3answers
702 views

Script matching literal pattern over multiple lines?

I have a multi-line string in the variable $PAT. $PAT must be searched for within the file $FILE. If $PAT is in $FILE, it needs to print the file with $PAT removed. If $PAT is not found, then print ...
6
votes
3answers
937 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
9
votes
2answers
4k views

How can I use column to delimit on tabs and not spaces?

I'd like to use Unix column to format some text. I have fields delimited by tabs, but within each field there are also spaces. column delimits on white space (tabs and spaces). How can I make column ...
8
votes
10answers
2k views

What's a good way to filter a text file to remove empty lines?

I have a .csv file (on a mac) that has a bunch of empty lines, e.g.: "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
2
votes
2answers
582 views

Deleting specific lines and duplicates from a 11gb wordlist text file

I have an 11gb wordlist file which is already sorted as each word is on its own line. I need to remove duplicates and lines starting from 077. I guess I need to run sed and sort -u together but I ...
3
votes
3answers
512 views

Getting Source as Flatfile and to remove the Carriage returns and \n …special charaters?

I have an issue where I am getting a flatfile source with a huge number of records with PIPE delimiters and one of the fields is getting carriage returns (in multiple lines) and starts with a newline ...
14
votes
10answers
18k views

How to print all lines after a match up to the end of the file?

Input file1 is: dog 123 4335 cat 13123 23424 deer 2131 213132 bear 2313 21313 I give the match the pattern from in other file ( like dog 123 4335 from file2). I match the pattern of the line ...
3
votes
2answers
468 views

Extract lines containing PAT1 but not PAT2 with sed

I want to extract the lines containing PAT1 but not PAT2 from a file. For example, with PAT1='dog', PAT2='cat', and the following input: 1 cat chicken 2 bird dog apple 3 dog orange cat 4 cat juice ...
7
votes
2answers
9k views

How to find difference between two timestamps up to milliseconds?

I am new to shell scripting. The heart of my script is to find the difference between two timestamps up to milliseconds. With me I have a file with content of timestamps only as 2012-09-13 ...
2
votes
2answers
718 views

How to traverse a file and extract words using their index?

The given file must not be stored in variable and then traversed due to memory size restrictions: Example: var=$(cat FILE) for i in $var do echo $i done How do you traverse all strings in a file ...
3
votes
2answers
237 views

add fields from different files

I have two files A and B with the same number of lines. Each of them has multiple columns. I want to add field 6 in file A with field 3 in file B for each line. e.g.: file A: 2769 RUT10EKl3fY 0 ...
4
votes
1answer
2k views

intersection of two files according to the first column

I have two files in file A, there are sequence_numbers in the other file B, there are many columns, and the first column is sequnce numbers, I want to get a files with all the lines in the B with the ...
23
votes
5answers
14k views

echo a file without the first and last lines

Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
5
votes
4answers
2k views

A unix command to truncate each line of a file

I have a CSV file and I want to truncate it from the third semicolon. For example, if I have this file: 1;foo;bar;baz;x;y;z 2;foo;bar;baz;x;y;z 3;foo;bar;baz;x;y;z I want to get the following ...
3
votes
2answers
622 views

Limit stdout from a continuously running process

I haven't had much luck finding an answer to my problem, but maybe I'm not asking for it correctly. I have a process I startup like the following: nohup ping 127.0.0.1 > log.txt >2>&1 & ...
1
vote
2answers
396 views

Easiest way to find/replace in a file using a list?

I have a file, foo.txt, and a regexp I want to find in that file. Each time I find the regexp, I want to take a line from another file, bar.txt, and substitute it in for the regexp match I found in ...
0
votes
1answer
633 views

Remove a block of lines between two patterns [duplicate]

Possible Duplicate: Show only text between 2 matching pattern In a bash script using sed how can I remove a block of lines beginning with -pattern a- and ending with -pattern b- where the ...
3
votes
4answers
729 views

Delete a block of lines with a condition on the last line

I have a log-file where at the end of a series of lines you can see if this block is relevant. Now I'm looking for a command like sed to delete de blocks ending with "Content-Length: 0" and beginning ...
4
votes
3answers
1k views

Find files that have words in common

What would be the best way to create a list of files that have common words with a given file. For example, if I had: $ ls mainFile file1 file2 file file4 $ cat mainFile exquisite malicious ...
2
votes
1answer
4k views

How to add header and footer to the flat file

I need to add a header and a footer to flat file - I just need to get values from the parameter like system date and report run date in the top line and for the footer I need to get the record count ...
6
votes
2answers
5k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
8
votes
4answers
1k views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
7
votes
4answers
14k views

How to do an IF statement from the result of a executed command

I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this? if [ "`netstat -lnp | ...
1
vote
4answers
1k views

Replace text between brackets

I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like: SOMETHING [BAR, WIBBLE] SOMETHING [BAR] This gives the desired result of: SOMETHING [FOO, FOO] SOMETHING [FOO] ...
4
votes
2answers
3k views

Skip first 3 byte of a file

I am using AIX 6.1 ksh shell I want to use one liner to do something like this cat A_FILE | skip-first-3-bytes-of-the-file I want to skip the first 3 bytes of the first line, is there a way to ...
17
votes
4answers
26k views

Delete last line from the file

I use sed to quickly delete lines with specific position as sed '1d' sed '5d' But, what if I want to delete the last line of the file and I don't know the count of lines (I know I can get that ...
6
votes
2answers
1k views

shell script/utility replace 2 byte unicode in binary file

I have many large Unicode files. I would like to replace each Unicode byte pair with a new Unicode byte pair. For example, the original file contains "C3 B9 C3 AB C3 B8 C3 B0" and I would like to ...
3
votes
3answers
309 views

Print only lines that are completely numeric

I'd like to filter through a text file and only print the lines where each column is a valid floating point number. For example: 3 6 2 -4.2 21.2 3 x 4.2 21.2 3 2 2.2.2 Only the first line would ...