Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

2
votes
2answers
2k views

How to divide a list of values by a number in command line?

I am trying to translate a simple program to the command line using unix utilities. For example, if I have a frequency list (after piping through uniq and sort) 5 x 4 y 1 z I want to print out, ...
3
votes
1answer
327 views

Character classes: construct my own

I want to construct my own character class in a script, then modify (and use) it, for example: [:myclass:] contains a, *, \n (as linefeed) and [WHITESPACE]. I want to add all characters to ...
4
votes
3answers
541 views

Why do I keep destroying my text files?

I have destroyed a bunch of non-essential files and I don't know why. I have been executing commands like: tr -sc 'A-Za-z' '\n' > somefile.txt | less there is no output (blank page with flashing ...
2
votes
3answers
674 views

dynamic display of a running process?

I am writing a script and acording to that when i run it certain info will be displayed on the output screen. for example say the constant data displayed is: my name is mukesh. i am 27 years old ...
6
votes
1answer
165 views

counting the number of occurences in a file of data list

I have a data file, like 7 2 10 9 10 3 2 4 2 4 6 Each line has a single value. I want to count the occurence of each value. For instance, 10 occurs two times in this file. Is there a simple way ...
6
votes
4answers
308 views

How can I print a conditional header BEFORE stdout, if there is any output on stdout

I have a process that filters a list of files from a directory (having find check to see if there are files older than a certain period to show a queue is stuck). It may or may not return anything, ...
12
votes
4answers
54k views

Finding text between two specific characters or strings

Say I have lines like this: *[234]* *[23]* *[1453]* where * represents any string (except a string of the form [number]). How can I parse these lines with a command line utility and extract the ...
7
votes
7answers
6k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" ...
1
vote
4answers
3k views

Count number of lines in files then compare which has more (BASH)

I need to count the number of lines in x files and compare which has more. The one I've done only takes two files and compares them. Any idea how to make it x amount of files? echo Enter the ...
7
votes
2answers
4k views

A shell script for joining two files

I want to write a shell script that get two files A and B, and get a result like this: File A: user_a tel_a addr_a user_b tel_b addr_b File B: process_1 user_a process_2 user_a process_3 user_b ...
2
votes
3answers
1k views

Sorting columns from two files with awk

I have two input files 28a39a289906c01159f999a68996091a tom@dong.com 274d1d2c7e931fb55ac0c91dd41f2be7 go@far.com 44d25d3b1b70b240d5058f1be1cef576 person@domain.com 2227a768f6d253b7bf81bb4ecb15b52d ...
1
vote
1answer
265 views

Tools to dump entries from arbitrary sets of log files between 2 dates

I had a outage incident a few days ago and the support team has asked for any available logs for the period, while thinking about how to do this, it occurred to me that it would be generally handy to ...
3
votes
3answers
232 views

How to perform an action only on the first line?

sed 's/.\(.*\)/\1/' myfile Say myfile contains: abc def ghi I want to remove the first character only from the first line but the above removes it from all the line.
5
votes
2answers
8k views

Grep: search and replace full line

The command grep "foo" myfile.txt prints all matching lines in my file. Now I want to replace the full line with another string. How can I do that?
4
votes
2answers
11k views

What is the python equivalent of grep -v?

I like grep -v. I use it all the time. But I am also doing some text processing in python, and there is one crucial thing that I lack. Usually, I use grep -v to take extraneous stuff out of text. ...
4
votes
2answers
1k views

How to output file & ignoring lines that start with “?”?

I do svn status --show-updates and then I want to either Q1: ignore (not to display) lines that start with ? Q2: display only lines that start with * Note that there are few spaces before * occurs. ...
69
votes
10answers
72k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid off those messages?
2
votes
4answers
270 views

separate a file into several small files according to columns

I have a data file, which can have N rows, and each row is composed M elements separated by white space. Currently, I want to separate each row into several segments. In other words, assume the ...
4
votes
6answers
6k views

How to swap columns in such a file?

I have a text file, each line is stored like this : "Video or movie" "parent" "Media or entertainment" "1" "1" "1" "0" "0" I want to swap the columns 3 with 2, i.e. "Video or movie" ...
1
vote
3answers
580 views

grep equivalent of the kwrite regex [A-Z][A-Z]+

So, it took me ages, but I finally learned to think in terms of regular expressions, thanks to using them in kwrite. But I still don't know how to translate that knowledge to grep. I love my grep, ...
3
votes
2answers
412 views

need shell script to grock a csv

Need some pointing in right direction on script to fetch and regex or sed. Site24x7 provides a URL with a CSV list of their source IP's used for monitoring. (they also provide other formats, CSV ...
7
votes
7answers
2k views

How to sort (by whatever key) BibTex entries in `.bib` file?

I work with LaTeX and do versioning with Git. For bibliography management I use Mendeley. The problem is that each time Mendeley synchronizes it's .bib exports, they are in different order, what ...
4
votes
2answers
402 views

How to expand tabs based on content?

I've got some tab-delimited data coming out of a Unix pipe. I'd like to format this data into a compact human-readable table. How can I expand these tabs into spaces, and automatically set the tab ...
4
votes
3answers
572 views

Is it possible to use split to make character chunks out of Chinese unicode bytes?

For a while, I've been dealing with Chinese unicode text. Of course, the usual rules apply. I can grep for characters the same way I'd do so for words. This is very useful to me. But there's one ...
2
votes
1answer
79 views

!fmt for current line only

When I want to format part of text with fmt in vim I usually select parts of interesting lines in Visual Mode, with: Ctrl+V Arrows, and than type !+f+m+t. When there are couple of lines they are ...
6
votes
4answers
568 views

extracting certain info from output

I'm trying to extract certain info from ffmpeg output. Sample ffmpeg output: configuration: --enable-memalign-hack --enable-mp3lame --enable-gpl --disable-vhook --disable-ffplay --disable-ffserver ...
50
votes
6answers
32k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
4
votes
2answers
335 views

Extract part of the first line of a file

I am trying to configure some external commands in Gedit3, for compiling LaTeX files. All works well, except for the following. I have a LaTeX document, consisting of many parts. Each file begins with ...
2
votes
4answers
1k views

Linux equivalent of Cisco's “begin” command

In the Cisco IOS, you can filter and manipulate the output of commands using various built-in tools. Some of these have clear Linux equivalents. For example, 'exclude' and 'include' provide 'grep ...
22
votes
10answers
8k views

How can I write to the second line of a file from the command line?

I have an external program that produces an output file (largish, 20K lines possible). I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
9
votes
5answers
5k views

Randomly draw a certain number of lines from a data file

I have a data list, like 12345 23456 67891 -20000 200 600 20 ... Assume the size of this data set (i.e. lines of file) is N. I want to randomly draw m lines from this data file. Therefore, the ...
21
votes
4answers
6k views

sort inplace - like sed --in-place - exists?

Am I blind or is there no option like --in-place for sort? In order to save results to the input file, sed uses -i (--in-place). Redirecting the output of sort to the input file sort < f > f ...
9
votes
4answers
2k views

Delete whitespace for a set of lines in Vim editor

I have some text like the following in a file: sample text some random text even more random text text with no indent worst indention I need to delete the empty space before each ...
6
votes
1answer
796 views

How to detect and delete lines containing ˆ@

I have a simple problem: In my file, the are lines containing the string ˆ@ˆ@ˆ@ˆ@ˆ@ˆ@. I just want to delete all lines with this string, using for example the sed or grep commands. And I would like ...
21
votes
4answers
28k views

How can I find out how many lines a text file contains without viewing it?

How can I find how many lines a text file contains without opening the file in an editor or a viewer application? Is there a handy Unix console command to see the number?
3
votes
1answer
128 views

Wrapping long cells in a tsv to keep them in same column

Formatting a TSV with long cells with column makes its readability very low. It simply breaks lines like this: $ python3 -c 'for i in (1,2,3): print(((str(i)*50)+"\t")*3)' | column -s $'\t' -t ...
4
votes
3answers
3k views

How to search and replace text in all php-files in a directory and it's subdirectories

I am looking for a shell script that recursively traverses all .php files in a directory and performs a search & replace of a particular text pattern. The search pattern is quite long ( > 5000 ...
3
votes
2answers
444 views

Extracting bigrams with sed

How do I split an entire text consisting of only letters into bigrams. For example: odd even od -> de dd -> ev This is what I have so far but it does not produce the desired result. [some ...
5
votes
2answers
1k views

How to display TSV (csv) in console, when empty cells are missed by: `column -t -s $'\t' `

I have file with columns spearated with tab. I have file when some rows have empty cells (on begining, in middle). In such cases column -t -s $'\t' simply fails: Input: $ echo -e ...
4
votes
2answers
67 views

changing the order of items in a list

I have a file with plain text. This file contains a list like this: @"Joe", @"211", @"Bob Nelson", @"132", @"Jack Sierra", @"3422", @"Walt Robert", @"14", The list goes on and on with thousands of ...
4
votes
2answers
5k views

list the difference and overlap between two plain data set [duplicate]

Possible Duplicate: Linux tools to treat files as sets and perform set operations on them I have two data sets, A and B. The format for each data set is one number per line. For instance, ...
5
votes
3answers
13k views

Count distinct values of a field in a file

I have a file contains around million number of lines. In the lines I have a field called transactionid, which has repeatitive values. What I need to do is to count them distinctly. No matter of how ...
4
votes
5answers
7k views

How can I convert tab delimited data to comma delimited data?

I'm requesting a list of ec2 snapshots via amazon's ec2 command line tool: ec2-describe-snapshots -H --hide-tags > snapshots.csv The data looks something like this: SnapshotId VolumeId ...
3
votes
2answers
3k views

use awk to replace parentheses?

How can I use awk to replace the parentheses in these 2 scenarios? Change empty parens to the word "None" 192.168.0.24,Up,() Remove parens to only host.domain.com is left ...
4
votes
6answers
1k views

How to align a column in a text file

Using bash on Linux, how can I process a text file such that space is inserted before a certain column so that column is aligned in the output? For example: Input 1653455 ASDFASDF22 bla bla bla asd ...
9
votes
4answers
2k views

Any (text based) plotting tools in unix toolbox?

I was wondering if there are any generic plotting tools available in the unix toolbox? By plotting I mean simple bars or things you see in many curses based utilities. And by unix toolbox I mean ...
3
votes
5answers
344 views

evaluate multiple patterns from program output and write into pattern specific files

I have a script outputting some value/numbers and I want to split those into two files. I am looking at something like: ./runme.sh | grep 'ook' >> ook.out | grep 'eek' >> eek.out Where ...
2
votes
1answer
571 views

Reading two files into an IFS while loop — Is there a way to get a zero diff result in this case?

I have a text file full of several hundred lines of sequences like this: b 29. b 52. c 84. c 83. c 94. c 93. c 61. b 38. c 81. c 92. c 28. c 37. ...
5
votes
4answers
11k views

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...
2
votes
4answers
798 views

Compare two files

I have two files in UNIX. 1st file is Entity and Second File is References. 1st File has only one column named Entity ID and 2nd file has two columns Entity ID | Person ID. I want to produce a output ...