Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

9
votes
7answers
588 views

Swapping an unlimited number of columns

I have a file with columns. See below for an example: a b c ... z 1 2 3 ... 26 I'd like to swap all columns where the 1st becomes the last, the second becomes the one before last...etc.. z y x ...
9
votes
10answers
3k views

What's a good way to filter a text file to remove empty lines?

I have a .csv file (on a mac) that has a bunch of empty lines, e.g.: "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
9
votes
5answers
912 views

Merge two files line by line with the delimiter triple pipe symbol “|||”

I have two parallel files with the same number of lines in two languages and plan to merge these two files line by line with the delimiter |||. E.g., the two files are as follows: File A: 1Mo 1,1 I ...
9
votes
6answers
526 views

How can I double the newlines in an output stream

When I have an output stream: a b c d e How can I double the newlines: a b c d e
9
votes
2answers
2k views

Tool in unix to subtract text files?

I have a large file composed of text fields separated by semicolons in the form of a large table. It has been sorted. I have a smaller file composed of the same text fields. At some point, someone ...
9
votes
2answers
4k views

How to remove multiple lines per occurrence in a file?

Say I have this 857835 line file, containing stuff like this: a1 rubbish1 rubbish2 rubbish3 rubbish4 a1 rubbish5 rubbish6 rubbish7 rubbish8 And I wish to remove all occurences of a1 and the next ...
9
votes
5answers
290 views

Shell Script: grab string in the middle of the text, sometimes in the beginning

I have a large text file where a part of it looks like this (edited values): JULIANA XXXX006060 LI1033322 THC BRL 730.00 XXXX006296 AA1004737 THC BRL 1,740.00 SANTOS JULIANA XXXX006668 AA1004786 THC ...
9
votes
4answers
3k views

combine two text files with adding some separator between?

cat file1 file2 will combine two text files. But if I want to add some separator between, like a line or two of ********************************, do I have to open the first file, and add the line at ...
9
votes
5answers
352 views

How to count rows ordered by the first field in bash

Here is a snippet from the INPUT: ... #################### Bala Bela;XXXXXX12345;XXXXXX12345678;A SERVER345Z3.DOMAIN.com0 SERVER346Z3.DOMAIN.com0 SERVER347Z3.DOMAIN.com0 SERVER348Z3.DOMAIN.com0 ...
9
votes
4answers
4k views

How to prepend a license header recursively for all .h and .cpp files in a directory

I'm trying to add a license header to all header files and source files in a project directory using a for loop. This is not working, is there any other approach using sed?
9
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
9
votes
3answers
471 views

Doing two things with output from a command

I have a program texcount that outputs the number of words in my LaTeX document. I can also pipe the output of this to sed to make the newlines TeX linebreaks and write this to a file which I can then ...
9
votes
4answers
4k views

How to delete commands in history matching a given string?

I need to delete all commands in my history matching a string. I've tried: $ history | grep searchstring | cut -d" " -f2 | history -d -bash: history: -d: option requires an argument $ history | grep ...
9
votes
4answers
7k views

Efficient in-place header removing for large files using sed?

The commands below may takes minutes depends on the file size. Is there any more effient method? sed -i 1d large_file
9
votes
6answers
3k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
9
votes
3answers
4k views

BASH base conversion from decimal to hex

In Bash, how does one do base conversion from decimal to another base, especially hex. It seems easy to go the other way: $ echo $((16#55)) 85 With a web-search, I found a script that does the ...
9
votes
5answers
9k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
9
votes
7answers
6k views

Combining 2 different cut outputs in a single command?

I have a file such as the following: 1234 ABCD EFGH I'd like to convert it to the following: 2341 BCDA FGHE The actual file has 4,000 words, so I would like to do this in an efficient manner. I ...
9
votes
5answers
375 views

Find last occurence of string in multiple files

I need to search multiple log files (all files generated in last 24 hours, all kept in same directory) to find the last occurrence of a string. This is the command I wrote: find . -mtime 1 | grep ...
9
votes
7answers
1k views

How to merge text of alphabetic lines with the numeric lines in shell?

I Have a file that has text like this: AAAA BBBB CCCC DDDD 1234 5678 9012 3456 EEEE 7890 etc... And i want to match up the Alphabetic lines with the Numeric lines so they are like this: ...
9
votes
2answers
3k views

How to replace all the tabs in each file in a file system branch with 2 spaces each?

I code Scala and it's convention is to use double spaces per level for identiation. But using different editors makes my sources filled with a mix of spaces and tabs. I'd like to refactor all the ...
9
votes
3answers
4k views

How do I make python programs behave like proper unix tools?

I have a few Python scripts laying around, and I'm working on rewriting them. I have the same problem with all of them. It's not obvious to me how to write the programs so that they behave like ...
9
votes
4answers
7k views

Concatenate lines by first column by awk or sed

How can I use awk in the following situation? I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh). The file may be ...
9
votes
4answers
2k views

Any (text based) plotting tools in unix toolbox?

I was wondering if there are any generic plotting tools available in the unix toolbox? By plotting I mean simple bars or things you see in many curses based utilities. And by unix toolbox I mean ...
9
votes
2answers
2k views

Deleting all C comments with sed

I am trying to write a script that will delete all comments and everything in between inside C files in my current directory. I've been using sed, and this is what I have so far: sed -i '/ * [^()] ...
9
votes
1answer
4k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
9
votes
3answers
10k views

How do I grep without leading whitespaces?

I'm greping through a large codebase, and leading whitespaces and tabulation seem to be quite annoying. Is there any way to get rid of it? grep -R "something" ./ E.g, instead of: foo/bar.cpp: ...
9
votes
2answers
2k views

How to display TSV (csv) in console, when empty cells are missed by: `column -t -s $'\t' `

I have file with columns spearated with tab. I have file when some rows have empty cells (on begining, in middle). In such cases column -t -s $'\t' simply fails: Input: $ echo -e ...
9
votes
4answers
259 views

Script to extract selected entries from a bibtex file

I have a large bibtex file with many entries where each entry has the general structure @ARTICLE{AuthorYear, item = {...}, item = {...}, item = {...}, etc } (in some cases ARTICLE might be a ...
9
votes
4answers
827 views

Split the file and put it in corresponding data structure?

I have a file which contains the single line below: {machineA=[0, 1024, 4, 1028], machineB=[1, 1025, 5, 1029]} in which I have two datasets: machineA=[0, 1024, 4, 1028] machineB=[1, 1025, 5, 1029] ...
8
votes
8answers
2k views

Using 'grep' to find lines that contain all of three specified characters in any order [duplicate]

We have a document which contains lines and we have to find if [a|b|c] appears at least once in every line no matter which order. For example: Input: abc bca cab hhfdhdhfabjfdjdjff acjfdjdfjdf ...
8
votes
5answers
999 views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to another file, without sed and awk? For example: $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
8
votes
6answers
3k views

How can I get a specific line from a file? [duplicate]

I want to extract an exact line from a very big file. For example, line 8000 would be gotten like this: command -line 8000 > output_line_8000.txt
8
votes
8answers
1k views

Grep for pattern at start or middle of a line

I'll start by saying that I think this problem is a little less innocent than it sounds. What I need to do: check for a folder within the PATH environment variable. It could be at the start or ...
8
votes
8answers
720 views

Split text file into lines with fixed number of words

Related, but no satisfactory answers: How can I split a large text file into chunks of 500 words or so? I'm trying to take a text file (http://mattmahoney.net/dc/text8.zip) with > 10^7 words all in ...
8
votes
4answers
799 views

Splitting file for every 10000 numbers ( not lines )

I have a file that looks like the following: chr19 61336212 + 0 0 CG CGT chr19 61336213 - 0 0 CG CGG chr19 61336218 + ...
8
votes
6answers
11k views

Sed — Replace first k instances of a word in the file

I want to replace only the first k instances of a word. How can I do this? Eg. Say file foo.txt contains 100 instances occurrences of word 'linux' . I need to replace first 50 occurrences only.
8
votes
4answers
4k views

How do I write a sed one-liner to add a character after every third character?

So, I have a string that looks like this: AUGGCCAUGGCGCCCAGAACUGAGAUCAAUAGUACCCGUAUUAACGGGUGA And I want to split the string into 3-character chunks delimited by a '+' sign. ...
8
votes
7answers
372 views

Review changes in .tex files

I have two files paper.tex and paper_suggested_changes.tex the latter is a modified version of the former with corrections. I need to go through the two files and choose which changes to accept or ...
8
votes
2answers
606 views

Merge three file in to a single master file by excluding the header in 2nd and 3rd files

I have three file with n number of lines as given below sample1.txt ------------------------------ Date Time Name ------------------------------ 2013/10/12 12:09:09 Krish ...
8
votes
6answers
1k views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
8
votes
6answers
3k views

Delete lines beginning with #

How do I delete lines beginning with a #, given that there can be whitespace on the left and right of the #? # Master socket provides access to userdb information. It's typically
8
votes
3answers
425 views

Extract text between three single quotes

I have the following in a file description: ''' This rule forbids throwing string literals or interpolations. While JavaScript (and CoffeeScript by extension) allow any expression to ...
8
votes
6answers
12k views

How to remove blank lines from a file in shell?

I want to remove all empty lines from a file. Even if the line contains spaces or tabs it should also be removed.
8
votes
7answers
17k views

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...
8
votes
3answers
3k views

Remove only the commas present within the double quotes

In a text file, I want to remove , (commas) and also the " (quotes) (only if the double quotes contains numbers seperated by commas). 56,72,"12,34,54",x,y,"foo,a,b,bar" Expected ouput ...
8
votes
5answers
974 views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
8
votes
4answers
8k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
8
votes
3answers
12k views

How to select specific processes with 'top'?

In case of one program having multiple instances, running pidof program gives: `1 2 3` top -p accepts comma-delimited arguments: 1, 2, 3. This means that top -p `pidof program` won't work: ...
8
votes
4answers
462 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...