Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

8
votes
8answers
983 views

Grep for pattern at start or middle of a line

I'll start by saying that I think this problem is a little less innocent than it sounds. What I need to do: check for a folder within the PATH environment variable. It could be at the start or ...
8
votes
4answers
732 views

Splitting file for every 10000 numbers ( not lines )

I have a file that looks like the following: chr19 61336212 + 0 0 CG CGT chr19 61336213 - 0 0 CG CGG chr19 61336218 + ...
8
votes
3answers
36k views

remove particular characters from a variable using bash

I want to parse a variable (in my case it's development kit version) to make it dot(.) free. If version='2.3.3', desired output is 233. I tried as below, but it requires . to be replaced with another ...
8
votes
4answers
3k views

How do I write a sed one-liner to add a character after every third character?

So, I have a string that looks like this: AUGGCCAUGGCGCCCAGAACUGAGAUCAAUAGUACCCGUAUUAACGGGUGA And I want to split the string into 3-character chunks delimited by a '+' sign. ...
8
votes
7answers
359 views

Review changes in .tex files

I have two files paper.tex and paper_suggested_changes.tex the latter is a modified version of the former with corrections. I need to go through the two files and choose which changes to accept or ...
8
votes
3answers
10k views

How do I insert a space every four characters in a long line?

I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...
8
votes
2answers
549 views

Merge three file in to a single master file by excluding the header in 2nd and 3rd files

I have three file with n number of lines as given below sample1.txt ------------------------------ Date Time Name ------------------------------ 2013/10/12 12:09:09 Krish ...
8
votes
6answers
759 views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
8
votes
6answers
2k views

Delete lines beginning with #

How do I delete lines beginning with a #, given that there can be whitespace on the left and right of the #? # Master socket provides access to userdb information. It's typically
8
votes
4answers
2k views

combine two text files with adding some separator between?

cat file1 file2 will combine two text files. But if I want to add some separator between, like a line or two of ********************************, do I have to open the first file, and add the line at ...
8
votes
3answers
2k views

Remove only the commas present within the double quotes

In a text file, I want to remove , (commas) and also the " (quotes) (only if the double quotes contains numbers seperated by commas). 56,72,"12,34,54",x,y,"foo,a,b,bar" Expected ouput ...
8
votes
2answers
17k views

How to cut (select) a field from text line counting from the end?

I know how to select a field from a line using the cut command. For instance, given the following data: a,b,c,d,e f,g,h,i,j k,l,m,n,o This command: cut -d, -f2 # returns the second field of the ...
8
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
8
votes
5answers
709 views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
8
votes
4answers
3k views

How to delete commands in history matching a given string?

I need to delete all commands in my history matching a string. I've tried: $ history | grep searchstring | cut -d" " -f2 | history -d -bash: history: -d: option requires an argument $ history | grep ...
8
votes
3answers
9k views

How to select specific processes with 'top'?

In case of one program having multiple instances, running pidof program gives: `1 2 3` top -p accepts comma-delimited arguments: 1, 2, 3. This means that top -p `pidof program` won't work: ...
8
votes
5answers
7k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
8
votes
4answers
7k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
8
votes
2answers
407 views

How can I delete all text between nested curly brackets in a multiline text file?

This question comes from How can I delete all text between curly brackets in a multiline text file? (just the same, but without the requirements for nesting). Example: This is { {the multiline ...
8
votes
7answers
2k views

How to sort (by whatever key) BibTex entries in `.bib` file?

I work with LaTeX and do versioning with Git. For bibliography management I use Mendeley. The problem is that each time Mendeley synchronizes it's .bib exports, they are in different order, what ...
8
votes
3answers
4k views

How do I make python programs behave like proper unix tools?

I have a few Python scripts laying around, and I'm working on rewriting them. I have the same problem with all of them. It's not obvious to me how to write the programs so that they behave like ...
8
votes
1answer
199 views

Filter the clipboard content without using an intermediate file

I'm trying to modify the text that is in my clipboard, remove certain lines of that contains some strings and then get the output for further manipulation (I prefer coping to my clipboard only if I ...
8
votes
5answers
619 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
8
votes
3answers
2k views

How do I take a list and remove it from a file?

I have a long list of domain names that I need to remove from /etc/remotedomains. They're probably not in any particular order in the file. Each domain is on one line. How could I iterate through the ...
8
votes
1answer
826 views

How can I write a new syntax highlighter for gedit?

Some times I wish the particular language I happen to be writing in had syntax highlighting. How can I add that?
8
votes
5answers
3k views

`highlight` command

Is there a command which can be used to easily highlight (with colors, bold text or inverted back-/foreground) specific strings in stdin while passing the entire string to stdout? I'm thinking of ...
8
votes
2answers
2k views

How could I use Bash to find 2 bytes in a binary file, increase their values, and replace?

I'm trying to find two bytes inside binary file, then increase value of those two bytes and replace them inside file. Those two bytes are on positions 0x82-0x83. For now on I have successfully ...
8
votes
4answers
1k views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
8
votes
2answers
1k views

Parsing XML, JSON, and newer data file formats in UNIX using command line utilities

The Unix environment has some excellent tools for parsing text in various forms. However, of late, the data is not in the traditional (historical) formats (CSV, TSV, record-based or some other ...
8
votes
3answers
2k views

Is it possible to nest a 'find -exec' within another 'find -exec'?

Something like the following is what I what I'm after, but my code doesn't work, no matter how I escape {} and + ; find ./ -maxdepth 1 -type d -name '.*' -exec \ find {} -maxdepth 1 -type f ...
8
votes
1answer
475 views

Is there a way to make perl -i not clobber symlinks?

A friend of mine points out that if you do: perl -pi.bak -e 's/foo/bar/' somefile when "somefile" is actually a symlink, perl does just what the docs say it will do: It does this by renaming ...
7
votes
7answers
839 views

Slick one-liner to convert a list like “1: 2, 3, 4, 5” to “1.2, 1.3, 1.4, 1.5”

Let's say I have a file that looks something like this: 23: a, b, c, d 24: b, d, f 25: c, g and I want to get output like this: 23.a 23.b 23.c 23.d 24.b 24.d 24.f 25.c 25.g Of course it's not ...
7
votes
5answers
749 views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to another file, without sed and awk? For example: $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
7
votes
8answers
3k views

Grep rest of line…after match

I have a file containing only two lines, with the following structure: $ cat /tmp/pwpower.log 000D6F0000D34227, -114.10 000D6F0001A405C4, -130.09 The values are power values of my solar plant. ...
7
votes
7answers
14k views

How to test whether the file is crlf or lf without modyfing it?

I need to periodically run a command, that ensures that some text files are kept in linux mode. Unfortunately dos2unix always modifies file, which would mess file's and folder's timestamp and cause ...
7
votes
2answers
2k views

How can I cut a large file in place? [duplicate]

I have a very big text file, about 80GB, and I need to cut a part form it that lies between two given lines. The part I need is not big, and I have not enough space left on the hard drive to do things ...
7
votes
5answers
12k views

Transposing rows and columns

I have a file with the lines like below. title1:A1 title2:A2 title3:A3 title4:A4 title5:A5 title1:B1 title2:B2 title3:B3 title4:B4 title5:B5 title1:C1 title2:C2 title3:C3 title4:C4 title5:C5 ...
7
votes
5answers
34k views

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
7
votes
4answers
14k views

How to do an IF statement from the result of a executed command

I am trying to do an IF statement from the output of an executed commmand. Here is how I am trying to do it, but it doesn't work. Does anyone know the right way to do this? if [ "`netstat -lnp | ...
7
votes
5answers
497 views

List out strings which are substrings of other strings in the list

I have a list of names like so: dog_bone dog_collar dragon cool_dragon lion lion_trainer dog I need to extract out names that appear in other names like so: dragon lion dog I looked through the ...
7
votes
4answers
542 views

How to grep-inverse-match and exclude “before” and “after” lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except ...
7
votes
6answers
697 views

Obtain values of a section of a file

I have a configuration file in the following format. <Title> [part1] A.File = a A.Val = val1 B.File = a B.Val = val1 [part2] A.File = a1 A.Val = val2 B.File = a B.Val = val1 ...
7
votes
5answers
1k views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
7
votes
2answers
4k views

A shell script for joining two files

I want to write a shell script that get two files A and B, and get a result like this: File A: user_a tel_a addr_a user_b tel_b addr_b File B: process_1 user_a process_2 user_a process_3 user_b ...
7
votes
7answers
402 views

Removing numeric values in certain columns whilst keeping minus signs?

I put my hands up to being a complete beginner, so apologies if I'm doing this wrong. Say I have the following data frame that continues indefinitely horizontally and vertically with negative numbers ...
7
votes
4answers
339 views

Need to move the last line of the file to second line of the same file

I have a file 'test' in linux with below data. aaaaaa bbbbbb cccccc dddddd eeeeee I need to cut the last line of this file and put it in the second position. It must look like below. aaaaaa eeeeee ...
7
votes
4answers
199 views

How to grep -v and also exclude the next line after the match?

How to filter out 2 lines for each line matching the grep regex? this is my minimal test: SomeTestAAAA EndTest SomeTestABCD EndTest SomeTestDEFG EndTest SomeTestAABC EndTest SomeTestACDF EndTest ...
7
votes
4answers
423 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
7
votes
2answers
2k views

In `while IFS= read..`, why does IFS have no effect?

I might have something absolutely wrong, but it looks convincing to me, that setting IFS as one of the commands in the pre-do/done list has absolutely no effect. The outer IFS (outside the while ...
7
votes
2answers
1k views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...