Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

10
votes
3answers
328 views

Split text file into short lines for reading?

Is there a program that can take a plain-text file with long lines, and add a newline after a certain number of characters, only splitting at words, to make it readable? For example, take this: Lorem ...
10
votes
6answers
3k views

How to extract a key value pair from ps command

I am trying to extract application name from ps command: ps -af -u sas |grep "java"| grep -v "grep" above command produces the following: sas 24431 1 0 Oct10 ? 00:51:08 /usr/lib/jvm/...
10
votes
5answers
2k views

Case matching pattern replacement with sed

I have a source code spread across several files. It has a pattern abcdef which I need to replace with pqrstuvxyz. The pattern could be Abcdef (Sentence Case) then it needs to be replaced with ...
10
votes
4answers
9k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
10
votes
5answers
11k views

Replace all newlines to space except the last

How can I replace all newlines with space except the last newline. I can replace all newline to space using tr but how I can do it with some exceptions?
10
votes
1answer
235 views

How to sort a file based on a section of a field

I would like to sort a file : D104HN-D104HA 8.320 4.521 1.69e+05 -- D104HN-D104HB* 8.320 2.823 2.93e+05 -- A90HB#-A90HA 1.655 4.207 7.12e+05 -- ...
10
votes
4answers
4k views

Delete whitespace for a set of lines in Vim editor

I have some text like the following in a file: sample text some random text even more random text text with no indent worst indention I need to delete the empty space before each ...
10
votes
3answers
2k views

Parsing log files for frequent IP's

So, I hacked this together while undergoing a DDOS attack to pull naughty ips out of my logs. Anyone have any improvements or other suggestions to make it better? Here's the general idea: pull ...
10
votes
2answers
3k views

How to replace all the tabs in each file in a file system branch with 2 spaces each?

I code Scala and it's convention is to use double spaces per level for identiation. But using different editors makes my sources filled with a mix of spaces and tabs. I'd like to refactor all the ...
10
votes
4answers
26k views

How to use bash script to read binary file content?

I want to read a character and then a fixed length of string (the string is not null terminated in the file, and its length is given by the preceding character). How can I do this in a bash script? ...
10
votes
4answers
748 views

How to extract data from a JSON file

I have bin searching a solution for my question but didn't find a or better said I did not get it with what I found. So lets talk about what my problem is about. I am using a Smart Home Control ...
10
votes
3answers
4k views

Process last line first using awk

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally. The ...
10
votes
8answers
846 views

sed in-place line deletion on full filesystem?

Due to an application bug as yet undiagnosed, I have several hundred servers with a full disk. There is one file that has been filled up with duplicate lines—not a log file, but a user environment ...
10
votes
1answer
4k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
10
votes
4answers
310 views

Script to extract selected entries from a bibtex file

I have a large bibtex file with many entries where each entry has the general structure @ARTICLE{AuthorYear, item = {...}, item = {...}, item = {...}, etc } (in some cases ARTICLE might be a ...
10
votes
3answers
3k views

Is it possible to nest a 'find -exec' within another 'find -exec'?

Something like the following is what I what I'm after, but my code doesn't work, no matter how I escape {} and + ; find ./ -maxdepth 1 -type d -name '.*' -exec \ find {} -maxdepth 1 -type f -...
9
votes
6answers
1k views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to another file, without sed and awk? For example: $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i 's/aaa/...
9
votes
7answers
635 views

Swapping an unlimited number of columns

I have a file with columns. See below for an example: a b c ... z 1 2 3 ... 26 I'd like to swap all columns where the 1st becomes the last, the second becomes the one before last...etc.. z y x ...
9
votes
6answers
13k views

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo name=...
9
votes
5answers
2k views

Merge two files line by line with the delimiter triple pipe symbol “|||”

I have two parallel files with the same number of lines in two languages and plan to merge these two files line by line with the delimiter |||. E.g., the two files are as follows: File A: 1Mo 1,1 I ...
9
votes
6answers
537 views

How can I double the newlines in an output stream

When I have an output stream: a b c d e How can I double the newlines: a b c d e
9
votes
7answers
17k views

Sed — Replace first k instances of a word in the file

I want to replace only the first k instances of a word. How can I do this? Eg. Say file foo.txt contains 100 instances occurrences of word 'linux' . I need to replace first 50 occurrences only.
9
votes
2answers
2k views

Tool in unix to subtract text files?

I have a large file composed of text fields separated by semicolons in the form of a large table. It has been sorted. I have a smaller file composed of the same text fields. At some point, someone ...
9
votes
2answers
4k views

How to remove multiple lines per occurrence in a file?

Say I have this 857835 line file, containing stuff like this: a1 rubbish1 rubbish2 rubbish3 rubbish4 a1 rubbish5 rubbish6 rubbish7 rubbish8 And I wish to remove all occurences of a1 and the next ...
9
votes
5answers
307 views

Shell Script: grab string in the middle of the text, sometimes in the beginning

I have a large text file where a part of it looks like this (edited values): JULIANA XXXX006060 LI1033322 THC BRL 730.00 XXXX006296 AA1004737 THC BRL 1,740.00 SANTOS JULIANA XXXX006668 AA1004786 THC ...
9
votes
4answers
4k views

How to prepend a license header recursively for all .h and .cpp files in a directory

I'm trying to add a license header to all header files and source files in a project directory using a for loop. This is not working, is there any other approach using sed?
9
votes
4answers
4k views

combine two text files with adding some separator between?

cat file1 file2 will combine two text files. But if I want to add some separator between, like a line or two of ********************************, do I have to open the first file, and add the line at ...
9
votes
5answers
357 views

How to count rows ordered by the first field in bash

Here is a snippet from the INPUT: ... #################### Bala Bela;XXXXXX12345;XXXXXX12345678;A SERVER345Z3.DOMAIN.com0 SERVER346Z3.DOMAIN.com0 SERVER347Z3.DOMAIN.com0 SERVER348Z3.DOMAIN.com0 ssh-...
9
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
9
votes
3answers
477 views

Doing two things with output from a command

I have a program texcount that outputs the number of words in my LaTeX document. I can also pipe the output of this to sed to make the newlines TeX linebreaks and write this to a file which I can then ...
9
votes
6answers
3k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
9
votes
5answers
10k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
9
votes
3answers
7k views

BASH base conversion from decimal to hex

In Bash, how does one do base conversion from decimal to another base, especially hex. It seems easy to go the other way: $ echo $((16#55)) 85 With a web-search, I found a script that does the ...
9
votes
5answers
49k views

Parse XML to get node value in bash script?

I would like to know how I can get the value of a node with the following paths: config/global/resources/default_setup/connection/host config/global/resources/default_setup/connection/username ...
9
votes
7answers
1k views

How to merge text of alphabetic lines with the numeric lines in shell?

I Have a file that has text like this: AAAA BBBB CCCC DDDD 1234 5678 9012 3456 EEEE 7890 etc... And i want to match up the Alphabetic lines with the Numeric lines so they are like this: ...
9
votes
3answers
5k views

How do I make python programs behave like proper unix tools?

I have a few Python scripts laying around, and I'm working on rewriting them. I have the same problem with all of them. It's not obvious to me how to write the programs so that they behave like ...
9
votes
6answers
17k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
9
votes
7answers
3k views

How to sort (by whatever key) BibTex entries in `.bib` file?

I work with LaTeX and do versioning with Git. For bibliography management I use Mendeley. The problem is that each time Mendeley synchronizes it's .bib exports, they are in different order, what ...
9
votes
4answers
8k views

Concatenate lines by first column by awk or sed

How can I use awk in the following situation? I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh). The file may be ...
9
votes
4answers
2k views

Any (text based) plotting tools in unix toolbox?

I was wondering if there are any generic plotting tools available in the unix toolbox? By plotting I mean simple bars or things you see in many curses based utilities. And by unix toolbox I mean ...
9
votes
2answers
9k views

tr complains of “Illegal byte sequence”

I'm brand new to UNIX and I am using Kirk McElhearn's "The Mac OS X Command Line" to teach myself some commands. I am attempting to use tr and grep so that I can search for text strings in a regular ...
9
votes
2answers
2k views

Deleting all C comments with sed

I am trying to write a script that will delete all comments and everything in between inside C files in my current directory. I've been using sed, and this is what I have so far: sed -i '/ * [^()] */...
9
votes
3answers
11k views

How do I grep without leading whitespaces?

I'm greping through a large codebase, and leading whitespaces and tabulation seem to be quite annoying. Is there any way to get rid of it? grep -R "something" ./ E.g, instead of: foo/bar.cpp: ...
9
votes
3answers
2k views

How do I take a list and remove it from a file?

I have a long list of domain names that I need to remove from /etc/remotedomains. They're probably not in any particular order in the file. Each domain is on one line. How could I iterate through the ...
9
votes
5answers
5k views

`highlight` command

Is there a command which can be used to easily highlight (with colors, bold text or inverted back-/foreground) specific strings in stdin while passing the entire string to stdout? I'm thinking of ...
9
votes
2answers
2k views

How to display TSV (csv) in console, when empty cells are missed by: `column -t -s $'\t' `

I have file with columns spearated with tab. I have file when some rows have empty cells (on begining, in middle). In such cases column -t -s $'\t' simply fails: Input: $ echo -e 'A\tB\tC\tD\n\tb1\...
9
votes
2answers
6k views

Why does xargs strip quotes from input?

Why does xargs strip quotes from input text? Here is a simplified example: echo "/Place/='http://www.google.com'" | xargs echo outputs /Place/=http://www.google.com Is there any way to work-...
9
votes
4answers
986 views

Split the file and put it in corresponding data structure?

I have a file which contains the single line below: {machineA=[0, 1024, 4, 1028], machineB=[1, 1025, 5, 1029]} in which I have two datasets: machineA=[0, 1024, 4, 1028] machineB=[1, 1025, 5, 1029] ...
8
votes
8answers
3k views

Using 'grep' to find lines that contain all of three specified characters in any order [duplicate]

We have a document which contains lines and we have to find if [a|b|c] appears at least once in every line no matter which order. For example: Input: abc bca cab hhfdhdhfabjfdjdjff acjfdjdfjdf ...
8
votes
6answers
4k views

How can I get a specific line from a file? [duplicate]

I want to extract an exact line from a very big file. For example, line 8000 would be gotten like this: command -line 8000 > output_line_8000.txt