Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

24
votes
6answers
73k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
19
votes
7answers
42k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...
27
votes
12answers
23k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
16
votes
14answers
5k views

How to read an IP address backwards?

If we have this string (IP address): 192.168.1.1 How can I derive the (DNS reverse record form) from this string, so it will be shown like 1.1.168.192.in-addr.arpa using a shell script?
17
votes
4answers
16k views

How to parse hundred html source code files in shell?

I have a couple of hundred html source code files. I need to extract the contents of a particular <div> element from each of these file so I'm going to write a script to loop through each file. ...
9
votes
5answers
10k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
36
votes
7answers
5k views

Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this: $ zcat * you end up with a command line like zcat xyz.log xyz.log.1 ...
9
votes
6answers
1k views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to another file, without sed and awk? For example: $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
8
votes
6answers
12k views

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo ...
8
votes
6answers
16k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
12
votes
5answers
57k views

Remove last character from line

I want to remove last character from a line: [root@ozzesh ~]#df -h | awk '{ print $5 }' Use% 22% 1% 1% 59% 51% 63% 5% Expected result: Use 22 1 1 59 51 63 5
10
votes
1answer
4k views

How to remove duplicate lines in a large multi-GB textfile?

My question is similar to this question but with a couple of different constraints: I have a large \n delimited wordlist -- one word per line. Size of files range from 2GB to as large as 10GB. I ...
6
votes
3answers
1k views

Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format

I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...
13
votes
5answers
738 views

non-line-oriented tool for string replacement?

I recently asked a question about how to remove the newline character if it occurs after another specific character. Unix text-processing tools are very powerful, but almost all of them deal with ...
11
votes
5answers
18k views

How to remove blank lines from a file in shell?

I want to remove all empty lines from a file. Even if the line contains spaces or tabs it should also be removed.
6
votes
6answers
3k views

split file into two parts, at a pattern

How to split a large file into two parts, at a pattern? Given an example file.txt: ABC EFG XYZ HIJ KNL I want to split this file at XYZ such that file1 contains lines up-to XYZ and rest of the ...
6
votes
5answers
7k views

How to get all lines between first and last occurrences of patterns?

How can I trim a file (well input stream) so that I only get the lines ranging from the first occurrence of pattern foo to the last occurrence of pattern bar? For instance consider the following ...
3
votes
4answers
1k views

How to use sed command to modify a file's entry line by line in shell script?

This is my file1: #$BQ #{ VOL @home } #database daba #relation tcdeatid #copy 1 #{ version 0 } #opendb #clear #.lruno := 72 #.infno := 1 #.tid.noel := 101 #.tid.info ...
10
votes
8answers
719 views

sed in-place line deletion on full filesystem?

Due to an application bug as yet undiagnosed, I have several hundred servers with a full disk. There is one file that has been filled up with duplicate lines—not a log file, but a user environment ...
4
votes
5answers
739 views

How to replace only the Nth occurrence of a pattern in a file?

How to replace the third occurrence of the string in the file using sed command. Example: Change only the third occurrence of is to us in the file. My input file contains: hai this is linux. hai ...
4
votes
3answers
286 views

Merge files using zipper method / late merge

I am searching for a nice method of merging two or more files line by line using the zipper method (also called late merge). Assuming we have three files, the result should look like this: line1 ...
3
votes
4answers
3k views

add space 5 times at the beginning of each line in a text file

I am looking for a command line or bash script that would add space 5 times before the beginning of each line in a file. for example: abc after adding spaces 5 times abc
3
votes
2answers
3k views

Find the line number which contains the pattern using custom regex delimiter

As in example I'm trying to get line numbers which contains the pattern. My pattern contains slashes so I wanted to add custom delimiter. This simple one works: sed -n '/file/=' temp.txt Using ...
1
vote
1answer
179 views

How to nest global matches with sed?

If I do: sed 's/match/replace/g' I know that sed will substitute replace for every occurrence of match on a line. But what if...? echo "match <please dont match this?>" | sed ...
0
votes
2answers
81 views

put data in columns

I am trying to put this output in columns can anyone help? Routeur 2 ==> 89.89.156.112 If_Name :vRtrIfName.1.1 :vRtrIfName.1.2 Routeur 2 ==> 89.89.156.112 If_index 546 789 Routeur 2 ==> ...
8
votes
7answers
20k views

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...
4
votes
3answers
702 views

Removal of lines with no more or fewer than 'N' fields?

I am working on mac with sed, perl, awk, bash.. I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I ...
3
votes
2answers
389 views

Outputting common lines from 2 files and uncommon lines from both the files in one output file

I have 2 text files. Lets name them file1.txt and file2.txt file1.txt is as follows chr10 181144 225933 chr10 181243 225933 chr10 181500 225933 chr10 226069 255828 chr10 255989 267134 ...
3
votes
2answers
421 views

Merging files by rows

Is it possible to copy the whole rows of File1 in a new File3 following the instruction given by File2 by using a simple bash script using sed or awk? File1: /*two or more columns*/ AC 456324 DC ...
3
votes
2answers
157 views

Extract information from a text file

I have a text file, i want to search for tags such as the following: <category="SpecificDisease">Type II human complement C2 deficiency</category> <category="Modifier">Huntington ...
2
votes
2answers
62 views

Remove characters if they do not follow specified patterns

I want to clean up some files, and make make the way in which they are written more uniform. So, my input looks something like this: $a$h$l )r ^9 ^5 l \ urd The thing is, some spaces are ...
2
votes
2answers
2k views

How do I move a line in a text file up or down by one line?

I have some text files, and I'd like to be able to move an arbitrary line in any of the files up or down one line (lines at the beginning or end of the file would stay where they are). I have some ...
2
votes
2answers
93 views

Search and Replace question

I need to find all occurrences of AAsomeArbitraryStringBB and replace it with CCsomeArbitraryStringDD. So AAHelloBB Text AAByeByeBB becomes CCHelloDD Text CCByeByeDD. It's important to note ...
1
vote
3answers
401 views

Find records present in file 1 also in both file 2 and file 3

I have three files file1.txt, file2.txt, file3.txt and they are of same format. I want to select records from file1.txt which are present in both file2.txt and file3.txt based on column 2 and column ...
89
votes
14answers
103k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid of those messages?
34
votes
9answers
19k views

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
39
votes
17answers
96k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
31
votes
2answers
107k views

Return only the portion of a line after a matching pattern

So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
34
votes
8answers
18k views

How to obtain inverse behavior for `tail` and `head`?

Is there a way to head/tail a document and get the reverse output; because you don't know how many lines there are in a document? I.e. I just want to get everything but the first 2 lines of foo.txt ...
8
votes
3answers
1k views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
18
votes
4answers
17k views

How do I remove certain lines (using line numbers) in a file?

There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
16
votes
4answers
3k views

How to do a continous 'wc -l' with gnu texttools?

I know of course that cat logfile.txt | wc -l 120 will tell me the number of lines in a file. Whereas tail -f logfile.txt will show me the new lines that another program writes to logfile.txt. ...
15
votes
5answers
36k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
7
votes
6answers
14k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
19
votes
11answers
31k views

How to print all lines after a match up to the end of the file?

Input file1 is: dog 123 4335 cat 13123 23424 deer 2131 213132 bear 2313 21313 I give the match the pattern from in other file ( like dog 123 4335 from file2). I match the pattern of the line ...
17
votes
7answers
26k views

How to test whether a file uses CRLF or LF without modifying it?

I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and ...
13
votes
4answers
27k views

How to insert text before the first line of a file?

I've been looking around sed command to add text into a file in a specific line. This works adding text after line 1: sed '1 a\ But I want to add it before line 1. It would be: sed '0 a\ but I ...
13
votes
8answers
3k views

How to perform a multi line grep

How would you perform a grep for text that appears on two lines? For example: pbsnodes is a command I use that returns the utilization of a linux cluster root$ pbsnodes node1 state = free ...
12
votes
3answers
1k views

How can I redirect matching lines to a file, and non-matching lines to a different file?

Is there a script/program/utility already available for the following requirement in a optimised way? someCommand | tee >(grep "pattern" > LinesWhichMatch) | grep -v "pattern" > ...
10
votes
4answers
9k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...