Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

6
votes
3answers
296 views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
6
votes
6answers
2k views

split file into two parts, at a pattern

How to split a large file into two parts, at a pattern? Given an example file.txt: ABC EFG XYZ HIJ KNL I want to split this file at XYZ such that file1 contains lines up-to XYZ and rest of the ...
3
votes
4answers
1k views

How to use sed command to modify a file's entry line by line in shell script?

This is my file1: #$BQ #{ VOL @home } #database daba #relation tcdeatid #copy 1 #{ version 0 } #opendb #clear #.lruno := 72 #.infno := 1 #.tid.noel := 101 #.tid.info ...
17
votes
9answers
12k views

How can I print lines from file backwards (without using “tac”)?

I want to print lines from a file backwards without using tac command. Is there any other solution to do such thing with bash?
11
votes
7answers
11k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" ...
5
votes
7answers
1k views

Turn list into single line with delimiter

I have to take a list (loads) of IP addresses in this format: 134.27.128.0 111.245.48.0 109.21.244.0 and turn them into this format with a pipe in-between (IPs made up) 134.27.128.0 | ...
5
votes
3answers
3k views

Remove all lines in file A which contain the strings in file B

I have a CSV file users.csv with a list of userNames, userIDs, and other data: username, userid, sidebar_side, sidebar_colour "John Lennon", 90123412, "left", "blue" "Paul McCartny", 30923833, ...
5
votes
5answers
7k views

How to get all lines between first and last occurrences of patterns?

How can I trim a file (well input stream) so that I only get the lines ranging from the first occurrence of pattern foo to the last occurrence of pattern bar? For instance consider the following ...
4
votes
3answers
270 views

Merge files using zipper method / late merge

I am searching for a nice method of merging two or more files line by line using the zipper method (also called late merge). Assuming we have three files, the result should look like this: line1 ...
2
votes
2answers
127 views

match first column of file a with paragraphs of file b

I have 2 files. The first, fileA looks like TCONS_00000066 XLOC_000030 - u q1:XLOC_000030|TCONS_00000066|0|0.000000|0.000000|0.000000|0.000000|- TCONS_00000130 XLOC_000057 - ...
1
vote
1answer
43 views

transpose sets of three rows to columns

I have a file that has below entries: [root@ovmm329 ovm_utils]# cat /tmp/disk.txt Physical Disk : 'IBM (67) device=/dev/mapper/360050768018206483000000000000453 shareddisk=true Physical Disk : 'IBM ...
1
vote
1answer
148 views

How to nest global matches with sed?

If I do: sed 's/match/replace/g' I know that sed will substitute replace for every occurrence of match on a line. But what if...? echo "match <please dont match this?>" | sed ...
0
votes
2answers
80 views

put data in columns

I am trying to put this output in columns can anyone help? Routeur 2 ==> 89.89.156.112 If_Name :vRtrIfName.1.1 :vRtrIfName.1.2 Routeur 2 ==> 89.89.156.112 If_index 546 789 Routeur 2 ==> ...
6
votes
1answer
185 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
4
votes
3answers
438 views

Removal of lines with no more or fewer than 'N' fields?

I am working on mac with sed, perl, awk, bash.. I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I ...
3
votes
2answers
347 views

Merging files by rows

Is it possible to copy the whole rows of File1 in a new File3 following the instruction given by File2 by using a simple bash script using sed or awk? File1: /*two or more columns*/ AC 456324 DC ...
3
votes
2answers
142 views

Extract information from a text file

I have a text file, i want to search for tags such as the following: <category="SpecificDisease">Type II human complement C2 deficiency</category> <category="Modifier">Huntington ...
2
votes
2answers
87 views

Search and Replace question

I need to find all occurrences of AAsomeArbitraryStringBB and replace it with CCsomeArbitraryStringDD. So AAHelloBB Text AAByeByeBB becomes CCHelloDD Text CCByeByeDD. It's important to note ...
1
vote
3answers
291 views

Find records present in file 1 also in both file 2 and file 3

I have three files file1.txt, file2.txt, file3.txt and they are of same format. I want to select records from file1.txt which are present in both file2.txt and file3.txt based on column 2 and column ...
80
votes
11answers
92k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid off those messages?
39
votes
17answers
85k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
31
votes
9answers
17k views

sort but keep header line at the top

I am getting output from a program that first produces one line that is a bunch of column headers, and then a bunch of lines of data. I want to cut various columns of this output and view it sorted ...
29
votes
2answers
92k views

Return only the portion of a line after a matching pattern

So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
34
votes
8answers
17k views

How to obtain inverse behavior for `tail` and `head`?

Is there a way to head/tail a document and get the reverse output; because you don't know how many lines there are in a document? I.e. I just want to get everything but the first 2 lines of foo.txt ...
17
votes
6answers
37k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...
8
votes
2answers
1k views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
15
votes
4answers
3k views

How to do a continous 'wc -l' with gnu texttools?

I know of course that cat logfile.txt | wc -l 120 will tell me the number of lines in a file. Whereas tail -f logfile.txt will show me the new lines that another program writes to logfile.txt. ...
14
votes
5answers
32k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
7
votes
6answers
12k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
16
votes
3answers
16k views

How do I remove certain lines (using line numbers) in a file?

There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
20
votes
9answers
37k views

Command to display first few and last few lines of a file

I have a file with many rows, and each row has a timestamp at the starting, like [Thread-3] (21/09/12 06:17:38:672) logged message from code..... So, I frequently check 2 things from this log file. ...
12
votes
3answers
1k views

How can I redirect matching lines to a file, and non-matching lines to a different file?

Is there a script/program/utility already available for the following requirement in a optimised way? someCommand | tee >(grep "pattern" > LinesWhichMatch) | grep -v "pattern" > ...
8
votes
5answers
988 views

Modify a file without creating another file

Is there a way to modify a file without writing the contents to another file, without sed and awk? For example: $ cat test.txt aaa aaa bbb ccc ddd Replacing using sed with -i option, sed -i ...
15
votes
7answers
21k views

How to test whether a file uses CRLF or LF without modifying it?

I need to periodically run a command that ensures that some text files are kept in Linux mode. Unfortunately dos2unix always modifies the file, which would mess file's and folder's timestamps and ...
14
votes
4answers
63k views

Finding text between two specific characters or strings

Say I have lines like this: *[234]* *[23]* *[1453]* where * represents any string (except a string of the form [number]). How can I parse these lines with a command line utility and extract the ...
9
votes
7answers
6k views

Combining 2 different cut outputs in a single command?

I have a file such as the following: 1234 ABCD EFGH I'd like to convert it to the following: 2341 BCDA FGHE The actual file has 4,000 words, so I would like to do this in an efficient manner. I ...
8
votes
4answers
8k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
7
votes
7answers
352 views

Convert rows to columns

I have a file that includes details about VMs running in a hypervisor. We run some command and redirect the output to a file. And the is data available in the below format. Virtual Machine : OL6U5 ...
7
votes
5answers
2k views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
7
votes
4answers
5k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
16
votes
6answers
36k views

How to cut part from log file?

I have an 8 Gb log file (Rails production log). I need to cut it between some dates (lines). Which command could I use to do this?
12
votes
4answers
2k views

How to report “sed” in-place changes

When using sed to replace strings in-place, is there a way to make it report the changes it does (without relying on a diff of old and new files)? For instance, how can I change the command line ...
10
votes
5answers
1k views

Combining large amount of files

I have ±10,000 files (res.1 - res.10000) all consisting of one column, and an equal number of rows. What I want is, in essence, simple; merge all files column-wise in a new file final.res. I have ...
10
votes
2answers
2k views

Where has the trailing newline char gone from my command substitution?

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 ...
9
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
8
votes
3answers
3k views

Remove only the commas present within the double quotes

In a text file, I want to remove , (commas) and also the " (quotes) (only if the double quotes contains numbers seperated by commas). 56,72,"12,34,54",x,y,"foo,a,b,bar" Expected ouput ...
8
votes
2answers
1k views

Parsing XML, JSON, and newer data file formats in UNIX using command line utilities

The Unix environment has some excellent tools for parsing text in various forms. However, of late, the data is not in the traditional (historical) formats (CSV, TSV, record-based or some other ...
6
votes
3answers
8k views

How to remove lines included in one file from another file in bash?

I have two lists, a complete list and a partial list. I want a bash script that will search the complete list for any lines that are in the partial, if there are, they will be removed from the ...
4
votes
2answers
512 views

Extract only chinese characters

I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this? I tried: [The range of Chinese Unicode chars is 4E00 thru 9FFF (344 ...
3
votes
2answers
370 views

How to write this pseudocode where condition

This question is extension of this question about listing questions of directories. I get a problem that I need to understand the original section of the text. Current output where you see the ...