Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

7
votes
2answers
1k views

How to do a text replacement in a big folder hierarchy?

I want to search and replace some text in a large set of files excluding some instances. For each line, I want a prompt asking me if I need to replace that line or not. Something similar to vim's ...
19
votes
9answers
29k views

Command to display first few and last few lines of a file

I have a file with many rows, and each row has a timestamp at the starting, like [Thread-3] (21/09/12 06:17:38:672) logged message from code..... So, I frequently check 2 things from this log file. ...
12
votes
4answers
54k views

Finding text between two specific characters or strings

Say I have lines like this: *[234]* *[23]* *[1453]* where * represents any string (except a string of the form [number]). How can I parse these lines with a command line utility and extract the ...
11
votes
1answer
549 views

How to do `head` and `tail` on null-delimited input in bash?

find command can output names of files as a null-delimited strings (if -print0 is provided), and xargs can consume them with -0 option turned on. But in between, it's hard to manipulate that ...
9
votes
4answers
22k views

How do I append text to the beginning and end of multiple text files in Bash?

I have a directory full of text files. My goal is to append text to the beginning and end of all of them. The text that goes at the beginning and end is the same for each file. Based on code I got ...
9
votes
7answers
4k views

Combining 2 different cut outputs in a single command?

I have a file such as the following: 1234 ABCD EFGH I'd like to convert it to the following: 2341 BCDA FGHE The actual file has 4,000 words, so I would like to do this in an efficient manner. I ...
7
votes
5answers
1k views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
2
votes
6answers
7k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
27
votes
7answers
3k views

Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this: $ zcat * you end up with a command line like zcat xyz.log xyz.log.1 ...
16
votes
6answers
30k views

How to cut part from log file?

I have an 8 Gb log file (Rails production log). I need to cut it between some dates (lines). Which command could I use to do this?
10
votes
2answers
2k views

Where has the trailing newline char gone from my command substitution?

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 ...
8
votes
3answers
2k views

Remove only the commas present within the double quotes

In a text file, I want to remove , (commas) and also the " (quotes) (only if the double quotes contains numbers seperated by commas). 56,72,"12,34,54",x,y,"foo,a,b,bar" Expected ouput ...
8
votes
4answers
2k views

How do I find the overlap of two strings in bash? [closed]

I have two strings. For the sake of the example they are set like this: string1="test toast" string2="test test" What I want is to find the overlap starting at the beginning of the strings. With ...
8
votes
2answers
1k views

Parsing XML, JSON, and newer data file formats in UNIX using command line utilities

The Unix environment has some excellent tools for parsing text in various forms. However, of late, the data is not in the traditional (historical) formats (CSV, TSV, record-based or some other ...
6
votes
3answers
6k views

How to remove lines included in one file from another file in bash?

I have two lists, a complete list and a partial list. I want a bash script that will search the complete list for any lines that are in the partial, if there are, they will be removed from the ...
5
votes
5answers
8k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
3
votes
2answers
227 views

How to write this pseudocode where condition

This question is extension of this question about listing questions of directories. I get a problem that I need to understand the original section of the text. Current output where you see the ...
11
votes
3answers
661 views

/usr/bin/ptx: Can you provide a use case or two?

I was going through the list of files included in coreutils and I was able to come up with an example of how I could personally use all of the commands provided except for ptx. Can you give one or two ...
9
votes
5answers
4k views

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
8
votes
2answers
384 views

How can I delete all text between nested curly brackets in a multiline text file?

This question comes from How can I delete all text between curly brackets in a multiline text file? (just the same, but without the requirements for nesting). Example: This is { {the multiline ...
8
votes
5answers
636 views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
8
votes
5answers
3k views

`highlight` command

Is there a command which can be used to easily highlight (with colors, bold text or inverted back-/foreground) specific strings in stdin while passing the entire string to stdout? I'm thinking of ...
7
votes
4answers
359 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
5
votes
7answers
511 views

Turn list into single line with delimiter

I have to take a list (loads) of IP addresses in this format: 134.27.128.0 111.245.48.0 109.21.244.0 and turn them into this format with a pipe in-between (IPs made up) 134.27.128.0 | ...
4
votes
1answer
2k views

How does one find and replace text in all open files with geany?

How does one find and replace text in all open files with geany?
3
votes
2answers
289 views

Find the line number which contains the pattern with custom delimiter

As in example I'm trying to get line numbers which contains the pattern. My pattern contains slashes so I wanted to add custom delimiter. This simple one works: sed -n '/file/=' temp.txt Using ...
3
votes
5answers
2k views

How to align fstab entries easily

Editing /etc/fstab with my text editor makes all the spacing go out of alignment. I could go through the file and insert/delete spaces to line everything up, but I'm looking for a more automated ...
3
votes
1answer
312 views

What is the difference between > and >> (especially as it relates to use with the cat program)?

Suppose that I have a file called temp.txt. Using the cat program, I would like to add the contents of this file to the end of myfile.txt -- creating myfile.txt if it does not exist and appending to ...
2
votes
4answers
138 views

Why isn't this awk command doing a full outer join?

Objective: Merge the contents of two files using common key present in the files file1.txt ========= key1 11 key2 12 key3 13 file2.txt ========= key2 22 key3 23 key4 24 key5 ...
2
votes
3answers
231 views

how to create a data file with a column of date from day1 to dayN?

I was wondering if I could generate a hourly or daily time step date in a column starting from some year in the past to some day in the near past or today. to be more clear, I want to create one ...
1
vote
1answer
94 views

How to merge two files in the same row?

I have two files. file1: Dave 734.838.9800 Bob 313.123.4567 Carol 248.344.5576 Mary 313.449.1390 Ted 248.496.2204 Alice 616.556.4458 file2: Bob Tuesday Carol Monday Ted Sunday ...
1
vote
2answers
861 views

Merging 2 files with based on field match

I want to create a file that contains columns from two input files. File1 is like: aa 32 bb 15 cc 78 File2 is: fa 19 bc 23 cc 50 de 28 aa 45 bb 31 The task is, read through File1, if the 1st field ...
10
votes
3answers
558 views

wc -L reports a line-length of 8 for a tab-char. bug or feature?

-L is a useful feature of wc, or so I thought. It prints the length of the longest line. For some reason it expands a single-byte tab-char to a length of 8. Is there some way to set this to not ...
6
votes
1answer
105 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
6
votes
6answers
1k views

Split large file into chunks without splitting entry

I have a rather large .msg file formatted in the UIEE format. $ wc -l big_db.msg 8726593 big_db.msg Essentially, the file is made up of entries of various length that look something like this: ...
6
votes
4answers
19k views

Shell script to send an alert mail if disk usage exceeds 90%

I am new to shell scripting and want to implement a script on my server which will automatically send email alerts if the disk usage goes over 90% and second email alert when it goes over 95%. My ...
6
votes
3answers
3k views

Split file into multiple small files, separate by the newline symbol

Is there a utility that split file by newline symbol? e.g if a file contains the following lines, aa bbb cccc If I want to split it to 3 files, the desired output would be: aa, bbb And cccc (in 3 ...
6
votes
2answers
512 views

sort every field numerically, varying field counts

I'm trying to sort some data using sort. I noticed it was sorting by digit rather than number, so I added the -n flag. It then seemingly only numerically sorts on the first field though. Breaking it ...
6
votes
3answers
35k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
6
votes
2answers
4k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...
5
votes
3answers
353 views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
5
votes
4answers
3k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
4
votes
2answers
97 views

Replace 1st with 1), 2nd with 2), … in GNU Sed

Pseudocode which is a continuation from this answer gsed 's/|/1)/g' 's/|/2)/2g' 's/|/3)/3g' 's/|/5)/4g' 's/|/5)/5g' <input.csv >output.csv which is of course not working. I am interested in ...
4
votes
2answers
131 views

how to sort by the day of the week?

I have two files. file1: Dave 734.838.9800 Bob 313.123.4567 Carol 248.344.5576 Mary 313.449.1390 Ted 248.496.2204 Alice 616.556.4458 file2: Bob Tuesday Carol Monday Ted Sunday ...
4
votes
5answers
626 views

What command(s) will feed a tab-delimited text file and cut each line to 80 characters?

I've got multiple-line text files of (sometimes) tab-delimited data. I'd like to output the file so I can glance over it - so I'd like to only see the first 80 characters of each line (I designed the ...
4
votes
3answers
893 views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
4
votes
2answers
362 views

tac command's option creates strange output [comprehension question]

Say I have this file, containing nothing but a b c b a Using tac --separator=a file in BASH [on a Debian based Linux], I get this: # empty line # empty line b c ...
4
votes
3answers
909 views

Delete the matching line and several more from a file

I have a text file called file_a.txt. My first command is grep -A 12 ".production =" file_a.txt The output is a few block. Each block of string contains 13 rows I specifically want to delete all ...
3
votes
6answers
327 views

redirect output to specific line number

I want to grep certain lines by keyword and redirect the output to specific line number of an existing file. Command grep "key" temp_file >> desired.txt What I need is that I can append the ...
3
votes
2answers
115 views

How can I partition a set of words given pairs that must end up together?

I have a space or comma separated table with two columns, each row representing equivalence of the two words. A B B C B D C E F G What I want is a table with each row listing all mutually ...