sed is a command-line stream editor for filtering and transforming text.

learn more… | top users | synonyms

6
votes
4answers
141 views

Substitute Column With Result of Command

I'm trying to find a way to quickly and efficiently replace a column value for every row in a file with the output of a command. I need to work on multiple files with about 500,000 rows each daily so ...
6
votes
2answers
341 views

Delete multiple lines in a file

Usually I find my answers in the previously answered questions, but not this time (or I don't understand the answers and can't modify them to do what I want them to do), so I'm asking my first ...
6
votes
1answer
694 views

How to ensure that string interpolated into `sed` substitution escapes all metachars

I have a script that reads a text stream and generates a file of sed commands that is later run with sed -f. The generated sed commands are like: ...
6
votes
6answers
7k views

SED : Delete 4 line above and below 5 line after pattern match

I have Hugh file with following details : define host{ use generic-host ; Name of host template to use host_name ServerA_172.29.16.102 ...
6
votes
4answers
9k views

Why is sed giving me an error about an unterminated `s'?

I have a set of sed replacements in a bash script and am getting an error about an unterminated `s' command. Here's what the sed line looks like: sed -n -e "s/TMPFOO1/$FOO1/" -e "s/TMPFOO2/$FOO2/" ...
6
votes
5answers
15k views

Use sed to find whole word and replace

I have the following block of text in a file: test3.legacy test4.legacy test3 test3.kami I only want to search for test3 as a whole and replace it with nothing. Unfortunately, all my attempts have ...
6
votes
1answer
11k views

Substituting the first occurrence of a pattern in a line, for all the lines in a file with sed

Is it possible to do it in one liner? I have an output like this: "First line" - Description " Second line" - Description "Third line" - Description " Fourth line" - Description This input ...
6
votes
2answers
1k views

How would I sort these directory names numerically?

I'm aware that I can somehow sort this output numerically (so cpu1/ follows cpu0/) ... I could probably get something to work eventually by splitting up the string various ways with awk, but is there ...
6
votes
3answers
10k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
6
votes
2answers
409 views

Execute a command within 'sed'

I have a list of random memory addresses saved in a text file, and I need to sed them out, decode them, and save them back to my file. However, the decoding operation I'm trying to perform is handled ...
6
votes
3answers
4k views

Regex & Sed/Perl: Match word that ISN'T preceded by another word

I'd like to use sed or perl to replace all occurrences of a word that doesn't have a certain word in front of it. For example, I have a text file that contains a plot of a movie and I want to replace ...
6
votes
1answer
161 views

Why can't I escape the braces “]” in sed?

Question: Why can't I escape the "]"? $ printf "*#$@~\!&*()[]\n" *#~\!&*()[] $ printf "*#$@~\!&*()[]\n" | sed 's/[*#$@~!&*()\[\]]//g' *#~\!&*() $
6
votes
3answers
541 views

Replace uppercase to lowercase except first letter of each word in specific line

I want to replace names that I received in uppercase letters only. The names are mixed with other information which should stay as it is. AUTH: at the beginning of the line identifies it as a line ...
6
votes
1answer
322 views

Variable in sed

I have a file that contains filenames with absolute path. In the middle of that path, is a variable directory, whenever I hard code the variable part, it works fine, but when I replace it with the ...
6
votes
1answer
2k views

Replace non-printable characters in perl and sed

I need to replace some non-printable characters with spaces in file. Specifically, all characters from 0x00 up to 0x1F, except 0x09 (TAB), 0x0A (new line), 0x0D (CR) Up until now, I just needed to ...
6
votes
2answers
417 views

sed replace only lines with matching groups

Currently, I have this command: sed 's/\([^C]*\)/\1AB/' It inserts "AB" before the first instance of "C" on a line. But it is also adding "AB" at the end of every line without a "C". How can I ...
6
votes
1answer
185 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
6
votes
2answers
343 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
6
votes
2answers
4k views

Remove unknown unicode character from textfiles - sed, other bash/shell methods

I need to search and replace all occurences of an unknown character in some files having the same name. Opening such files with vi, I read <91> code for that character. Opening them with nano, I ...
6
votes
2answers
302 views

Regex `/pattern/g` and ed `:g/pattern/`: which came first, and why `g`?

The g option (e.g. s/pattern/replacement/g) for many tools that use regex-style pattern matching and the :g command in ed, ex, vi, and vim have pretty similar usage and meaning: match the given regex ...
6
votes
3answers
2k views

Multi-line replace

I spent ages trying to figure this out. As it happens, I don't need to do it any more; I found another way. But for the sake of my sanity, I'd like to know the answer anyway. It's quite easy to use ...
6
votes
1answer
83 views

How to extract first base64 code out of file?

On Linux I'm having text files and each contains 1 or 2 base64 blocks of code. Following is part of the file and 1 base64 block. Content-Transfer-Encoding: base64 ...
6
votes
2answers
312 views

How come sed is so fast?

I thought about whether this question is suitable for SE or not, I hope you agree it is. Some time ago I asked on SE how to find text in files and leave the file with only the matching lines that ...
5
votes
8answers
4k views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
5
votes
4answers
2k views

How to append end of every line with the line itself

This is in file.txt: redcar bluecar greencar Im looking for ways to make it become: redcar redcar bluecar bluecar greencar greencar I've tried many ways using sed with no luck
5
votes
5answers
1k views

How to replace a word with new line

I have a text file with following data and each row ends with |END|. T|somthing|something|END|T|something2|something2|END| I am tryig to replace |END| with \n new line with sed. sed ...
5
votes
8answers
545 views

awk or sed command to match regex at specific line, exit true if success, false otherwise

I need to determine if a file contains a certain regex at a certain line and to return true (exit 0) if found, and otherwise false. Maybe I'm overthinking this, but my attempts proved a tad unwieldy. ...
5
votes
4answers
639 views

Remove next-to-last extension in a file name

Suppose a folder with many files with bad extensions,for example Song1.avi.mp3 Song2.avi.mp3 Song32.web.mp3 Song23.mp4.mp3 Song2a.mp9.mp3 I want to remove only the second field (web, avi, mp4). I ...
5
votes
5answers
217 views

How to SED these paragraphs to MCQ format?

My data: Question Nr. 311 Main proteins are in the lorem ipsun A Lorem RNA test B Cells C Metoclom D Cells E Muscles Question Nr. 312 Main proteins are in the lorem ipsun A Lorem B Cells C ...
5
votes
5answers
2k views

Sed: extending a string of numbers to a certain number of digits by padding with zeros

I'm currently trying to modify an echoed string of alphanumeric characters by piping it to sed. The area I'm interested in modifying is a block of numbers, which I would like to make 8 characters long ...
5
votes
4answers
487 views

Reduce the over use of files in text processing

Currently, I have a file (Results.txt) which has 27k rows. The file content is as below. 45008657 | A. | | Long | 0 49210987 | A. | ...
5
votes
3answers
801 views

Why does sed act differently depending on the output file?

If I run: cat messages.txt | sed -e 's/a/a/g' > messages.txt on one large file (2500+ lines) I find that the resulting file will only have about 900 lines after the command in cygwin and will ...
5
votes
7answers
2k views

Replace a character except last x occurences

I have a file that has a bunch of hostnames correlated with IPs that looks like this: x-cluster-front-1 192.168.1.2 x-cluster-front-2 192.158.1.10 y-cluster-back-1 10.1.11.99 y-cluster-back-2 ...
5
votes
2answers
8k views

How can I filter those “Permission denied” from find output? [duplicate]

I started using sed recently. One handy way I use it is to ignore unimportant lines of a log file: tail -f example.com-access.log | sed '/127.0.0.1/d;/ELB-/d;/408 0 "-" "-"/d;' But when I try to ...
5
votes
4answers
1k views

shorten long lines in a log file

I keep a logfile of a program's output. The trouble is sometimes errors occur dumping very long lines of 7bit ascii data(encoded binary) I don't care to keep. These lines can be 200KB+ long before ...
5
votes
2answers
1k views

Delete every Nth line in shell

I'm trying to delete a line after N lines using awk and I can't seem to get it right. The file format is like this YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX The real example would be office3 3 1 ...
5
votes
4answers
14k views

Convert all text from uppercase to lowercase and vice versa?

My question is how can I convert all text from uppercase to lowercase and vice versa? That is to change the cases of all the letters. It has to be done with a sed replacement somehow.
5
votes
5answers
378 views

Converting long numbers to hex with sed

I'm trying to make a sed command so that numbers longer than 3 digits were converted to hex. I.e. a string like 124 3275 7535 should result in 124 0xccb 0x1d6f. Here's what I currently have: sed ...
5
votes
7answers
5k views

can we print the last word of each line in linux using sed command?

suppose , if there is a file consisting of following lines , if they are 12345 567 7878 66 er3 t45t y6y46y 4y6 y656y y5y 46y6 65y7 y66uyuy yy46y6y The output has to look like: 66 ...
5
votes
7answers
294 views

Remove lines from tab-delimited file with missing values

I have a large (~900MB) tab-delimited text file that I will be processing in a downstream program. I need to delete any line with a missing value. The correct number of columns are on each line (so ...
5
votes
6answers
479 views

sed command to swap characters

My input file layout is: mm/dd/yyyy,hh:mm,other fields I need to format it as: yyyy-mm-dd hh:mm:00,other fields sample input: 01/02/1998,09:30,0.4571,0.4613,0.4529,0.4592,6042175 ...
5
votes
3answers
2k views

Deleting Lines if there is a dot in the 5th Column

I have a file and I want to remove all the line if it has a dot in the 5th Column. Input File: sc2/80 20 . A C 80 PASS N=2 F=5;U=4 sc2/60 ...
5
votes
2answers
2k views

sed pattern space and hold space

i was just going thorugh this and i was unable to understand. sed -n '1h; 1!H; ${ g; s/foo\nbar/bla\nblub/ p }' file. where cat file is : foo foo bar why there is 1!H because to me 1h ...
5
votes
5answers
7k views

How to get all lines between first and last occurrences of patterns?

How can I trim a file (well input stream) so that I only get the lines ranging from the first occurrence of pattern foo to the last occurrence of pattern bar? For instance consider the following ...
5
votes
2answers
178 views

Command to remove the first N number of lines in input

Background I am running an SSH server and have this user that I want to delete. I cannot delete this user because he is currently running a few processes that I need to kill first. This is the ...
5
votes
2answers
2k views

sed: read whole file into pattern space without failing on single-line input

Reading a whole file into pattern space is useful for substituting newlines, &c. and there are many instances advising the following: sed ':a;N;$!ba; [commands...]' However, it fails if the ...
5
votes
3answers
2k views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
5
votes
4answers
633 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...
5
votes
2answers
541 views

Does `sed` exist in all *nixes?

My application will be running on all versions of Unix-based OS, say, Linux, MacOS, HP-UX, Sun Solaris , OSX, etc. I'm using the following sed command in a common place to delete the string matched ...
5
votes
5answers
8k views

How can I convert tab delimited data to comma delimited data?

I'm requesting a list of ec2 snapshots via amazon's ec2 command line tool: ec2-describe-snapshots -H --hide-tags > snapshots.csv The data looks something like this: SnapshotId VolumeId ...