Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

1
vote
1answer
20 views

Filtering and extracting PIDs as space-separated list from ps using sed

I want to filter and format the output of ps -ax -o pid,command, but not sure what pipe to use for it. I want to filter those lines matching a fixed string at a fixed position, just keep the pids for ...
17
votes
9answers
13k views

Text processing - join every two lines with commas

I have more than 1000 lines in a file. The file starts as follows (line numbers added): 1. Station Name 2. Station Code 3. A N DEV NAGAR 4. ACND 5. ABHAIPUR 6. AHA 7. ABOHAR 8. ABS 9. ABU ROAD 10. ...
6
votes
5answers
794 views

Grouped sorting of continuous paragraphs (separated by blank line)?

I think I'm pretty experienced now in sorting by columns; however, I haven't found anything so far how to sort continuous rows. Supposing we have a text file that looks like this: (very simplified, ...
0
votes
5answers
37 views

update the json file

I have a file, actually json file. How I can add a comma to the end of every but the last line ? For example: {"a":1, "b":1, "c":2} {"a":3, "b":3, "c":1} {"a":1, "b":2, "c":3} What I want: {"a":1, ...
2
votes
2answers
27 views

adding column in CSV file using awk

I need to a add a column in CSV file from an array using awk. For example, input.csv a,10 b,11 array = (100 200) output.csv should look like a,10,100 b,11,200 I tried this command but it does ...
2
votes
1answer
35 views

Question about replacing character in text file

So I have some data that is parsed in this format seperated by colons: username:email@email.com:password:salt Sometimes the username/salt can contain a colon (:) in it. I want to replace the ...
2
votes
3answers
88 views

FNR and NR variables problem

I have two files, one of them has 10 records, and the other one with 15 record, the problem is when I cat them together, always FNR=NR. For example, consider below files: File1: 1,boo 2,foo 3,boo ...
0
votes
2answers
38 views

How to search between the 2nd and 3rd delimiters

So, I have a data file and I would like a match to occur only if a match is found between the 2nd and third Vertical Bars (|) So given this data sample, I if I search for 'wilson' I want the 2nd line ...
19
votes
9answers
3k views

Grep lines but let the first line through [duplicate]

Consider the following output from df. Filesystem Size Used Avail Use% Mounted on /dev/sda1 23G 6.1G 16G 29% / udev 10M 0 10M 0% /dev tmpfs 397M 420K ...
4
votes
2answers
189 views

grep string where next line does not contain string

I want to search all files inside a directory and its subdirectories for lines containing a certain string, but I want to exclude those results that contain a different certain string in the line ...
3
votes
4answers
210 views

Conditional if clause in awk

Please consider below file: foo,1000,boo,A foo,1000,boo,B foo,1001,boo,B foo,1002,boo,D And we have below rules: If $2 equal 1000, $4 should be equal A If $2 equal 1001, $4 should be equal B If ...
0
votes
1answer
34 views

How to add new column, where the value is based on existing columns with awk

I have a file of multiple columns. I would like to make an additional column based on the values of 2 columns from this file. Example input: A B C D E F 1 2 T TACA A 3 2 3 4 I R 8 2 9 3 A C 9 3 ...
3
votes
3answers
47 views

Find a pattern and insert # at the beginning of 2 line befoe that and 1 line after that

I have a big file (more than 2000 lines). Where i have to insert a # at the beginning of 2 above lines and at the beginning of 1 below line after finding a pattern. Also insert # at the beginning of ...
4
votes
4answers
129 views

Current date in awk

Is there a constant variable in awk, that store today's date? if no, is there a way that can store today's date for daily use? let's say we have below file: boo,foo,2016-08-30 foo,boo,2016-07-31 ...
0
votes
5answers
68 views

Appending One File to the Second Until End of File

I have two files. The first is just a single number. This number is calculated and changes every time. The second file has a bunch of columns, and I do not know the number of rows (which can change). ...
4
votes
6answers
192 views

Changing matrices into single lines of data

I have multiple matrices in a .txt file and I need each matrix to be in a single line. For example, matrices.txt: 1 2 3 4 2 3 4 5 3 4 5 6 3 4 5 6 2 3 2 5 2 3 4 5 2 3 5 6 2 3 4 5 ... What I would ...
1
vote
1answer
42 views

How to print the line numbers with corresponding line that matches a pattern using shell command? [duplicate]

File, .false alarm is bla no. 11 .no alarm generated is bla .application will be killed is bla wall command .doc file is document file with .doc extension .no authority for this selection bla Now ...
1
vote
2answers
32 views

How to replace string in files except with a line begins with # [duplicate]

I've looked at some other questions related to using sed to replace strings, but I'm having trouble figuring out how to apply it to my particular need. In a case like the snippet below, I want to ...
1
vote
5answers
116 views

format file to remove " characters

I have a file with the following data "MG1507XXXXXX|" "|020000XXXXXX" "20261031|" "|3,827.92" "|3,581.41" "|542,729.62" "MBA" "MG1507XXXXXX|" "|020000XXXXXX" "20261130|" ...
0
votes
1answer
82 views

How to replace \1\ with new line

I have a record like: "evSchema"\1\"UAT" "evSN"\1\"uadb" "evDirsep"\1\"/" "evRootPath"\1\"/work_area/APP_UAT/" "evSchema"\1\"RMS13" "evUser"\1\"STAGE" I want to replace \1\ with new line. I tried ...
3
votes
2answers
441 views

Need shell script to transform a CSV into Apache httpd format

Need some pointing in right direction on script to fetch and regex or sed. Site24x7 provides a URL with a CSV list of their source IP's used for monitoring. (they also provide other formats, CSV ...
-1
votes
1answer
28 views

replace numbers with zero while we have the lines number [on hold]

I am trying to write a bash script (containing awk or sed ) to replace some numbers with zero. for example : replace line 3 second and third number, and line 4 first and third number and line 5 first ...
1
vote
3answers
24 views

fetch a column from a file based on another column

I have a file which contains approximately 5 million records, as follows: - 1223423,21,foo,data1,data2,data3,data4,data5,45,267,index1 4234234,34,bar,cat1,cat2,cat3,cat4,cat5,34,2323,index2 ...
0
votes
1answer
33 views

Repeatedly extract every second and third line of a file with 3-line data blocks

I have a file where I have data in 3-line sets. I want the 2nd and 3rd line from each set. How do I get that? For an example from: [set no][no of line in set] 11 12 13 21 22 23 31 32 33 I want: ...
11
votes
4answers
343 views

Sort lines by number of words per line

Given input: hello: world foo bar baz bar: baz: bin boop bop fiz bang beep bap: bim bam bop boatkeeper: poughkeepsie I would like to sort it into most words at the top, to least at the end, like ...
4
votes
2answers
2k views

grep one line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...
19
votes
4answers
28k views

Grabbing the first [x] characters for a string from a pipe

If I have really long output from a command (single line) but I know I only want the first [x] (let's say 8) characters of the output, what's the easiest way to get that? There aren't any delimiters.
6
votes
3answers
179 views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
0
votes
4answers
988 views

How to get data count between two strings with multiple occurences in the same file

I have the data in my file as START-OF-FIELDS ID NAME DEPT END-OF-FIELDS START-OF-DATA 1|joy|cs 2|sam|ec END-OF-DATA START-OF-FIELDS ID NAME DOB DEPT ADDRESS END-OF-FIELDS START-OF-DATA ...
2
votes
3answers
61 views

remove lines that contain an specific text in a file

I have a file with a list of emails in it and each line has an email in it. I want to remove lines that contain example.com or test.com. I tried this: sed -i 's/example\.com//ig' file.txt But it ...
2
votes
2answers
63 views

Extract multiple instances of text between two words inclusive of starting word but exclusive of ending word

A PDB file contains numerous paragraphs of conformations of a protein. Each conformation starts with the keyword ATOM and ends with keyword END. I am trying to read the file in bash such that I ...
0
votes
2answers
92 views

Get a CSV file containing names of files within a directory?

I am trying to figure out how to export the names of the files I have in a directory to be saved in a CSV file that I can edit in Excel. The directory looks like this: $ ls Sample_38_41_1_A01 ...
0
votes
1answer
42 views

Read all the values of a particular column of a file and store it in another file

I have a file.txt which contains data like below: col11 col12 col13 col21 col22 col23 I fetch 2nd column of all the available rows (no. of rows may vary) using awk '{print $2}' file.txt Now i ...
1
vote
1answer
306 views

grep (/sed/awk) month range

I've looked around for solutions to this, but it's sort of difficult to search for the solution to such a specific problem (lot of words to put in Google). So, I have a file with data that looks like ...
0
votes
3answers
49 views

Append a string to the beginning of second line using unix

I want to append a string to the beginning of second line of my text file, for example: 1 2 3 should become: 1 42 3 Any idea about how to go about with this thing?
5
votes
5answers
221 views

Merging columns in a file using Awk

Input: ABC,SSSD,12345,NSS,12345,xxx,TS11 ,,,,,,TS21 ,,,,,,TS22 ,,,,,,BS26 ,,,,,,GPRS ABC,SSSD,12356,NSS,12356,xxx,TS11 ,,,,,,TS21 ,,,,,,TS22 ,,,,,,GPRS ,,,,,,BS26 Output: ...
32
votes
8answers
3k views

Is there a tool that combines zcat and cat transparently?

When handling log files, some end up as gzipped files thanks to logrotate and others not. So when you try something like this: $ zcat * you end up with a command line like zcat xyz.log xyz.log.1 ...
0
votes
2answers
41 views

Monitor a logfile and execute different commands depending on different conditions

I have a logfile which I want to monitor and depending on which condition is met, different commands should be executed. I found a solution that comes close to this here. Unfortunately it doesn't ...
2
votes
4answers
56 views

Join pattern line and consecutive non-empty lines, separated by commas and enclosed in ()

I have a file that has click css_add_violation_false click css_add_claim_false and select css_driver1_birth_date_month birth_date_month select css_driver1_birth_date_day birth_date_day How ...
2
votes
2answers
6k views

Convert csv to HTML table

I have a Medical.csv file with rows of following format, field: 'participation.type', displayName: 'program_type', type: 'String',path:'participation' field: 'participation.program', ...
10
votes
6answers
455 views

Remove lines from a file depending on lines found in another file

File file1.txt contains lines like: /api/purchase/<hash>/index.html For example: /api/purchase/12ab09f46/index.html File file2.csv contains lines like: <hash>,timestamp,ip_address ...
1
vote
3answers
54 views

Find lines in (muliple) files that contain the given line in an input file

So, if I have a number of files like the following: file1: 123 456 789 012 file2: line1 922 line2 392 line3 456 line5 291 line6 201 ... file3: line1 111 line2 123 line3 19 line5 542 ...
1
vote
2answers
47 views

How to subtract lines, in reverse, only if greater than previous line

I am working on a shell script that will read the lines of a file, in reverse. I need it to subtract the previous line from the latter, but only if the latter is larger. If it's smaller, then I just ...
3
votes
4answers
58 views

sed - how to remove all lines that do not match

I have a html file. I want to remove all lines that do not start with <tr>. I tried: cat my_file | sed $' s/^[^tr].*// ' | sed '/^$/d' but it deleted all the lines.
0
votes
2answers
30 views

How to add a new text after every matched string using shell command?

Input: Job name: ns, Job ID: 2312, Status: ODB_ACTIAVTION SUCCESSFUL Job name: ps, Job ID: 3353, Status: ODB_ACTIAVTION SUCCESSFUL Job name: pm, Job ID: 1265, Status: ODB_ACTIAVTION SUCCESSFUL Job ...
5
votes
3answers
46 views

Censor text with regex

I'm currently running this command to censor an indented to-do list. sed -e 's/\(\s\+- \)\(.*\)/\1XXX/g' It's great except that I'd like the number of Xs to match the number of matched characters. ...
4
votes
2answers
92 views

POSIX behavior of sed's ^ and $ regex anchors with multiline pattern spaces

What is the POSIX behavior of the sed ^ and $ anchors when used at the beginning and end of a regex if the pattern space contains embedded newlines, like after you use the N command? Should they ...
0
votes
1answer
2k views

shell script to print rows if there is a value in column 2

1 2 3 4 3 1 4 2 1 4 3 2 4 2 1 3 How can I print all rows, on which there is a 2 in the second column.
71
votes
12answers
38k views

how do you sort du output by size?

How do you sort du -sh /dir/* by size? I read one site that said use | sort -n but that's obviously not right. Here's an example that is wrong. [~]# du -sh /var/* | sort -n 0 /var/mail 1.2M ...
8
votes
4answers
730 views

Splitting file for every 10000 numbers ( not lines )

I have a file that looks like the following: chr19 61336212 + 0 0 CG CGT chr19 61336213 - 0 0 CG CGG chr19 61336218 + ...