Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

3
votes
4answers
93 views

Grep the text which matches with a pattern

I was looking for a command which will print the selected portion of a line which matches the conditions. Eg, my text line is like this: p1=X||p2=Y||p3=X||p4=X||p5=X||p6=Y||p7=X I want to print ...
1
vote
2answers
31 views

Escaping double quotes in string

I have a string message which contains data in double quotes. I need to put \ in front of every double quote how can u do it in shell ? I used Perl and it works fine, but I'm looking for sh command: ...
2
votes
2answers
27 views

Shell script - text processing help

I want to create a ldif modify file to update the users on LDAP. I have an input file(alluserdns.ldif) with list of all Dn entries in ldap: dn: cn=testuser2@abc.com, ou=persons,ou=pp,dc=abc,dc=com ...
1
vote
0answers
37 views

Convert / import flat file with multiple records, one variable per line

I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or ...
2
votes
3answers
111 views

How to print the before lines of search keyword with out using grep in a gz file?

How to print the lines before a search keyword without using grep in a .gz file? ID:342N000390AAAAAAAA 07/14/15 10:26 (MV90 ) * Register Data Imported * Warning - No ...
0
votes
1answer
26 views

How to replace the content of specific column with awk? Tab Delimited File

Here's a similar question but I ran into a problem that was not addressed in the question. How to replace the content of specific column with awk? I want to do something similar, however my file is ...
1
vote
1answer
19 views

AWK: where is my error?

I have a file with :-delimited fields. If the 3rd field of a line is greater than 22, that line should be printed to file44. The file looks like this: 1:2:3:4 1:2:3:411 1:2:33:42 1:2:3:44444 ...
5
votes
4answers
126 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
1
vote
1answer
25 views

Loop over lines of file, partition file and classify each partition

In the following I've put up my code for iterating over lines of a file in order to collect data within a segment from the file in following fashion: Illustration for understanding procedure: segL ...
1
vote
4answers
133 views

UNIX to print a content from a .gz file

I have a .gz file with contents like the following: ID:123455 23-07-2015 mvni warning: RTM post warning : no profile data Using the string no profile data, I want to print the ID. I tried with ...
2
votes
3answers
55 views

Read a column from a file and append a specific column of another file

I have two files as such: fileA: Name,Site Name,Product UPEI,Product Name,NMS Flag,Product Model, SW FileB: NE_Name SW ABC 4.4 ASD 4.3 ... ... Note that ...
3
votes
4answers
94 views

Efficient data extraction from multiple files to a single CSV file

I have a large collection of XML files with the same exact structure: $ cat file_<ID>.xml ... ... ... <double>1.2342</double> <double>2.3456</double> ...
2
votes
5answers
154 views

Perl script to compare two consecutive lines for the first character and ignore the second line if both first characters are same

Can anyone help how to compare two consecutive lines from a text file for the first character and if both the first characters are same ignore/delete the second line and print into new file only the ...
0
votes
0answers
16 views

Adapt text lines to fixed length [duplicate]

With the shell command fmt -w 30 myfile it is possible to truncate the myfile lines so that their length is at most 30 characters. The result is however not a justified file: some lines will be ...
-1
votes
2answers
51 views

Substitute a string containing && with sed [closed]

I've wrote some script for automated technical acceptance tests. Since we're not allowed to write password in clear text, the script I used is associated with many variables, one of which is the ...
2
votes
4answers
187 views

Need to print the matching lines in different file

Eg: I have data like this 1111111 test1 test2 test3 1111111 block 1 block 2 block 3 1111111 page1 page2 page3 I would like to print the content as different files as bellow. Test1 - File 1111111 ...
0
votes
3answers
70 views

Search for string(s) in files and report four lines of context (match and three following)

I have a string (or a list of strings) that is/are to be searched in all the files of a directory.  For example, if I am searching for ABC, and file1.txt contains ABC on line 6, then I want lines 6, ...
1
vote
2answers
101 views

Remove lines from CSV file if the second column is in a word list

I have found a long list of free email providers that I want to remove from my email lists - https://gist.github.com/tbrianjones/5992856 Below are two commands I currently use that do the same job ...
1
vote
2answers
69 views

How to remove characters of text in one line before specific character

How to remove a characters of text in one line before specific character I will remove characters to the left of the first letter "F" Example: null###F01|54646466|00K4234001|IFD|1990101 ...
1
vote
1answer
33 views

Helping with sed - remove specific lines with certain string

These are the contents of the text file: cat dog elephant cat lion cat tiger penguin penguin emperor zebra I want to remove the line contain just cat and retain cat lion and cat tiger lines. There ...
0
votes
1answer
35 views

Replace pattern with line from file using awk/sed [duplicate]

I have a file first.html which contains the following code: <tr> <td class="headerValue">One</td> </tr> <tr> <td class="headerValue">Two</td> </tr> ...
2
votes
4answers
260 views

Replace regex match with string containing match

I have a file with text as Afghanistan=+93 Albania=+355 Algeria=+213 American Samoa=+1 Andorra=+376 Angola=+244 It has all the country list and its dialing code. I want to replace: ...
1
vote
1answer
36 views

Split column into separate columns

I want to split the column1 at dot . into two columns and rest of the columns remain same. Input file: chr21.9411551 7 13 53.8461538461538 7 29 24.1379310344828 Expected output file: chr21 ...
2
votes
1answer
52 views

combine two files to single file with combined columns

I am new to linux/unix shell scripting and struggling to combine two files into a single file with all columns from both files. I am providing my example files. File 1 chr loc T1 C1 chr1 100 2 ...
2
votes
4answers
56 views

CSV - Only keep certain entries

I have a bunch of data inside "" and separated by , - Example below of two lines "stampthisandthat.com","GANDI ...
5
votes
3answers
133 views

How to print a line if next line does not contain a particular string using nawk or sed

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 ...
0
votes
3answers
33 views

How to print empty spaces in first column using awk or sed

Here is my sample input file. ip.txt: john math science paul math science rosy math jill science rob math science hary math Desired output: john paul ...
2
votes
5answers
282 views

Duplicate lines and make them unique?

I have the following structure of text lines: 3923 001 L05 LV,L05 RM 3923 002 L12 RA,L12 LA 3923 003 I06 ALL 3923 004 G04 RV,Z09 ALL but i would need this: 3923 001 L05 LV 3923 001 ...
1
vote
3answers
66 views

Random sampling and outputting the largest value

I have a fairly large data set ~500 million rows. The data set looks like below. Col 1 is float number, col 2 is mac id(device id) 1616.93,ac:22:0b:a6:22:c3 2872.32,c0:bd:d1:36:bb:49 ...
1
vote
2answers
50 views

Matching “keyword value” pairs from semi-structured input

On occasion I get output from a command that includes "key value" pairs, with possibly more than one pair per line. As a repeatable example, consider the command ip addr show dev eth0: ip addr show ...
0
votes
2answers
25 views

Replace fields in file inline

I have file (a.txt) with following contents: Redhat master 12345678 V1A 20.0/20.0.1 Windows slave 12345678 V1B 19.0/19.0.1 I want to grep line with word master and replace 4th and fifth field with ...
1
vote
1answer
32 views

Group by Functionality on Mac Address

I'm looking for a solution that should do grouping of identical mac address(device ids). I tried using sort -k 2, but I'm not confident of the result. Data Set looks like: 1616.93,ac:22:0b:a6:22:c3 ...
0
votes
2answers
52 views

Merging large data set in fastest possible way

I have a large data set ~100 GBs having smaller csv files ~ 100,000. I wish to merge all .csv files together in the fastest possible way. Files are located in two different folders. Someone ...
1
vote
1answer
32 views

bash ignorecase match pattern in file and color that line, print all to screen

I have almost zero idea about coding or bash scripting, so please bare with me and excuse my horrible formatting. What I want is to cat a log file and highlight certain lines with color based on a ...
11
votes
9answers
2k views

How can I delete the 5th word of every line in a file?

I want to delete the 5th word of each line in a file. The current content of the file: File is not updated or and will be removed System will shut down f within 10 seconds Please save your work ...
3
votes
1answer
36 views

Store specific output to a shell variable

I am looking to store the output of the svn info command, more precisely the revision number into a variable for further processing. I just need the 4 stored. Can anyone help me please? $ svn info ...
0
votes
3answers
41 views

Shell append string after last match

I would like to insert a string after the last match, example: String to insert after last instance of banana: Orange Input Apple Banana Apple Banana Output Apple Banana Apple Banana Orange I ...
1
vote
3answers
69 views

Bash convert string to array of strings?

I've got a string like this: 8080 "ac ac df asd" 9019 "f v adfs" 1 "123 da 123x" Is there a clever way to convert this into arrays like this using Bash? 8080 "ac ac df asd" 9019 "f v adfs" 1 "123 ...
0
votes
0answers
21 views

Output a particular character if the same column consists of a few defined characters with AWK

Let say I have a text file as below: G W G A A T G A A T R A T A R A G - G A And I already know that if the same column consist of "A", "G" and "T", I would like to output a "D". For column 2, ...
1
vote
2answers
68 views

How to extract 1st, 3rd, 5th of group of three lines by sed or awk?

Say I have a file: Line1 Line2 Line3 Line4 Line5 Line6 Line7 Line8 Line9 Line10 Line11 Line12 … I want to get: Line1 Line2 Line3 Line7 Line8 Line9 Line13 Line14 Line15 It means that I want ...
2
votes
4answers
59 views

How to parse table for pattern if pattern includes spaces

A continuation of this question: parse first column of command output, get corresponding second column value Say I have a command that outputs a string formatted as a table, as shown below. What if ...
0
votes
1answer
47 views

parse first column of command output, get corresponding second column value

I am fairly new to BASH scripting for Linux. I am executing a command that outputs a table, as such: firstValue: a secondValue: b.y7 thirdValue: c-x-g 10:42 GMT Say I need the secondValue. I ...
3
votes
6answers
334 views

Grep a file on specific field

I have two files, let's say File1: Locus_1 Locus_2 Locus_3 File2: 3 3 Locus_1 Locus_40 etc_849 3 2 Locus_2 Locus_94 * 2 2 Locus_6 Locus_1 * 2 3 Locus_3,Locus_4 ...
0
votes
1answer
32 views

How to make PDF viewer responsive to changes in Debian

I have a fresh Debian 8.1 installation and am trying to make its usability feel like OSX. I run xelatex document.tex && evince document.pdf & but it only successfully runs xelatex and ...
1
vote
2answers
38 views

Copy text only contained within quotes “” from a file

/etc/opendkim/keys/"$(hostname)"/default.txt contains the following default._domainkey IN TXT "v=DKIM1; r=postmaster; g=*; k=rsa; ...
7
votes
6answers
584 views

How do I display the nth result of an ls command?

I used this command to display the first result of files in my directory. ls | head -n 1 My simple question is, how can I modify this command to display say the nth result? Thanks!
2
votes
1answer
43 views

How to read and do a min/max/avg from a NMEA stream

I have a weather station attached to my device spitting NMEA 0183 on /tty/USB0 I have alreeady sorted out how to parse the data and echo what I want in a script. What I need to do now is write the ...
1
vote
4answers
71 views

optimize command with or and pipe to parse the output of ifconfig

I have this command line, but it's doubling the grep and awk ifconfig eth1 2> /dev/null | grep "inet " | awk '{gsub("addr:","",$2); print $2 }' || ifconfig eth0 2> /dev/null | grep "inet " | ...
1
vote
3answers
45 views

compare multiple files(more than two) with two different columns

I have four different files: file1, file2, file3, file4. Each file has 2 different columns separated by tab. I want to match first column of file1 (as reference) with the first column of second file, ...
1
vote
1answer
58 views

Pick columns from a variable length csv file

I have two CSV files, which share a column that is unique to each row in each file, like an ID. The files don't have a header. The file_2 has variable length columns e.g. 1,2,3 4,5,6,7,8 I joined ...