Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

0
votes
0answers
3 views

Get the filename that the record come from

I have a huge number of .gz files, which being uncompressed using zcat together. I am looking to get from where each record came from, as below example: File1: 1,2,3,4,5,6,7,NEW,9,10,11,12 ...
1
vote
1answer
19 views

Form a output by combining multiple lines

I have a log file as below: /export/home/got/logs/o2877612.01:job_1432787863184_159408 Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: ...
1
vote
1answer
42 views

Using portion of a field in awk

How to use portion of a field for awk script? The input file: 123456789,0,0,4,1,6,8,6 987654321,0,0,8,4,6,9,8 251335549,8,9,6,3,1,8,8 897465419,8,1,8,9,4,7,9 I need to select the 3rd character ...
1
vote
3answers
23 views

Using awk to merge 2 files

I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working. awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ...
1
vote
1answer
25 views

How to add hostname, date, and fix disk size

I'm collecting disk usage data and want to add the date, time, and hostname to this command. How can I do it in this format: YYYY-MM-DD HH:MM:SS,HOSTNAME,DISK_USAGE_DATA Command I have so far: ...
1
vote
2answers
38 views

Extract columns of a string using grep

I need to extract from /proc/net/dev the second and tenth columns separated by spaces. In other words, the number of received and transmitted bytes. Is it possible to do this using grep and ...
3
votes
4answers
98 views

Concatenate table without header

I would like to concatenate several tables and just keeping the header in the first line (and thus not having a new header each time that a new table has been concatenate). Does anyone know how to do ...
13
votes
7answers
2k views

How to delete every second line from a file?

File: Data inserted into table. Total count 13 No error occurred Data inserted into table. Total count 45 No error occurred Data inserted into table. Total count 14 No error occurred Data inserted ...
0
votes
0answers
20 views

problem in extracting fasta sequence based on header name [on hold]

I'm facing a crazy problem during extracting fasta sequence based on their header name. I have a fasta sequence file and a text file containing their header name. I tried to extract sequence of ...
-1
votes
1answer
39 views

How to add a word at the first of every lines? [duplicate]

File1: Data will be inserted on Thursday Messages has been sent at 6pm Office hours is from 9am to 6pm All the employees are present Output what I want: DMS Data will be inserted on ...
2
votes
1answer
65 views

Show nth occurrence of a record

Using awk, how to show the nth occurrence of a record and delete the other occurrence, for example" Input File: 79933,2 79933,4 79933,5 55251,1 49494,4 49494,2 49494,2 49494,1 in this example, ...
2
votes
5answers
137 views

How to sort file by character occurrences per line?

I'm quite new to Linux, and I've found quite a bit of useful information on how to do character counts in a file, but is there a way in Linux/terminal to sort a text file by the number of times a ...
1
vote
5answers
335 views

Check number of words in a file that contain a specific letter

Bash command to check number of words in a file that contain letter “a”
1
vote
2answers
29 views

Append mth and nth columns of a file with the columns of another file

I have 2 files having different number of columns and with huge number of rows: bash-3.00$ cat fileA a1,a2,a3,a4,a5,a6,a7,a8,a9 q,w,,,,y,u,, a,z,,,,q,n,, ......................... z,p,,,,w,e,, ...
2
votes
3answers
142 views

Remove the ^L (aka \f/FF/form-feed/page-break) character

How do I remove the ^L character and the blank lines that come after it on a unix file? I have tried the below and have been able to remove the VT and spaces but am failing to remove the ^L character ...
3
votes
3answers
146 views

If statement string comparison

I'm reading a csv file with 14 values per line. I want to convert the HEX value from 10th [f10] value to a DEC value. In order to do this if the 7th [f7] value must hold EnHr or EnSt or SpJb or Chem ...
1
vote
3answers
45 views

Adding a pattern to each field of the following line

I need some help here please. I have the following input: MX04A; DMX04A; DMX04A; LMX04A; LMX04A; -17.2; -15.3; -14.3; -13.6; -16.8; -15.4; -16.0; -15.3; LH36A; DLH36A; DLH36A; -11; -117.2; -11; ...
2
votes
5answers
70 views

bash expanding to same value instead of line by line value in sed command

I have a file containing numbers in it say numbers.txt as follows. 9374541632553 51243747879841 32030098896914 84654557358238 11000656847765 I want to convert it to the following format. ...
0
votes
0answers
32 views

Drawing a histogram from a bash command output

I have the following output from a miRNA Predictor: Read Sequence:hsa-miR-576-3p MIMAT0004796 (22 nt) Read Sequence:gi|182396|gb|M74088.1|HUMFAPAPC Human APC gene mRNA, complete cds (8972 nt) ...
3
votes
3answers
207 views

awk comparison using arrays

I have the following file: 6180,6180,0,1,,1,0,1,1,0,0,0,0,0,0,0,0,4326,4326,,0.440000, 6553,6553,0,1,,1,0,1,1,0,0,0,0,1,0,1,0,4326,4326,,9.000000, ...
2
votes
2answers
19 views

How to replace value for a given condition in specific column of file

I have file separetd by pipe | and I want that when column 6 contain letter I print 0758000 in column 7 and when the column 6 contain letter A print 0800000 in column 7, I can not find how to do it!!! ...
4
votes
6answers
366 views

Grep the text which matches with a pattern

I was looking for a command which will print the selected portion of a line which matches the conditions. Eg, my text line is like this: p1=X||p2=Y||p3=X||p4=X||p5=X||p6=Y||p7=X I want to print ...
1
vote
2answers
37 views

Escaping double quotes in string

I have a string message which contains data in double quotes. I need to put \ in front of every double quote how can u do it in shell ? I used Perl and it works fine, but I'm looking for sh command: ...
2
votes
2answers
29 views

Shell script - text processing help

I want to create a ldif modify file to update the users on LDAP. I have an input file(alluserdns.ldif) with list of all Dn entries in ldap: dn: cn=testuser2@abc.com, ou=persons,ou=pp,dc=abc,dc=com ...
1
vote
0answers
46 views

Convert / import flat file with multiple records, one variable per line

I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or ...
2
votes
3answers
131 views

How to print the before lines of search keyword with out using grep in a gz file?

How to print the lines before a search keyword without using grep in a .gz file? ID:342N000390AAAAAAAA 07/14/15 10:26 (MV90 ) * Register Data Imported * Warning - No ...
0
votes
1answer
30 views

How to replace the content of specific column with awk? Tab Delimited File

Here's a similar question but I ran into a problem that was not addressed in the question. How to replace the content of specific column with awk? I want to do something similar, however my file is ...
1
vote
1answer
19 views

AWK: where is my error?

I have a file with :-delimited fields. If the 3rd field of a line is greater than 22, that line should be printed to file44. The file looks like this: 1:2:3:4 1:2:3:411 1:2:33:42 1:2:3:44444 ...
5
votes
4answers
137 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
1
vote
1answer
27 views

Loop over lines of file, partition file and classify each partition

In the following I've put up my code for iterating over lines of a file in order to collect data within a segment from the file in following fashion: Illustration for understanding procedure: segL ...
1
vote
4answers
135 views

UNIX to print a content from a .gz file

I have a .gz file with contents like the following: ID:123455 23-07-2015 mvni warning: RTM post warning : no profile data Using the string no profile data, I want to print the ID. I tried with ...
2
votes
3answers
60 views

Read a column from a file and append a specific column of another file

I have two files as such: fileA: Name,Site Name,Product UPEI,Product Name,NMS Flag,Product Model, SW FileB: NE_Name SW ABC 4.4 ASD 4.3 ... ... Note that ...
3
votes
4answers
107 views

Efficient data extraction from multiple files to a single CSV file

I have a large collection of XML files with the same exact structure: $ cat file_<ID>.xml ... ... ... <double>1.2342</double> <double>2.3456</double> ...
2
votes
5answers
173 views

Perl script to compare two consecutive lines for the first character and ignore the second line if both first characters are same

Can anyone help how to compare two consecutive lines from a text file for the first character and if both the first characters are same ignore/delete the second line and print into new file only the ...
0
votes
0answers
16 views

Adapt text lines to fixed length [duplicate]

With the shell command fmt -w 30 myfile it is possible to truncate the myfile lines so that their length is at most 30 characters. The result is however not a justified file: some lines will be ...
-1
votes
2answers
51 views

Substitute a string containing && with sed [closed]

I've wrote some script for automated technical acceptance tests. Since we're not allowed to write password in clear text, the script I used is associated with many variables, one of which is the ...
2
votes
4answers
192 views

Need to print the matching lines in different file

Eg: I have data like this 1111111 test1 test2 test3 1111111 block 1 block 2 block 3 1111111 page1 page2 page3 I would like to print the content as different files as bellow. Test1 - File 1111111 ...
0
votes
3answers
71 views

Search for string(s) in files and report four lines of context (match and three following)

I have a string (or a list of strings) that is/are to be searched in all the files of a directory.  For example, if I am searching for ABC, and file1.txt contains ABC on line 6, then I want lines 6, ...
1
vote
2answers
105 views

Remove lines from CSV file if the second column is in a word list

I have found a long list of free email providers that I want to remove from my email lists - https://gist.github.com/tbrianjones/5992856 Below are two commands I currently use that do the same job ...
1
vote
2answers
74 views

How to remove characters of text in one line before specific character

How to remove a characters of text in one line before specific character I will remove characters to the left of the first letter "F" Example: null###F01|54646466|00K4234001|IFD|1990101 ...
1
vote
1answer
34 views

Helping with sed - remove specific lines with certain string

These are the contents of the text file: cat dog elephant cat lion cat tiger penguin penguin emperor zebra I want to remove the line contain just cat and retain cat lion and cat tiger lines. There ...
0
votes
1answer
42 views

Replace pattern with line from file using awk/sed [duplicate]

I have a file first.html which contains the following code: <tr> <td class="headerValue">One</td> </tr> <tr> <td class="headerValue">Two</td> </tr> ...
2
votes
4answers
268 views

Replace regex match with string containing match

I have a file with text as Afghanistan=+93 Albania=+355 Algeria=+213 American Samoa=+1 Andorra=+376 Angola=+244 It has all the country list and its dialing code. I want to replace: ...
1
vote
1answer
43 views

Split column into separate columns

I want to split the column1 at dot . into two columns and rest of the columns remain same. Input file: chr21.9411551 7 13 53.8461538461538 7 29 24.1379310344828 Expected output file: chr21 ...
2
votes
1answer
55 views

combine two files to single file with combined columns

I am new to linux/unix shell scripting and struggling to combine two files into a single file with all columns from both files. I am providing my example files. File 1 chr loc T1 C1 chr1 100 2 ...
2
votes
4answers
56 views

CSV - Only keep certain entries

I have a bunch of data inside "" and separated by , - Example below of two lines "stampthisandthat.com","GANDI ...
5
votes
3answers
138 views

How to print a line if next line does not contain a particular string using nawk or sed

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 ...
0
votes
3answers
36 views

How to print empty spaces in first column using awk or sed

Here is my sample input file. ip.txt: john math science paul math science rosy math jill science rob math science hary math Desired output: john paul ...
2
votes
5answers
287 views

Duplicate lines and make them unique?

I have the following structure of text lines: 3923 001 L05 LV,L05 RM 3923 002 L12 RA,L12 LA 3923 003 I06 ALL 3923 004 G04 RV,Z09 ALL but i would need this: 3923 001 L05 LV 3923 001 ...
1
vote
3answers
66 views

Random sampling and outputting the largest value

I have a fairly large data set ~500 million rows. The data set looks like below. Col 1 is float number, col 2 is mac id(device id) 1616.93,ac:22:0b:a6:22:c3 2872.32,c0:bd:d1:36:bb:49 ...