Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

3
votes
1answer
121 views

how to use awk to do subtraction with numbers in a large file

I have a large file like this AT5G44030,TAACARA,1000,+,200,206 AT5G44030,TAACARA,1000,+,418,424 AT5G44030,TAACARA,1000,+,773,779 AT5G44030,NGATT,1000,+,114,118 AT5G44030,NGATT,1000,+,267,271 ...
1
vote
3answers
61 views

sed delete all besides first and last line of many files

I'd like to process many *.txt files in a directory - they have a common structure (; separated, common headers) but vary in the number of lines each is comprised of i.e. some are just a single line, ...
0
votes
3answers
44 views

Command to find a string inside a pattern

Contents of log file are Caused by: com.ofss.fc.framework.exception.BusinessException: The memo start date cannot be earlier than the process date. at ...
0
votes
2answers
79 views

Curl and Grep to Parsing Content

I need to parse the output of a curl command: curl --user test:test http://192.168.1.1/security/pkm.html | egrep '@company|config.pkm.password' This returns: <input type="text" ...
-1
votes
2answers
29 views

Replace string if it has a common pattern

I have a file like the following: 0/1:34:89:09 0/0:985:083:67 0/1:687:90887:975 0/1:54:79:25 0/0:5:83:67 0/1:67:9087:97 0/1:574:779:225 0/0:275:73:627 0/1:67:47:77 I want to replace any ...
0
votes
1answer
33 views

Copy process output few lines to a file

I want to copy the xinput-calibrator output to the temp.txt file from my Qt application. I am starting the QProcess from the Application Using the command xinput_calibrator | tee log.txt, I am ...
0
votes
1answer
28 views

get unique values from 1 column and sum up 2 next columns [closed]

I have a text file that looks like this: A 500 100 ABC B 200 100 BBB B 200 100 ACC D 150 235 CDD A 500 100 RER C 210 653 DDD C 210 653 DWR D 150 653 DDW D 150 653 DER What I try to accomplish is to ...
2
votes
4answers
52 views

Not enough disk space: remove first n lines of text file without copying the file [duplicate]

I want to remove the first $rmv lines of a huge text file called $filename. This text file is so big that I cannot fit two copies of it on my hard drive. The following leaves me with a blank file ...
0
votes
1answer
28 views

How to implement config file expiration time?

I have a Debian server which runs a service using multiple config files (in a similar way to apache2) which I need to add an expiration date to. The config file that has an expiration date starts with ...
-2
votes
1answer
34 views

Replace character X with character Y in bash string

I am making bash script and I want to replace one character with another character in my string variable. Example: #!/bin/sh string="a,b,c,d,e" And I want to replace , with \n. output: ...
0
votes
2answers
53 views

Problem with grep -o

I have a text file as shown below. 192.168.10.1 192.168.45.45 192.168.255.255 192.168.0.5 Random text comes here 8.8.8.8 224.23.25.12 225.1.2.3 226.0.0.0 ...
1
vote
1answer
33 views

Split text file in random halves based on category

I have a text file that looks like this: n03250847/n03250847_0.JPEG n03250847 n03250847/n03250847_1.JPEG n03250847 ... n03250847/n03250847_499.JPEG n03250847 ... n03255030/n03255030_0.JPEG n03255030 ...
1
vote
2answers
51 views

Count times an expression is true in awk.

In unix/awk code Essentially, I need to iterate through the file (data1.txt) and count the number of times the substr($0,29,2) of each line of the file == "04". data1.txt: 6597243042 ...
0
votes
2answers
60 views

format command output with variable and fixed space

How can I achieve this best maintaining the hierarchy on the left and aligning the size value on the right? Also, at the same time I'd like to round the size to one or two digits. I tried with printf ...
2
votes
1answer
43 views

find and replace a string in a file without using temp file with SED

I have a text file called sample.text which looks like this Continent Lat/Lon: 46.07305 / -100.546 Country Lat/Lon: 38 / -98 City Lat/Lon:(37.3845) / (-122.0881) I want to process this text to get ...
2
votes
1answer
27 views

Insert string or line after last instance of a specific search variable, in a loop

I'm wondering if someone could help me with a specific coding question. I have a DNA sequencing file that reads something like this (as an example): ...
1
vote
2answers
64 views

Bash script - chomp parts out of a line

I have a data file (see sample three lines below) separated by pipes. What I want to be able to do is to "chomp" out the fields and put them into variables. They are of pre-defined lengths. I want to ...
3
votes
1answer
41 views

comparing values in separate columns

I've got a csv that looks like this, (there are other columns but these are the pertinent ones) region, city, actual rating, predicted rating NSW, SYD, 0.765, 0.5 Victoria, Melbourne, 4.81225, 5.0 ...
0
votes
1answer
33 views

extract the data from 2 files

I have two files, the first file contain only 1 field, the second file has more lines than first file and also it has different field in each line. File 1: 5 15 20 File 2: 3 5 O 1.25 2.30 3.75 ...
0
votes
1answer
42 views

Removing new line characters around patterns

I have a fasta file of millions of paired sequences, that looks like this: >7001289F:56:HKH3FBCXX:2:1101:1692:2074 1:N:0:CGATGT ...
0
votes
2answers
62 views

How to get only certain fields from a given file?

I have list of employees in following format Name: Phone No: Email: Name: Address: Phone No: Email: Name: Country: Address: Phone No: Email: Name: Address: Name : Email: Address: All I want ...
4
votes
3answers
100 views

Insert text at specific line number

I'm working on a bash script that will split the contents of a text document depending on the data in the line. If the contents of the original file were along the lines of 01 line 01 line 02 line ...
3
votes
2answers
54 views

sed pattern and multiline substitution

I need to parse a text file and replace a certain portion of it. I believe sed is the way to do it but not sure how to handle multiple lines with it The pattern I am looking is set cells { \ cell1 ...
1
vote
4answers
102 views

Bash: How to print additional column matching IP Address to Name

I'm using netstat to collect information about connected IP's. And I have the following format: netstat -tn 2>/dev/null | awk '/:80 / {print $5}' | sed 's/.*::ffff://' | sed 's/:.*//' | sor t | ...
3
votes
2answers
46 views

Sorting by two columns

I need help with this homework assignment: Write a shell script named sales1 using nawk and sort commands to do the following: 1. print the file in two sections, east and west, with lines alphabetized ...
1
vote
1answer
64 views

What does :&lt;i&gt; code stand for in subtitles?

I downloaded a subtitle from subscene http://subscene.com/subtitles/the-5th-wave/english/1298314 and it contains some coding such as :&lt;i&gt; and &lt;/i&gt; which is not being ...
1
vote
1answer
52 views

Extract random sample of N lines based on pattern

I have a file formatted like this: train/t/temple/east_asia/00000025.jpg 94 train/t/temple/east_asia/00000865.jpg 94 ... train/s/swamp/00000560.jpg 92 train/s/swamp/00000935.jpg 92 .... ...
2
votes
2answers
26 views

How to append a string to the first column of a matrix file?

I have a matrix file as follows. The first column is a name list. I want to add _i1 to the end of each name. A1A A1B A1C A1D A1E TR6764_c0_g2 0.00 0.02 0.00 0.00 0.00 ...
3
votes
3answers
69 views

Replace strings in a file based on a list of strings and a list of corresponding replacements

I am trying to replace strings in a file A: Hello Peter, how is your dad? where is mom? where the strings to be replaced are in file B: Peter dad mom and their corresponding replacements are in ...
0
votes
0answers
12 views

Linking and combining text from different documents into dynamic document

I am taking notes on many, related topics from multiple sources. For instance, one of the sources is a textbook on machine learning. I type the notes in an .odt document (but whatever, it could be ...
3
votes
2answers
49 views

On applying commands to groups of lines from stdin

The Unix package datamash supports the application of several summarizing operations to groups of input lines. For example1, here datamash is used to compute the sums of column 2 for each value in ...
1
vote
1answer
35 views

Removing columns from a file

I'm trying to remove columns 75 through 85. The colrm command works, but the problem is that it stops whenever it finds an extended ASCII character in the file. So, my question is this: is there ...
1
vote
1answer
41 views

Compare independent lines in two files [closed]

I have two files, each with 500k+ user accounts. I need to find the differences between the two (the missing accounts/delta). Each file is in a simple format, each line contains an email address in ...
3
votes
1answer
40 views

Get contents of a file from first instance to last instance of some arbitrary string

How can I view in less from the first instance of some arbitrary string "foo" to the last instance? This arbitrary string will be on most every line of the log. I don't want to do grep "foo" bar.log ...
1
vote
1answer
35 views

Sed delete line if nth character is not specific character

I have a csv file, and I want to delete lines if their 12th character is not ;. So, for example, my file looks like this: 2266308;A;B;dfsgsfdg 2266309;A;BJHSADFK;gfsdg 2266310;A;B;dfg and I ...
2
votes
2answers
66 views

Split File into 10 files with unique item in each file and maximum line number limit in each file

I have a huge file (more than 2M records in it). Here is my requirement: Initially, break the one big file up in to 10 smaller files. The files should be formatted as follows: ...
1
vote
1answer
36 views

How to set csplit output filenames by pattern?

Is there a way to set output names of csplit through the pattern? I want to split the ssh-config by hosts to corresponding files. If there is a solution without csplit I'm happy as well.
0
votes
2answers
43 views

Bash Script with Multiple Lines and Grep

I have output from a specific command that can provide the number of times an application has been marked as 'used' on an OS X system. The command is mdls. The output looks similar to the following ...
0
votes
1answer
42 views

subset a file just on one column

I have a tab delim file, I want to make a subset on the basis of column 2, choosing "A". 435 A 2 27.92464882 8.63E-07 1 752721 564 F 2 30.12532071 2.87E-07 1 798959 654 A 2 ...
0
votes
1answer
30 views

sed find line in file and remove characters

I have a file that contains lines like this proto udp . . . .. server 10.10.8.0/24 255.255.255.0 I need a way to remove only /24 from this line server 10.10.8.0/24 255.255.255.0 to become server ...
0
votes
2answers
54 views

How do I find a range of numbers in the content of file and move files containing the range of numbers to another directory?

Okay so I have about 90 files, each with a body like 187942 78.9 I want to filter files that has the values between 70 to 90 on the second column and move them to another directory. I'm ...
-2
votes
3answers
83 views

Extract sequences from a fasta file

I have a fasta file (not in right format) that contains hundreds of thousands of different lengths of DNA sequences like this: ...
3
votes
2answers
56 views

match columns and replace

I am newbie in linux. I have two very large text files with space-delimited fields: File1 527858 51 2 27.92464882 8.63E-07 570289 82 2 30.12532071 2.87E-07 571034 90 2 ...
1
vote
2answers
47 views

How to add strings from a name list file to another file?

I have a fasta file (seq.fa) which is a standard file format for genetic info, like so: >TR1|c0_g1_i1 GTCGAGCATGGTCTTGGTCATCTTCCTTTCAAAGAA >TR6|c0_g1_i1 GTGGAATATCGCCAGTGACCATCACTGATTAACCTG I ...
1
vote
3answers
73 views

How to do following formatting using sed or awk or bash script on linux

Suppose, I have two files: file1 having below contents. message_aaa.wav test 1 of 1 test Label : aaa test Sentence : test aaa test String: test message Interpretation ...
1
vote
2answers
51 views

AWK: How to put field ($1) inside regular expression to select all prefixes?

Assume I have a file A: fileA fileB Suppose I have now a file named: fileA_someprefix_20160101.txt Now I want to match all lines from A which prefix this filename, so I thought: ...
0
votes
2answers
52 views

Transforming a list of files separated by new lines into a variable

Goal: I'm trying to find all instances of *.clj or *.cljs files recursively within a directory, store them in a string variable (separated by new lines) and then transform them. So if the following ...
-1
votes
1answer
22 views

Sorting and outputting in another file

How can I sort a file then output it into another file I never created or do I have to create the file I want to put in. Might not make sense how I am typing it so... Let's say the file I have is ...
4
votes
3answers
85 views

String replacement using a dictionary

What is a good way to do string replacements in a file using a dictionary with a lot of substituend-substituent pairs? And by a lot, I actually mean about 20 – not much, but many enough so that I want ...
0
votes
3answers
79 views

What is the most concise script to list the missing ones in a digit sequence? [duplicate]

For example, we have the content 001 002 004 008 010 in a text file named file, how to extract the missing 3 5 6 7 9?