Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

Filter by
Sorted by
Tagged with
5
votes
2answers
62 views

Column deletion based on number of string matches within column

I need a command that will remove any columns in a text file if they have =>${MaxAllowedNumberOfFs} 'F's within the column (a column that will have a varying number of rows). I have some pseudo code ...
18
votes
18answers
11k views

How to read an IP address backwards?

If we have this string (IP address): 192.168.1.1 How can I derive the (DNS reverse record form) from this string, so it will be shown like 1.1.168.192.in-addr.arpa using a shell script?
1
vote
0answers
27 views

Histogram with FASTA file

I am new to Linux. I have a FASTA file that looks something like this: >scaffold1 AAGACATAATATTTTGGAGGAATTAAAAAATTTAAGATGTATTTTATTATACATGTATTTTATTTATAACATAAATAAACATCCCAAGGAAAAGCAGTAGCT >...
1
vote
3answers
53 views

How to extract domain name from email address and add it to a new column in a pipe delimited file?

I have a pipe-delimited file with an email address in the third column. I need help to create the fourth column out of the domain name from the email address in the third column using awk or sed ...
0
votes
5answers
52 views

Remove n character after specific word

MY Data: 7468656.xml-87- 7468656.xml-88- 7468656.xml-89- 7468656.xml-90- 7468656.xml-91- 7468656.xml-92- 7468656.xml-93- 7468656.xml:94: 7468656.xml-115- 7468656.xml-116- 7468656.xml-117- 7468656.xml-...
1
vote
1answer
19 views

How to search and replace for multiple words in one shot

I need to search and replace multiple words. For example, in my file, I want to search for any name of week days that appears with a comma after it: Sat, or Sun, or Mon, or Tue, or Wed, or Thur, or ...
2
votes
4answers
85 views

More elegant approach to append text to a file only if the String doesn't exist?

I want to append NoDisplay=true to a .desktop file, but only if the entry doesn't exist. I do this the following way: grep -q 'NoDisplay=true' '/usr/share/applications/yelp.desktop' || bash -c 'echo "...
0
votes
5answers
40 views

add specific string at the beginning and at the end of line 1

The input file looks like this H1 O1 C1 H2 H3 C2 H4 H5 O2 C3 H6 H7 C4 H8 H9 O3 C5 H10 H11 C6 H12 ...... I would like to have the following output file z(A) H1 O1 C1 H2 H3 C2 H4 H5 O2 C3 H6 H7 C4 H8 ...
-1
votes
4answers
50 views

Copying first line of text to another text file

Before: file1 123 456 file2 foo testing After: file1 123 456 file2 foo testing 123 Any help would be greatly appreciated.
0
votes
1answer
66 views

Sed to replace string of 10 characters with 'XXXX '

I have file which looks like ACCINTCCADVANCESC ACCINTCCADVANCESE ACCINTCCADVANCESW I want to replace INTCCADVAN with XXXX␣␣␣␣␣␣, the ␣s are spaces. Specifically I want to replace a column with more ...
1
vote
3answers
75 views

Partition a file based on one column

I have the following data set: M1 1 1233 M2 1 3212 M3 1 55323 M4 1 4444233 M5 1 23444 M6 1 555333 M7 1 55567 M8 2 22224 M9 2 55566 M10 2 4567 M11 3 44242 M12 3 234234 ...
0
votes
2answers
30 views

multiply specific column in a file which consists of thousands columns

The in file looks like this -17.3644 0.00000000 0.00000000 .... -17.2703 0.00000000 0.00000000 .... -17.1761 0.00000000 0.00000000 .... -16.5173 0.00000000 0....
2
votes
1answer
84 views

A utility to append content into a file, if this content isn't already in the file (even in part)

Is there a *Nix utility by which I could append content to a file (say, append the content of a template file into the end of a second file), only if the content isn't already available in the second ...
2
votes
2answers
63 views

Command for converting rows to CSV file

I have a file of the format, with a leading space before each line: "Western Overseas", "Western Overseas", "^", "--", "^", "--", "--", null, 24995, 9977, "CR", "Western Refrigeration ...
0
votes
2answers
29 views

Split delimited string using variable substitution

I have a sample file the lines of which are delimited by "#" and it contains 3 fields. A#B#C D#E#F Can I use UNIX variable substitution instead of the following to get values of each field in the ...
2
votes
2answers
38 views

Split into multiple files by using unique columns in a unix file

I have requirement to split below file (sample.csv) into multiple files by using the unique columns (first 3 are unique columns) sample.csv 123|22|56789|ABCDEF|12AB34|2019-07-10|2019-07-10|443.3400|...
0
votes
2answers
69 views

sed command or awk commands to remove text

I have this input file: target_id length eff_length est_counts tpm ENST00000583162.1 1066 967 1.69899 1.18376 ENST00000583355.1 891 792 13.8057 11.7445 ...
0
votes
1answer
49 views

Filtering a .txt file using a regular expression and a second file

I have a file.txt (it does not have the same number of columns for each row): e.g. 1 21:10 21:23 2 1:94 1:100 1:123 3 14:1 14:60 14:23 I have another file (file2.txt) that contains 4 columns (...
2
votes
2answers
35 views

Changing the date format within a CSV file

I have a program that outputs data in the format date, time,field1,field2,field3,fieldn 12/20/14,08:01:53,318.622,0.93712,21.6867,1.1089 the file has many columns which all need to stay the same ...
0
votes
5answers
73 views

Replace substring of characters with awk and sed

I have a file which contains a very long string of characters and I would like to replace a substring of it with Ns. Example: test ABCDABCDABCD I would like to replace a substring of it with all ...
41
votes
3answers
88k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same files in Jedit, it warns me of invalid UTF-8 encoding... The HTML meta tag states "charset=ISO-8859-...
0
votes
3answers
56 views

Swapping words with sed

I need help with sed, actually with Linux regular expressions: I need to change two parameters between parentheses, for instance, if the line is mysqli_select_db($par1, $par2)<br/> I need to ...
0
votes
3answers
33 views

Replace and increment a line in bash

How can I read, increment and replace a value in a file? foo="val" ver="1.2.0001" ... Now I would like to increment the "0001" to 0002".
2
votes
2answers
117 views

Why does my script create an extra character?

I know this isn't a good thing to do and/or a good solution but I was just messing around trying to answer this question and I came across the following problem. The goal is to read the input file ...
0
votes
2answers
50 views

Replace character based on range of coordinates

Hi I have a text file with thousands of lines which looks like this: NNNNN NNNNN NNNNN NNNNN I would like to substitute based on their location coordinate all the letters within a specific range to ...
0
votes
2answers
33 views

Replacing all characters in text file with specific character

I have a file which looks like: file.txt NNNP PPZL ZZNN I would like to replace all the letters to "N" nd have an output like this with the same number of lines as the input: (my file here has 3 ...
0
votes
1answer
33 views

Shell script question for getting 2 output text file in a single file [closed]

I have a output in one file and another output in another file. I need output1 and ouput 2 in a single line and in a single file. What command i should use?
0
votes
4answers
50 views

How to copy a value between parenthesis into another part of the same line

I have a CSV file with many lines like these: 1003,CC,L1008,,(LB1) Urbà de Barberà del Vallès,3,,, 1006,CK,L0520,,Móra d'Ebre - Móra la Nova (estació),3,,, 1007,AV,L0358,,Granollers-Terrassa-Manresa,...
1
vote
2answers
34 views

extract starting and ending with a special character

I have a comma separated file which is not a standard csv file like this: XYZ,143,ABC/genomes/date/pa341,dog,available 567,FTR/genomes/date/ha76870,horse,waiting 214,GEN/genomes/date/btr256,N/A,...
1
vote
1answer
37 views

How to read a file with two columns into a script so that each line read as two variables

I have a file that has two columns like such. cat test.txt 100 2007 FFF 1993 7E7 1994 4BB 1995 I need to input each of the lines into a script, and have the two columns assigned as separate ...
0
votes
5answers
55 views

Putting a command inside awk to merge columns of different files

I have a file.txt for example (it does not have the same number of columns for each row): 1 2 3 4 5 5 6 7 7 7 7 9 10 I have another file (file2.txt) that contains 2 columns a b c d e f ...
0
votes
3answers
55 views

how to select columns in a file by the column name in a txt file in unix

I have a big file with ~ 1000 columns. The first 2 columns are chromosome and SNP (single nucleotide polymorphism)position, the rest are samples. Here is the first few line and rows: #CHROM POS ...
2
votes
2answers
1k views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
2
votes
2answers
34 views

append names of multiple .txt files and rename concatenation destination file

I am new to bash shell scripting, apologies if this was already asked. I have combinations of multiple files such as: USA.txt Florida.txt Miami.txt I would like to join those files and create a new ...
0
votes
3answers
117 views

Invert the last byte of a file?

I'd like to test my hashing program by inverting a single byte at the end of a file without modifying the timestamp. Keeping the timestamp from changing is easy enough, but how do I use a unix utility ...
0
votes
2answers
29 views

get all the lines from second file matching with values of a specific column in the first file

I have two files: first file: ab34,ax,55,gt ab46,56,thy,ix ab34,tg,47,xy at87,kl,89,th ab46,ut,87,er second file: x1,ab34,78,got wq,at87,100,op 90,ab46,la,gf expected result: ab34,ax,55,gt,x1,...
-1
votes
3answers
63 views

insert current date in a place of empty column (field) of a file

I have a file with four columns delimited by "|" as below 23414|test doc for me|| 32322|test doc for you|2004_12_03| 13454|test doc for all||2010_11_01 I am trying to insert the current system date ...
1
vote
4answers
634 views

How do I grep file paths out of a text file?

I have a file that has a list of paths like so: "1" "/user/bin/share" "2" "/home/user/.local" "3" "/root/" Is there a way to extract just the paths? I dont want the numbers or ...
-2
votes
2answers
42 views

Perl::Find a substring and display a whole word

I have a text file that has some data like: PADHOGOA1 IOP055_VINREG5_1 ( .IO(VINREG5_1), .MONI(), .MON_D(px_IOP055_VINREG5_1_MON_D), .R0T(px_IOP054_VINREG5_0_R0T), .IO1() ); PADV30MA0 ...
1
vote
1answer
28 views

Search through line for a string and print that line only once to another file

I have a text file that has unique lines by timestamps. This below is the example of the lines in that text file: 111.111.11.11: Jun 18 08:01:34 2017: abc\john.doe1 Windows... 111.111.11.11: Jun 18 ...
133
votes
9answers
280k views

How do I trim leading and trailing whitespace from each line of some output?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into? Example file: test space at back test ...
3
votes
1answer
1k views

Modify filenames in Makefile

For reasons that are not important here, I have a source code that I process automatically file-by-file and the processed source files are renamed in a systematic manner. For example, I start out with ...
22
votes
1answer
47k views

Reuse rsync --dry-run output to speed up the actual transfer later on

I usually do rsync -n before doing an actual rsync. The thing is, how could I automate the second run to reuse exactly the list generated with --dry-run? I mean, if I run rsync -n then I get a ...
9
votes
6answers
7k views

Tool to create text files from a template

I have to regularly create 100+ text files from templates. I currently use an overly complicated shell script. I think there is a smarter way to handle this, but I don't know how. I have a "database" ...
2
votes
2answers
173 views

Using Sed to add counter to keyword

Suppose I have a file like this. Keyword "name" aaa bbb ccc ddd Keyword "another name" eee fff ggg hhh iii and so on. I want to change the keyword lines in the file to have counters starting from ...
1
vote
3answers
104 views

Perl: Inserting at nth position in file

I have content in my file : 123456789 My output is: 1234CC789 Desired Output is: 1234CC56789 I am insert PP at nth position suppose. But when I am inserting any text it is inserting the text at ...
0
votes
1answer
16 views

How to insert comment at specific variable line number by sed command? [duplicate]

I want to comment out a line (add a # to the beginning). I used this command: sed -i '96s/^/#&/' file.txt on Linux to insert comment at line # 96. If the line number is a variable, then how can I ...
3
votes
1answer
7k views

Proper way to convert PDF to word from bash command-line

I need to convert 1K pdf files to doc on a debian server. I can convert a PDF to word using libreoffice commandline: libreoffice --headless --invisible --convert-to doc Sample-doc-file-100kb.pdf Or ...
1
vote
2answers
29 views

how to add new column with constant value in a text file

I have a large text file with several columns and want to add a new column (no matter where) with constant value of 10000 please help
0
votes
0answers
31 views

removing duplicates [duplicate]

I have a large text file with many rows but some of the rows have duplicates, however only in the first column: A B A C E F E G I want to remove the duplicate lines and it doesn't matter which ...