Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

Filter by
Sorted by
Tagged with
0
votes
4answers
2k views

How to use variables in substitution with vim regular expressions

I have a big file that contains among others serveral lines of the following form: USet07-1 USet07-2 USet08-1 USet08-2 . . . USet22-2 . . . I want to remove the hyphen/dash - from these strings in ...
1
vote
1answer
1k views

grep logs between two timestamps

2021-02-10 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a 2021-02-10 08:17:50 dell AptDaemon.Worker: INFO: Finished ...
5
votes
6answers
8k views

Replace strings in a file based on a list of strings and a list of corresponding replacements

I am trying to replace strings in a file A: Hello Peter, how is your dad? where is mom? where the strings to be replaced are in file B: Peter dad mom and their corresponding replacements are in ...
4
votes
3answers
1k views

grep pattern before another pattern and print it all

Given intput: Via: 1.1.1.1 not relevant line keyword + some text ... not relevant line N keyword + some text ... not relevant line N Via: 2.2.2.2 not relevant line keyword + some text ... not ...
5
votes
8answers
804 views

Use awk to delete everything after the ","

I have a variable, var, that contains: XXXX YY ZZZZZ\n aaa,bbb,ccc All I want is aaa in the second line. I tried: out=$(echo "$var" | awk 'NR==2{sub(",.*","")}' ) but I ...
1
vote
1answer
1k views

How to remove newline character within a column in a tsv file?

I need "This contains newline" in one row, newline from the column2 and column4 need to remove. nb: For your understanding the data I upload an image if you want I will upload the text content "...
1
vote
2answers
88 views

Search and replace with AWK but in file with multiple columns and separators

I want to replace the gene names in file 1 with gene annotation information from file 2. File 1 looks like this: an OG identifier in the first column and gene names in the 2nd to the Xnd column): ...
9
votes
9answers
755 views

sed : have a range finishing with the last occurrence of a pattern (greedy range)

Take the following file : $ cat f1 stu vwx yza uvw xyz abc abc def ghi def ghi jkl ghi jkl mno jkl mno pqr mno pqr stu pqr stu vwx stu vwx yza To print all lines from the first one containing abc to ...
-1
votes
3answers
190 views

ADD or SUB numbers based on op-code (directive) in the line

How can I write an awk script that can add or subtract based on what the line in the input file is asking?  I figured out how to do either only addition or subtraction, but not either/or in one script....
-2
votes
5answers
89 views

Add single-value content of one file as additional column to another file

I have two files. One contains several columns and rows, while the other one only contains one value: file1: abc def ghi jkl file2: hello I want to add another column to file1 that always contains ...
3
votes
3answers
1k views

Format the content of the file using Perl

I have a file with the following format: >Country1 Aus trali a >Country5 Swi tzer land >Country2 Net herland s I want to output a file with the following format: >Country1 Australia &...
-2
votes
1answer
42 views

How to trim space between two delimiters - Unix

In a file we have key and value pairs separated by |, my request is to remove space in key pair as shown in Input and Output. Example: UTC Time Offset -> UTCTimeOffset I have tried different sed ...
91
votes
12answers
25k views

Linux tools to treat files as sets and perform set operations on them

Does anyone know of any linux tool specifically designed to treat files as sets and perform set operations on them? Like difference, intersection, etc?
0
votes
2answers
39 views

Extract sub-directory based on matching pattern from path and create symbolic links

I extracted the paths for files using find. The command to find paths is as follows: tmp=$(find $Some_Dir -name "*.gz") The output is: echo "$tmp" /xxxx/abc/bcd/def/something/...
1
vote
2answers
30 views

Intersection of two files based on two columns and one condition

I have two files. One file is tab separated and has content like col1. col2 col2 col4 Stef. 123 SE 383 Lena 938 Y X John 738 T Y ...
4
votes
1answer
5k views

Why doesn't ^s$ in regex match a string like "starts with s and ends with s"?

If ^ matches the beginning of a line and $ matches the end of a line, then why doesn't ^$ match the beginning and end of a string like the one in the title? How do you make such a regex that matches a ...
5
votes
6answers
62k views

How to find a specific 'string' and print out the whole line?

I have a file named myfile.csv containing the following: abc:123:myname:1231 def:423324:arbitrary:value:string StackExchange:Unix:Linux From the terminal I run ./myscript.sh def The contents of ...
162
votes
6answers
569k views

Return only the portion of a line after a matching pattern

So pulling open a file with cat and then using grep to get matching lines only gets me so far when I am working with the particular log set that I am dealing with. It need a way to match lines to a ...
0
votes
4answers
34 views

How to extract a portion of an XML tag value in a Bash script

I have a XML file like this (A.xml) : <?xml version="1.0"?> <RunParameters xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-...
1
vote
5answers
140 views

How to use awk to extract just the band name from the output of tageditor?

I'm trying to organize my Music better m4a, mp3, etc. Since the files are tagged I thought I could make a script to read the file and pull the album and artist info from the file then mv the file in ...
-2
votes
1answer
56 views

How to only display the text after the third, first, and fifth comma in this order in Unix?

I want to display the text after the 3rd, 1st and 5th comma from a given input text. Doing cut -d ',' -f 3,1,5 does the job but doesn't keep the order I want. Instead, it displays the text in ...
8
votes
9answers
3k views

Convert multi lines to single line with spaces and quotes

How can I convert multi lines to a single line with spaces and quotes using awk or tr or any other tool which makes it simple (but not for loops)? $ cat databases.txt Wp_new Frontend DB DB_EXT ...
-4
votes
0answers
24 views

Extract only `.lib` files from list of filenames, and write them to another file

I have a file file.txt that contains the names of .tcl and .lib files. I want to extract the names of the .lib files and write them to another file. I am using this, but not working grep '*.lib' file....
0
votes
2answers
64 views

Extract first block of consecutively matching lines from a file?

So I have a file like this: echo 'this line is added for demo purpose' echo 'do not extract this line and the line above' #!/usr/bin/env bash # header: add, replace, and delete header lines. # # ...
1
vote
5answers
114 views

Write AWK command's result into new CSV file

I have full list of files which contains this below texts. Sun Aug 22 19:00:00 2021 User-Name = "407359687" Acct-Status-Type = Interim-Update Acct-Output-Octets = ...
0
votes
0answers
34 views

how to get the evaluated text of bash output

When I run a command line it prints some output, and some of the text is different colors and there is some other formatting going on. When I get the string programmatically, it has a bunch of control ...
0
votes
2answers
73 views

Merging two files based on the first column depending on specific pattern's location in file 1

I have the following files: File 1 (around 7000 lines): 1010089 1402 6814 5543 1010121 6948 1402 2344 1305789 7589 7890 1402 3456889 1254 7389 1256 File 2 (around 300k lines): 1010089 26 48 33 ...
1
vote
5answers
341 views

awk case insensitive with gsub

I have "|" delimited text data, and want to transform a column values $ cat infile Mark|father Jason|SOn Jose|son Steffy|daugHter I want to search for (father|son|daughter) case ...
8
votes
7answers
1k views

Delete lines matching pattern in file1 and save these deleted lines to file2

I have file1, and I need to delete lines matching a pattern. But I would like to save these deleted lines in another file2. sed '/zz/!d' file1 > file2 sed -i '/zz/d' file1 Is there a way to ...
0
votes
0answers
25 views

Sed not working with / in the content itself [duplicate]

I need some help changing this config file via sed, for some reason its not working. I think it's cause i have a /. CONFIG="# 100.25.255.255/8 10.0.0.1 fd00:00:00::1" CURRENT_IP_RANGE="...
-2
votes
0answers
40 views

How do you sort list of strings according to number after the first comma?

Say we have a list of strings which are all the type attribute1,attribute2,etc and you want to sort them according to attribute2. All strings have the same number of attributes in the same order. I ...
-1
votes
0answers
80 views

How can I use awk, grep, and other tools to replace a single character in a file?

I am trying to replace a single a single IP in a string, the current shell script I am using to find this data is DATA WIREGUARD_CONFIG="# 10.0.0.0/8 fd00:00:00::0/8 0.0.0.0:51820 <RANDOM_DATA&...
4
votes
7answers
1k views

how to perform mathematical operations on numbers in a file using perl or awk?

I have a fileA.txt: RS0255_RS0083:115,124,129,141,143,168,170,180 RS0343_RS0083:112,113,163,175,181 RS0343_RS0255:94,101,107,164,179,183 I would like to perform mathematical operations on the numbers ...
1
vote
1answer
41 views

How to keep double quotes in a bash execution inside sed

I have a log file containning a lot of line like this: 2021/09/03-13:11:05;QUEUE.INBOUND;4;ID:myhost.mydomain.net-7756-1629822315072-199:973:1:1:1;MISC;"<?xml version="1.0" encoding=&...
1
vote
1answer
9k views

Ansible, attempting to get specific regex pattern to work with lineinfile module

As per my question above: "Ansible, attempting to get specific regex pattern to work with lineinfile module" Essentially, I'm trying to get it to work in a single play as opposed to 2 separate plays, ...
-3
votes
1answer
36 views

how to replace a nth column with column separated string using sed/awk

Assume I a have string like: xyz="123" ; acd="234" ; plk="1234" ; apl="3456" ; www="2342" Now I want to replace plk="123" with plk="...
1
vote
2answers
49 views

Merge sorted files on 2 columns

I want to merge some pre-sorted tab-delimited files: file bygroup.0: ancient-american mercury 1 164 ancient-american mh25 2 8717664 ancient-neolith tk11 262 40074321970 ancientdna ...
2
votes
2answers
95 views

compare columns from two different files and PRINT RECORDS FROM FIRST FILE those that DO NOT match from second file

I would like to compare columns from file one to two. Where column 2 of file1 should un-match from column 1 or 2 from file2 and print output from file 1. file1. cat test.head20.R2.fastq.tab @...
7
votes
10answers
16k views

Easily get a particular column from output without sed or awk

Is there a quicker way of getting a couple of column of values than futzing with sed and awk? For instance, if I have the output of ls -hal / and I want to just get the file and directory names and ...
1
vote
3answers
2k views

How to extract unique values from column #x with its corresponding values of column #y?

I have a comma',' FS filename as csv with n number of columns. I need to extract the unique value from colm.#1 with only corresponding values in colm.#10. So basically the column 10 is the date which ...
1
vote
1answer
42 views

Unix script to process data by comparing it to same key records

I have a problem in which Input file has Effective date and End date records. First 6 fields are the keys (12345A). We need to update End date(18 position) based on effective(8 position) date of the ...
0
votes
2answers
41 views

How to change a column name in multiple files?

I am wanting to change the name of column number 5 in each file to the file name itself for all files in a given directory. I have 250 files in the directory and the column names are tab-delimited. At ...
0
votes
6answers
143 views

Remove adjacent duplicated words from string

I have a string like this string: one one tow tow three three tow one three How can i remove duplicated words to make it like this: one tow three tow one three The point is that I want to write a ...
0
votes
1answer
33 views

Is there any software which can "get frequency of all strings in a file" or "which sort a file" as fast as possible using GPU?

I want to know if there is any linux utility or software, which is "highly opimized for sorting a file" or "which counts the frequency of all strings in file" (as an alternative) ...
0
votes
0answers
25 views

Text process filter IP addressees that appear in log too frequently and add them to deny list

I am attempting to extract any IP address from nginx that loads my websites page more than 10 times within a 60 second window but it must be for the homepage only GET / HTTP part of the nginx log /var/...
1
vote
3answers
385 views

csv to tsv with nested quoting

I need to convert CSV to TSV in bash. I found this solution and it works well, but not for all datasets as I will show below. For example, for a.txt: a,"test, part2 ""the start"&...
2
votes
2answers
2k views

Editing metadata with bash

I have a bunch of photos(jpeg, bmp, png) that I have arranged in such a manner that the metadata is in html form in a separate file. I would like create some script to read this html file and edit ...
3
votes
1answer
71 views

How to select based on an allowlist/blocklist using jq

Using jq, how can we select json elements from an array based on inclusion/exclusion of each element's key in some allowlist/blocklist? I want to do a case-insensitive contains (so allowlist/blocklist ...
23
votes
5answers
110k views

cut column 2 from text file

My text file has no delimiter to specify separator just spaces, how do I cut out column 2 to output file, 39 207 City and County of San Francisc REJECTED MAT = 0 78 412 Cases and ...
1
vote
2answers
99 views

filter lines featuring some character on sed

I have two unsorted lists of words, and I want to produce a new list excluding the words featuring some selected characters, and in order. I tried the following command: cat roa7.lst gr7.usl | sed -e ...

1
2 3 4 5
152