Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
8,365
questions
0
votes
2
answers
2k
views
Use reverse i search to cycle through only commands matching pattern
The contents of the command history file using history | less is:
555 ls
556 ls -a
557 echo "hello"
558 echo "hello again"
559 cd
560 pwd
561 echo "hello hello"
562 ls
563 echo "hello hello hello"
...
- 300
4
votes
7
answers
824
views
Rearrange file content to prepend record 'header' line to each record 'body' line
Anyone have idea how to turn this:
FX_AM140_EML AM140
Backend/aa.java
Backend/bb.java
Backend/cc.java
Backend/dd.java
Backend/ee.java
FX_AM172_EML AM172
Backend/aa.java
Backend/bb.java
Into this?
...
- 111k
-1
votes
3
answers
209
views
Need help regarding making multiple columns
I need to convert my data into multiple columns. Input data:
a1 n1
a2 n2
a3 n3
. .
. .
. .
an nn
a1 m1
a2 m2
a3 m3
. .
. .
. .
an mn
a1 x1
a2 x2
a3 x3
. .
. .
. .
an xn
a1 y1
a2 y2
a3 y3
. .
...
- 1,780
15
votes
3
answers
16k
views
Main difference between tr (translate) to sed and awk
AFAIC both sed and awk are general purpose text processing utilities with whom a user can get quite similar results, in a slightly different syntax:
With both, a user could add, replace/translate and ...
- 327k
-1
votes
1
answer
143
views
Create new CSV files based on latest timestamp data
I have a script that accesses multiple dat files and produce csv files based on previous day’s data. these DAT files are updated on per minute bases with data from various instruments.
Script snippet:
...
- 267
3
votes
5
answers
695
views
How can I reformat blocks of data until the end of the file is reached?
I have a file that looks like this:
# Time-averaged data for fix avetimeall
# TimeStep Number-of-rows
# Row c_gyrationchunkall
1000 3
1 2.09024e-14
2 4.88628
3 5.69321
2000 3
1 2.10518e-14
2 8.33702
3 ...
- 4,590
0
votes
3
answers
93
views
Remove everything in a third column but only keep specific text
I have a data set with three columns:
https://drive.google.com/file/d/1gtCssfAXHxRjGfX8uTAaimGPWCA2cnci/view?usp=sharing
Here are the first few lines:
ID transcript_id go_description
...
- 1,780
2
votes
2
answers
2k
views
How to copy strings from middle of lines (between strings) to the end of lines
I have a text file (1.txt) as follows:
word1
word2
word3
I use sed to add strings to the beginning and end of every line as follows:
sed 's/^/blahblah1 /g' 1.txt > 2.txt
sed 's/$/ blahblah2/g' ...
CommunityBot
- 1
11
votes
3
answers
11k
views
Basic sed command on large one-line file: couldn't re-allocate memory
I have a 250 MB text file, all in one line.
In this file I want to replace a characters with b characters:
sed -e "s/a/b/g" < one-line-250-mb.txt
It fails with:
sed: couldn't re-allocate memory
...
- 101
0
votes
4
answers
2k
views
How to do a sed substitution (s///g) based on a list? I need to swap multiple words, with other corresponding words
I don't think this question has been asked before, so I don't know if sed is capable of this.
Suppose I have a bunch of numbers in a sentence that I need to expand into words, a practical example ...
CommunityBot
- 1
0
votes
3
answers
73
views
Copy matching lines to the next line and modify?
Given a file like this:
a foo 1
b bar 2
c foo 3
d baz 4
e bar 5
I want to modify that file as follows: copy every line containing bar to the following line and change bar to bar2, so that the result ...
- 30.3k
31
votes
5
answers
31k
views
How to report "sed" in-place changes
When using sed to replace strings in-place, is there a way to make it report the changes it does (without relying on a diff of old and new files)?
For instance, how can I change the command line
...
- 534k
0
votes
0
answers
14
views
How to backup Thunderbird to mbox for use with the Pine or Alpine e-mail client?
How should the mbox backup from Thunderbird show? Perhaps there's a validation tool?
First screen:
ALPINE 2.25 FOLDER LIST ...
-2
votes
3
answers
519
views
If line starts with a character remove text until another character in same line
Some lines start with a #, then any number (one or more digits), then =. Those lines starting with hash, remove text from beginning of line up to equal sign (including it).
Example in file:
some text
#...
- 205
1
vote
1
answer
53
views
Replacing a user entered string with another
So I am writing a program in bash involving a madlibs and I have to create a function that replaces the first occurence of a word for example. If the first word inputted was apple I can use this ...
- 46
9
votes
7
answers
5k
views
How to efficiently split up a large text file wihout splitting multiline records?
I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
- 902
3
votes
4
answers
272
views
Isolate one specific instance of a word in a string
I have a grep command to process the string
test-test test
I want the command to only show me the "free-standing" word test highlighted, but not test-test.
However, when I run
echo "...
- 2,613
0
votes
6
answers
175
views
Delete all lines before n lines above the line containing the first match
If a file contains the following:
Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
How to delete the lines above two lines before "Line 5", so that the file will now contain ...
- 1,780
2
votes
3
answers
544
views
Copying second occurrence pattern to a new file in sed
I have a certificate file which has 2 certificates in it. Starts with ---BEGIN CERTIFICATE-- and ends with ---END CERTIFICATE----.
Example:
---BEGIN CERTIFICATE--
blah1 blah1
---END CERTIFICATE----
--...
- 160
4
votes
4
answers
19k
views
Use `cut` to extract a list from /etc/passwd
How could I use the cut command to extract a list of usernames and login shells from the /etc/passwd file, where the resulting usernames and login shells are separated by a single space?
Sort the ...
- 4,845
23
votes
3
answers
171k
views
Adding a column of values in a tab delimited file
How can I add a column of values in a tab-separated file which has a certain number of rows? I have a input file like this:
SPATA17 1 217947738
LYPLAL1 1 219383905
FAM47E 4 77192838
SHROOM3 4 ...
- 22.4k
0
votes
5
answers
106
views
Delete lines below, excluding the line with match pattern
One can use sed '/pattern/Q' or sed '/pattern/,$d' to delete lines below a match pattern, but it also deletes the line containing the match pattern.
How to exclude the line containing the match ...
- 2,613
-3
votes
1
answer
35
views
Divide values in all columns of one file by a value in the corrosponding column if a second file
I have two files. The first one looks like this:
The second file looks like this:
I need to normalize the first table by the second, meaning for all values in column "sample" in file 1, I ...
-4
votes
3
answers
334
views
Counting matches with grep, when multiple matches per line are possible
The standard usage of grep is to return lines that match a pattern.
If a line can contain several matches of the pattern, how can I count each match individually, not the total number of matches?
- 239k
1
vote
4
answers
318
views
Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs
I have a tsv file with tab-separated columns. I want to obtain the 5th column, which has space-separated values. Convert the space-separation to tab-separation and save as a new file.
Attempt:
cut -d&...
- 327k
2
votes
4
answers
174
views
Parse text file add semicolons before and after numbers; join all other words that are before/ between numbers with hyphens
I have a text file with accounts like:
Input Sample
Paid 100 15/02/2022 3000
recd 50 15/02/2022 nelur trip 3050
PAID 80 25/03/2022 Adjusted towards trip 3130
14 PAID 50 26/03/2022 Given to Nate Cash (...
- 66.7k
5
votes
3
answers
17k
views
Why grep doesn't show me the matching word until the next space?
I have a temp file, and I want to grep the only word that matches the pattern instead of whole word. i tried grep -o <pattern> file but its not giving me desired output
Input
xi29 vddf vss ...
- 534k
1
vote
4
answers
155
views
edit text/srt file
I have a .srt file like this
Input File
1
00:00:17,920 --> 00:00:21,159
The essential is invisible to the eye. 3
2
00:00:21,160 --> 00:00:22,559
This phrase comes from 4
3
00:00:22,560 --> ...
- 2,613
-4
votes
4
answers
149
views
Bash Script to covert rows in to columns between matching patterns
My Input file looks like this..
START
line1
line2
line3
END
START
line11
line22
line33
END
I want my script output to be like this:
START line1 line2 line3 END
START line11 line22 line33 END
Please ...
- 2,613
27
votes
12
answers
38k
views
awk print from nth column to last
I want to print all columns from nth to last column of a line
Input String in file
vddp vddpi vss cb0 cb1 cb2 cb3 ct0 ct1 ct2 ct3
Command
cat <file> | awk ' { for (i=3; i<=NF; i++) ...
- 534k
7
votes
3
answers
678
views
Merge files using zipper method / late merge
I am searching for a nice method of merging two or more files line by line using the zipper method (also called late merge). Assuming we have three files, the result should look like this:
line1 ...
- 2,452
697
votes
4
answers
310k
views
Why is printf better than echo?
I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
- 534k
622
votes
11
answers
1.3m
views
How to append multiple lines to a file
I am writing a bash script to look for a file if it doesn't exist then create it and append this to it:
Host localhost
ForwardAgent yes
So "line then new line 'tab' then text" I think its a ...
- 661
0
votes
4
answers
2k
views
Shell Script: Want to delete two consecutive lines matching pattern from specific line
I want to delete specific two consecutive lines matching patterns from specific line from a file.
For e.g. file contents are like below.
Line1: a
Line2: b
Line3: c
Line4: Name: 123
Line5: xyz
Line6: ...
CommunityBot
- 1
5
votes
6
answers
1k
views
sort the whole .csv based on the value in a certain column
I have a csv file with different revenues. I want to sort the csv file on revenue from high to low value. I can't find how to do this in terminal without the use of python.
I don't want to use python.
...
- 2,613
3
votes
8
answers
683
views
How to enclose each line in single/double quote and end with comma except for the last line should not have a comma?
I want to enclose these in single/double quote and ending with a comma. But I also need it so that the last line does not include the comma as it is the last record.
I've found several post of the ...
- 2,613
5
votes
6
answers
9k
views
CSV file processing - remove quotes and replace comma delimiter with tab
Need to convert delimited files with a shell command. There are two variations for the received input file, one with double quotes and another without quotes and both files have the comma as a ...
- 2,613
3
votes
1
answer
66
views
How can I pipe text that contains ANSI movement characters ESC[xC and ESC[xD and output the text formatted accodingly
I have text files that include the ANSI movement codes ESC[xC and ESC[xD. I want to filter these out but with each line output with those codes taken into account.
Consider the line:
this cat is ...
- 33
5
votes
6
answers
677
views
Keep unique values (comma separated) from each column
I have a .tsv (tab-separated columns) file on a Linux system with the following columns that contain different types of values (strings, numbers) separated by a comma:
col1 col2
. NS,NS,...
- 22.4k
1
vote
4
answers
192
views
Using sed to replace the 1st ";" of a line with a string and then the 2nd ";" with another string
I have a semicolon separated playlist in the following format:
Thåström ; Stjärna som är din ; Skebokvarnsv. 209
I want to replace the first ; with a dash (-) and the 2nd ; with the string "- ...
- 30.3k
278
votes
19
answers
450k
views
How to add a newline to the end of a file?
Using version control systems I get annoyed at the noise when the diff says No newline at end of file.
So I was wondering: How to add a newline at the end of a file to get rid of those messages?
- 203
3
votes
4
answers
4k
views
How to get the last occurrence of lines between two patterns from a file?
I have a log file which reports on the output of a process, I'd like to extract all lines from between the last occurrence of two patterns.
The patterns will be along the lines of;
Summary process ...
- 1
2
votes
1
answer
5k
views
Need to find a response time which takes from 1-3 seconds in Apache logs
I need to find the API response time from an Apache log file. It's like a response time which is takes between 1 to 2 secound or 2 to 3 second. $6 is response time and values comes in microseconds.
I ...
CommunityBot
- 1
120
votes
13
answers
161k
views
Replace environment variables in a file with their actual values?
Is there an easy way to substitute/evaluate environment variables in a file?
Like let's say I have a file config.xml that contains:
<property>
<name>instanceId</name>
<...
- 22.4k
134
votes
20
answers
344k
views
How to count the number of a specific character in each line?
I was wondering how to count the number of a specific character in each line by some text processing utilities?
For example, to count " in each line of the following text
"hello!"
Thank you!
The ...
- 2,613
1
vote
5
answers
127
views
Add two delimiters after every sixth comma
I'm trying to expand on this question but can't figure out this issue:
Let's say I've got a file roll.txt:
echo "'123456789','987651234','129873645','213456789','987612345','543216789','432156789'...
- 1,780
0
votes
3
answers
177
views
How can I sort the lines in this text file?
I have a text file with book name and chapter in the bible. I would like to sort the lines so that
lines are first sorted according to the order between their books, the lexical order between the ...
- 2,613
0
votes
2
answers
5k
views
Comparing csv files to lookup column 1 then check values in column 2
OK, I will try and explain what I need to do as best as possible.
Basically I have two CSV files, as per the examples below:
File 1:
Column 1, Column 2
abc , 123
def , 234
adf , 567
...
CommunityBot
- 1
41
votes
4
answers
31k
views
How to display dependencies given in a makefile as a tree?
Problem
I want to see the dependencies for one or more targets of a makefile. So I am looking for a program that can parse makefiles and then will represent the dependencies in some tree-like format (...
- 399
1
vote
5
answers
569
views
Copy contents of one file to another using sed
I want to copy all text from one file into another file using sed on Alma Linux 8. For example, the first file, old.txt, is:
192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
...
And the ...
- 2,613