Questions tagged [text-processing]
Manipulation or examining of text by programs, scripts, etc.
8,338
questions
2
votes
8
answers
290
views
Replace last word occurrence in file
The scenario is I want to comment out the last occurrence of a word in shell script using sed.
Assume I have temp.sh with the following content:
Abc 123 Abc
Sdf 2
Abc
Abc
Utyr
Qww
I want to replace ...
- 23
0
votes
2
answers
62
views
Fitering rows of one CSV file based on two values from another file in Bash
I have two CSV files.
One is long with columns: Chr_Name, h, j, start_pos, end_pos.
Another is short file including columns Chr_Name and position.
I need to filter the rows from the first file based ...
- 1
1
vote
5
answers
144
views
Duplicate part of a line to another part
I would like to copy the first part (IxoscaEVm****t1_, without the '.p[number]') of the line starting with ">" and paste a before the last ":" of the same line.
Input:
>...
- 139
0
votes
0
answers
38
views
How to convert accented characters in an email
I want to send an email with the body ($BODY) containing accented characters like é, à, ô...
echo "$TO" | mail -s $SUBJECT $BODY
But the result shows unreadable characters.
I tried to ...
0
votes
2
answers
63
views
perl: replace a string with variable containing new lines (line breaks)
I want to make edits containing meta characters. Here https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed one of solutions is via perl with s\Q ...
- 1,083
0
votes
2
answers
88
views
How to check the headers between 2 CSV files in unix?
File1 - filename.csv
File2 - filename.csv
I wanted to compare the headers between these 2 files in linux using a ksh script.
case 1:-
Say,
the header of file1 is INPUT -->
NAME,UNIT CODE, VAR
...
- 1
0
votes
2
answers
56
views
How can I associate special logic with file extensions?
Is there an easy way to have arbitrary text editors call special logic based on a file extension?
For example, let's say I had a zip file that had a collection of files.
text.txt
image.jpg
othercrap....
- 1,204
0
votes
2
answers
77
views
sed "invalid command code W"
I am trying to prepend text to the begging/header of a set of .vtt text files with sed in a loop. I'm not replacing any text, only adding text.
I need some line breaks in the text.
The text should ...
- 185
4
votes
4
answers
615
views
Can I extract complete dates from file with grep command?
I need help using grep to extract a zoned date time from a file on a Linux system.
Source file is a XML with the data below:
<item start="20231010073000 +0100" stop="20231010100000 +...
- 43
1
vote
3
answers
158
views
Convert lower-case to uppercase with the output to a new column using miller
I want to copy column NAME to column NAME-LOWERCASE. NAME-LOWERCASE should only contain lowercase letters. The uppercase should be untouched in all column expect NAME-LOWERCASE
input
NAME,test
PTC,N
...
- 175
2
votes
4
answers
95
views
Delete a whole word from a CSV file that is not part of another word using SED
I've searched for an answer to this and have come close, but not close enough.
We receive a CSV file that contains the text "NULL" which needs to be replaced by nothing. Example:
Input
...
- 23
0
votes
1
answer
49
views
How to list process with their depth?
I have a slightly different process tree question. I need to list the process and their depth in scalar value. pstree gives the result in a "graphic" manner.
root 100930 0.0 0.3 ...
- 875
0
votes
1
answer
92
views
Need Help with Date expression - AWK in a csv file
Situation: csv file has date column with different date formats, I want to convert them into a unambiguous date format (like +"%m-%d-%Y").
Example File data : File name = Date_Test_new.csv
3/...
0
votes
1
answer
44
views
What is the best practice to change variables in automations
I am trying to create auto install Ubuntu OS with packer on Hyper-V and I have project like this:
packer-main/
|
|--->http/
| |--->user-data
|
|--->templates/
| |--->build.pkr....
- 111
4
votes
5
answers
575
views
Replace a block of spaces with a comma
I have a file in txt format that I want to convert to csv. Between each field there are blocks of spaces. The amount of spaces is not the same between every field.
Is there a command with sed or awk ...
- 175
1
vote
4
answers
78
views
Remove duplicates by adding numerical suffix
How do I append a numerical suffix to lines to remove duplicates?
Pseudo code:
if currLine.startsWith("tag:")
x = numFutureLinesMatching(currLine)
if (x > 0)
currLine = currLine + ...
- 738
6
votes
6
answers
1k
views
Making single digit numbers between 1 and 9 into double digits, inside CSV file
I have a CSV file with thousands of lines like these
1664;4;5;35;37;43;5;6
1663;21;23;32;40;49;8;11
1662;16;17;34;35;44;5;10
1661;2;9;23;32;40;6;7
1660;23;25;30;44;47;9;12
1659;3;5;9;32;43;6;10
1658;4;...
- 4,534
-1
votes
2
answers
82
views
Use find, cat, etc... to generate an M3U playlist from album playlists
I have a music folder with selected playlists in each folder titled "playlist.m3u" (should also match *.m3u8).
What I would like to do is have a shell script generate a playlist concatenated ...
- 25
5
votes
1
answer
306
views
Why does 'tr' with '-c' option and set 2 extension add an unwarranted character to the end?
I wanted to use tr to substitute "illegal" characters in a string with a replacement character, where "illegal" characters are all outside of a set of "allowed" ...
- 21.8k
0
votes
5
answers
199
views
Uniquify first column and add numbers in second column in a file in linux
I am collecting data from multiple files and dumping into a single summary file.
cat *files* | sort -u > final.rpt
a 1
a 5
a 6
b 2
b 3
b 0
c 1
c 7
How can I uniquify based on string in first ...
- 17
4
votes
9
answers
1k
views
Convert hundred of numbers in a column to row separated by a comma
How can I convert the following from a column:
1
2
3
4
5
6
.
.
.
.
98
99
100
To a row separated by a comma:
1,2,3,4,5,6,....,98,99,100
I am using Linux.
- 57
0
votes
5
answers
119
views
How to create a list of words from custom elements?
I want to make a list of words in a text file from custom elements:
element1:
word1
word2
word3
element2:
word4
word5
element3:
word6
word7
I want each to be made of a combination of the words in ...
- 1,067
0
votes
2
answers
90
views
AWK: If two key columns match between files, add the 16th column of one file to the matching row of the other while keeping non-matching rows
I have two tab separated files (FileA.tsv and FileB.tsv).
FileA.tsv
id
graph
circle
several columns...
length
196-0
196
0
----
12874
195-1
195
1
----
12874
56-0
56
0
----
3349
115-1
115
1
----
...
- 3
0
votes
1
answer
29
views
Parse results of a "dig" bulk query
We use a security rating service which among other things, searches the world for imposter domains that resemble our domains. So it does find some of those but also finds ones that don't exist. It ...
- 13
25
votes
3
answers
5k
views
Is it now safe to parse the output of GNU ls?
The accepted wisdom for the past few decades has been that it is never a good idea to parse the output of ls ([1],[2]). For example, if I want to save a file's modification date along with its name ...
- 235k
0
votes
6
answers
151
views
Merge successive lines into one line
In a text file, if the first 25 characters in a line are a space, how can I append that line to the previous line until another line comes along that starts with an ASCII character in column one. ...
- 13
1
vote
2
answers
206
views
Need help with a sed script for specific word replacements in a text file
I'm trying to automate a specific text replacement task for Romanian text using sed or any other suitable Unix tool. Here are my requirements:
Replace all instances of the letter â with î inside ...
- 141
4
votes
4
answers
265
views
Using the command line, multiply all entries within a column to a fixed number, but only if the line is not a comment
I am trying to:
Always add 0 0 0 to the first row of the file.
Multiply 2*pi or 6.2832 to the first column of a three-column file formatted similar below, but only if the line begins with a numeral. ...
- 161
0
votes
3
answers
76
views
How can i show just "total36" in this case?
For my Homework I tried in linux to just get the output total 36 from the output of ls -s as produced here:
vboxuser@Ubuntu:~$ ls -s
insgesamt 36
4 Bilder 4 Dokumente 4 Downloads 4 Musik 4 ...
- 9
0
votes
1
answer
104
views
Process CSV file through command line: Remove only middle rows between consecutive row entries if consecutive entries have the same 2nd column value
I have a CSV file with only 2 columns (but lots of rows) and the occasional irregular row which always starts with an asterisk (*) character and may span more than two columns. Using just the Linux ...
- 161
1
vote
3
answers
110
views
How to execute a command after the previous non-terminating command prints a certain string?
I want to execute commandA which prints the following:
line1
line2
line3
started successfully
line4
...
...
And when the line started successfully is printed I want to start another command (and ...
- 779
0
votes
2
answers
74
views
Can some command or script subtract lines in one file from another faster than grep?
I have a shell script that runs regularly and the following part of it causes a slowdown.
grep -v -f RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt > RemainingBadIPs.txt
It works. It just takes 156 ...
- 45
1
vote
5
answers
48
views
Get value after a keyword that is in the middle of the line only from given line
Hey so I have the following line in the middle of a file and I only need to get the value following "energy=".
The line number is stored in a variable called "lineNumber".
There ...
- 13
1
vote
2
answers
54
views
Replace the first column of a text file to be 6 minus the current value
I have the following file, space separated
2.5 235215 3.1925
3.7 125255 1.3125
I need to change the first column to be 6 minus the current value. The desired result is
3.5 ...
- 123
0
votes
1
answer
41
views
Auto-translation of newline characters in the terminal
I'm following along here.
I notice (this is not complete, just for demonstration) that at this point:
#include <termios.h>
raw.c_iflag &= ~(ICRNL | IXON);
raw.c_oflag &= ~(OPOST);
...
- 143
0
votes
1
answer
75
views
Transforming a filename with 'tr' using RegEx doesn't work
I want to use the tr command to rename something like filename.ext to someName.ext.
To do that I've tried
echo "filename.ext" | tr -c ".a-z" "someName"`
to replace the ...
- 111
3
votes
3
answers
92
views
Creating text files with comma separated list of values from one column based on shared value in second column from the commandline
I am hoping to create a series of text files each containing a comma separated list of values from column 1 of an input file based on their shared value in column 2 of the input file.
Input file ...
1
vote
2
answers
116
views
extract file name from a list of files in a text file
probably simple, but my head hurts.
i have a list of files with their path in a file named stuff.txt. Looks like this:
c:\users\none\file1.txt
c:\users\none\file2.txt
g:\home\share\make.log
how ...
- 11
5
votes
5
answers
809
views
How to convert all text into 1 except "0"s and first two fields in a csv file?
I have several large.csv files that I'd like to convert to binary (1 and 0) format. In which all cells containing text would become 1 and 0 would remain 0, except first two fields.
head Test.csv
Iss1,...
-6
votes
1
answer
51
views
How to align columns in this cide? [closed]
awk '{ print $1 ," ", $2 ," ", $3," " }' irsc_stn_namesort >lististgah.txt
ABH1 30.6000 50.2532
AFJ 35.8553 51.7124
AFRZ 33.4252 59.0150
AKL 36....
- 1
5
votes
5
answers
120
views
How to emulate #include <myFile.txt> with Linux commands
So, I am trying to create an output file from a source file that includes other files.
(My use-case is actually YAML for Kubernetes/OpenShift but these .txt files show the aim.)
For example:
% cat ...
- 51
0
votes
2
answers
50
views
How can I encode a plaintext configuration file then `tee` it into its destination?
I have a vanilla configuration for Nginx (nginx.conf) in plaintext format. It has Latin characters, space indents and is ~250 lines total. I use it when I compile Nginx from source. After the source ...
- 131
14
votes
1
answer
861
views
Can `head` read/consume more input lines than it outputs?
Given the following 3 scripts:
printf 'a\nb\nc\n' > file && { head -n 1; cat; } < file
printf 'a\nb\nc\n' | { head -n 1; cat; }
{ head -n 1; cat; } < <(printf 'a\nb\nc\n')
I'd ...
- 29.2k
1
vote
3
answers
74
views
Command or script to delete all text in between two flags (html tags), in all files in a directory?
I have a directory full of html files that all have certain tags that I want deleted. For instance, in all files I have <p class="message"> </p> that I want gone, but the text ...
- 13
5
votes
6
answers
1k
views
sort the whole .csv based on the value in a certain column
I have a csv file with different revenues. I want to sort the csv file on revenue from high to low value. I can't find how to do this in terminal without the use of python.
I don't want to use python.
...
- 175
-5
votes
1
answer
96
views
delete specific lines with regex [closed]
I would like a regex string to remove the second line, for example:
Word 1
Word 2
I have a blank line between words 1 and 2, I would like a regex string where I can specify which line I want to ...
- 1
2
votes
2
answers
184
views
Update any date to the current date in a text file
I have a text file which contains several occurrences of dates. All the dates are written in the format Mon-DD-YYYY. I have to update all these dates in the current date keeping the format Mon-DD-YYYY....
- 79
0
votes
3
answers
55
views
comment out line between in a file based on pattern
File: /home/jian/Desktop/pg_sources/main/postgres/src/test/regress/sql/test_setup.sql
246: --
247: -- Create some C functions that will be used by various tests.
248: --
249:
250: CREATE FUNCTION ...
- 539
0
votes
0
answers
61
views
Converting a TXT file to PDF from the command line using a custom font
I want to convert a UTF-8 txt file to PDF for printing with CUPS. The problem I'm facing is that all tools I've tried have either failed to format the PDF (resulting in a fully white document with no ...
- 1
-1
votes
1
answer
74
views
Why no one thought of the concept of joins before the 70s? [closed]
So, let me see if I have my history right:
In 1976, the first system sold as an RDBMS was Multics Relational Data Store. I don't know if it had JOIN, but let's assume it did.
The UNIX command join ...
- 166