Questions tagged [text-processing]

Manipulation or examining of text by programs, scripts, etc.

Filter by
Sorted by
Tagged with
2 votes
8 answers
290 views

Replace last word occurrence in file

The scenario is I want to comment out the last occurrence of a word in shell script using sed. Assume I have temp.sh with the following content: Abc 123 Abc Sdf 2 Abc Abc Utyr Qww I want to replace ...
Curiouscat's user avatar
0 votes
2 answers
62 views

Fitering rows of one CSV file based on two values from another file in Bash

I have two CSV files. One is long with columns: Chr_Name, h, j, start_pos, end_pos. Another is short file including columns Chr_Name and position. I need to filter the rows from the first file based ...
nishat's user avatar
  • 1
1 vote
5 answers
144 views

Duplicate part of a line to another part

I would like to copy the first part (IxoscaEVm****t1_, without the '.p[number]') of the line starting with ">" and paste a before the last ":" of the same line. Input: >...
alex kiarie's user avatar
0 votes
0 answers
38 views

How to convert accented characters in an email

I want to send an email with the body ($BODY) containing accented characters like é, à, ô... echo "$TO" | mail -s $SUBJECT $BODY But the result shows unreadable characters. I tried to ...
Romeo Dejent's user avatar
0 votes
2 answers
63 views

perl: replace a string with variable containing new lines (line breaks)

I want to make edits containing meta characters. Here https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed one of solutions is via perl with s\Q ...
Martian2020's user avatar
  • 1,083
0 votes
2 answers
88 views

How to check the headers between 2 CSV files in unix?

File1 - filename.csv File2 - filename.csv I wanted to compare the headers between these 2 files in linux using a ksh script. case 1:- Say, the header of file1 is INPUT --> NAME,UNIT CODE, VAR ...
Jay's user avatar
  • 1
0 votes
2 answers
56 views

How can I associate special logic with file extensions?

Is there an easy way to have arbitrary text editors call special logic based on a file extension? For example, let's say I had a zip file that had a collection of files. text.txt image.jpg othercrap....
DeepDeadpool's user avatar
  • 1,204
0 votes
2 answers
77 views

sed "invalid command code W"

I am trying to prepend text to the begging/header of a set of .vtt text files with sed in a loop. I'm not replacing any text, only adding text. I need some line breaks in the text. The text should ...
Bleakley's user avatar
  • 185
4 votes
4 answers
615 views

Can I extract complete dates from file with grep command?

I need help using grep to extract a zoned date time from a file on a Linux system. Source file is a XML with the data below: <item start="20231010073000 +0100" stop="20231010100000 +...
aris's user avatar
  • 43
1 vote
3 answers
158 views

Convert lower-case to uppercase with the output to a new column using miller

I want to copy column NAME to column NAME-LOWERCASE. NAME-LOWERCASE should only contain lowercase letters. The uppercase should be untouched in all column expect NAME-LOWERCASE input NAME,test PTC,N ...
pwrsheller's user avatar
2 votes
4 answers
95 views

Delete a whole word from a CSV file that is not part of another word using SED

I've searched for an answer to this and have come close, but not close enough. We receive a CSV file that contains the text "NULL" which needs to be replaced by nothing. Example: Input ...
aRBee's user avatar
  • 23
0 votes
1 answer
49 views

How to list process with their depth?

I have a slightly different process tree question. I need to list the process and their depth in scalar value. pstree gives the result in a "graphic" manner. root 100930 0.0 0.3 ...
piovisqui's user avatar
  • 875
0 votes
1 answer
92 views

Need Help with Date expression - AWK in a csv file

Situation: csv file has date column with different date formats, I want to convert them into a unambiguous date format (like +"%m-%d-%Y"). Example File data : File name = Date_Test_new.csv 3/...
ArielUnix2023's user avatar
0 votes
1 answer
44 views

What is the best practice to change variables in automations

I am trying to create auto install Ubuntu OS with packer on Hyper-V and I have project like this: packer-main/ | |--->http/ | |--->user-data | |--->templates/ | |--->build.pkr....
robotiaga's user avatar
  • 111
4 votes
5 answers
575 views

Replace a block of spaces with a comma

I have a file in txt format that I want to convert to csv. Between each field there are blocks of spaces. The amount of spaces is not the same between every field. Is there a command with sed or awk ...
pwrsheller's user avatar
1 vote
4 answers
78 views

Remove duplicates by adding numerical suffix

How do I append a numerical suffix to lines to remove duplicates? Pseudo code: if currLine.startsWith("tag:") x = numFutureLinesMatching(currLine) if (x > 0) currLine = currLine + ...
Steven's user avatar
  • 738
6 votes
6 answers
1k views

Making single digit numbers between 1 and 9 into double digits, inside CSV file

I have a CSV file with thousands of lines like these 1664;4;5;35;37;43;5;6 1663;21;23;32;40;49;8;11 1662;16;17;34;35;44;5;10 1661;2;9;23;32;40;6;7 1660;23;25;30;44;47;9;12 1659;3;5;9;32;43;6;10 1658;4;...
Duck's user avatar
  • 4,534
-1 votes
2 answers
82 views

Use find, cat, etc... to generate an M3U playlist from album playlists

I have a music folder with selected playlists in each folder titled "playlist.m3u" (should also match *.m3u8). What I would like to do is have a shell script generate a playlist concatenated ...
mrjpaxton's user avatar
5 votes
1 answer
306 views

Why does 'tr' with '-c' option and set 2 extension add an unwarranted character to the end?

I wanted to use tr to substitute "illegal" characters in a string with a replacement character, where "illegal" characters are all outside of a set of "allowed" ...
AdminBee's user avatar
  • 21.8k
0 votes
5 answers
199 views

Uniquify first column and add numbers in second column in a file in linux

I am collecting data from multiple files and dumping into a single summary file. cat *files* | sort -u > final.rpt a 1 a 5 a 6 b 2 b 3 b 0 c 1 c 7 How can I uniquify based on string in first ...
Nirav's user avatar
  • 17
4 votes
9 answers
1k views

Convert hundred of numbers in a column to row separated by a comma

How can I convert the following from a column: 1 2 3 4 5 6 . . . . 98 99 100 To a row separated by a comma: 1,2,3,4,5,6,....,98,99,100 I am using Linux.
golf4life's user avatar
0 votes
5 answers
119 views

How to create a list of words from custom elements?

I want to make a list of words in a text file from custom elements: element1: word1 word2 word3 element2: word4 word5 element3: word6 word7 I want each to be made of a combination of the words in ...
thebunnyrules's user avatar
0 votes
2 answers
90 views

AWK: If two key columns match between files, add the 16th column of one file to the matching row of the other while keeping non-matching rows

I have two tab separated files (FileA.tsv and FileB.tsv). FileA.tsv id graph circle several columns... length 196-0 196 0 ---- 12874 195-1 195 1 ---- 12874 56-0 56 0 ---- 3349 115-1 115 1 ---- ...
JAL's user avatar
  • 3
0 votes
1 answer
29 views

Parse results of a "dig" bulk query

We use a security rating service which among other things, searches the world for imposter domains that resemble our domains. So it does find some of those but also finds ones that don't exist. It ...
JetJaguar's user avatar
25 votes
3 answers
5k views

Is it now safe to parse the output of GNU ls?

The accepted wisdom for the past few decades has been that it is never a good idea to parse the output of ls ([1],[2]). For example, if I want to save a file's modification date along with its name ...
terdon's user avatar
  • 235k
0 votes
6 answers
151 views

Merge successive lines into one line

In a text file, if the first 25 characters in a line are a space, how can I append that line to the previous line until another line comes along that starts with an ASCII character in column one. ...
Walter Reis's user avatar
1 vote
2 answers
206 views

Need help with a sed script for specific word replacements in a text file

I'm trying to automate a specific text replacement task for Romanian text using sed or any other suitable Unix tool. Here are my requirements: Replace all instances of the letter â with î inside ...
Mafsi's user avatar
  • 141
4 votes
4 answers
265 views

Using the command line, multiply all entries within a column to a fixed number, but only if the line is not a comment

I am trying to: Always add 0 0 0 to the first row of the file. Multiply 2*pi or 6.2832 to the first column of a three-column file formatted similar below, but only if the line begins with a numeral. ...
Carla H.'s user avatar
  • 161
0 votes
3 answers
76 views

How can i show just "total36" in this case?

For my Homework I tried in linux to just get the output total 36 from the output of ls -s as produced here: vboxuser@Ubuntu:~$ ls -s insgesamt 36 4 Bilder 4 Dokumente 4 Downloads 4 Musik 4 ...
AweBrawl's user avatar
0 votes
1 answer
104 views

Process CSV file through command line: Remove only middle rows between consecutive row entries if consecutive entries have the same 2nd column value

I have a CSV file with only 2 columns (but lots of rows) and the occasional irregular row which always starts with an asterisk (*) character and may span more than two columns. Using just the Linux ...
Carla H.'s user avatar
  • 161
1 vote
3 answers
110 views

How to execute a command after the previous non-terminating command prints a certain string?

I want to execute commandA which prints the following: line1 line2 line3 started successfully line4 ... ... And when the line started successfully is printed I want to start another command (and ...
Marinos An's user avatar
0 votes
2 answers
74 views

Can some command or script subtract lines in one file from another faster than grep?

I have a shell script that runs regularly and the following part of it causes a slowdown. grep -v -f RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt > RemainingBadIPs.txt It works. It just takes 156 ...
Darius Dauer's user avatar
1 vote
5 answers
48 views

Get value after a keyword that is in the middle of the line only from given line

Hey so I have the following line in the middle of a file and I only need to get the value following "energy=". The line number is stored in a variable called "lineNumber". There ...
karri104's user avatar
1 vote
2 answers
54 views

Replace the first column of a text file to be 6 minus the current value

I have the following file, space separated 2.5 235215 3.1925 3.7 125255 1.3125 I need to change the first column to be 6 minus the current value. The desired result is 3.5 ...
sunxd's user avatar
  • 123
0 votes
1 answer
41 views

Auto-translation of newline characters in the terminal

I'm following along here. I notice (this is not complete, just for demonstration) that at this point: #include <termios.h> raw.c_iflag &= ~(ICRNL | IXON); raw.c_oflag &= ~(OPOST); ...
user129393192's user avatar
0 votes
1 answer
75 views

Transforming a filename with 'tr' using RegEx doesn't work

I want to use the tr command to rename something like filename.ext to someName.ext. To do that I've tried echo "filename.ext" | tr -c ".a-z" "someName"` to replace the ...
Abhishek A Udupa's user avatar
3 votes
3 answers
92 views

Creating text files with comma separated list of values from one column based on shared value in second column from the commandline

I am hoping to create a series of text files each containing a comma separated list of values from column 1 of an input file based on their shared value in column 2 of the input file. Input file ...
Zoe Diaz-Martin's user avatar
1 vote
2 answers
116 views

extract file name from a list of files in a text file

probably simple, but my head hurts. i have a list of files with their path in a file named stuff.txt. Looks like this: c:\users\none\file1.txt c:\users\none\file2.txt g:\home\share\make.log how ...
Scott's user avatar
  • 11
5 votes
5 answers
809 views

How to convert all text into 1 except "0"s and first two fields in a csv file?

I have several large.csv files that I'd like to convert to binary (1 and 0) format. In which all cells containing text would become 1 and 0 would remain 0, except first two fields. head Test.csv Iss1,...
Debajyoti Kabiraj's user avatar
-6 votes
1 answer
51 views

How to align columns in this cide? [closed]

awk '{ print $1 ," ", $2 ," ", $3," " }' irsc_stn_namesort >lististgah.txt ABH1 30.6000 50.2532 AFJ 35.8553 51.7124 AFRZ 33.4252 59.0150 AKL 36....
ali asgh's user avatar
5 votes
5 answers
120 views

How to emulate #include <myFile.txt> with Linux commands

So, I am trying to create an output file from a source file that includes other files. (My use-case is actually YAML for Kubernetes/OpenShift but these .txt files show the aim.) For example: % cat ...
Jon Kemp's user avatar
0 votes
2 answers
50 views

How can I encode a plaintext configuration file then `tee` it into its destination?

I have a vanilla configuration for Nginx (nginx.conf) in plaintext format. It has Latin characters, space indents and is ~250 lines total. I use it when I compile Nginx from source. After the source ...
Pete Cooper's user avatar
14 votes
1 answer
861 views

Can `head` read/consume more input lines than it outputs?

Given the following 3 scripts: printf 'a\nb\nc\n' > file && { head -n 1; cat; } < file printf 'a\nb\nc\n' | { head -n 1; cat; } { head -n 1; cat; } < <(printf 'a\nb\nc\n') I'd ...
Ed Morton's user avatar
  • 29.2k
1 vote
3 answers
74 views

Command or script to delete all text in between two flags (html tags), in all files in a directory?

I have a directory full of html files that all have certain tags that I want deleted. For instance, in all files I have <p class="message"> </p> that I want gone, but the text ...
tei's user avatar
  • 13
5 votes
6 answers
1k views

sort the whole .csv based on the value in a certain column

I have a csv file with different revenues. I want to sort the csv file on revenue from high to low value. I can't find how to do this in terminal without the use of python. I don't want to use python. ...
pwrsheller's user avatar
-5 votes
1 answer
96 views

delete specific lines with regex [closed]

I would like a regex string to remove the second line, for example: Word 1 Word 2 I have a blank line between words 1 and 2, I would like a regex string where I can specify which line I want to ...
Doom's user avatar
  • 1
2 votes
2 answers
184 views

Update any date to the current date in a text file

I have a text file which contains several occurrences of dates. All the dates are written in the format Mon-DD-YYYY. I have to update all these dates in the current date keeping the format Mon-DD-YYYY....
user9952796's user avatar
0 votes
3 answers
55 views

comment out line between in a file based on pattern

File: /home/jian/Desktop/pg_sources/main/postgres/src/test/regress/sql/test_setup.sql 246: -- 247: -- Create some C functions that will be used by various tests. 248: -- 249: 250: CREATE FUNCTION ...
jian's user avatar
  • 539
0 votes
0 answers
61 views

Converting a TXT file to PDF from the command line using a custom font

I want to convert a UTF-8 txt file to PDF for printing with CUPS. The problem I'm facing is that all tools I've tried have either failed to format the PDF (resulting in a fully white document with no ...
psvg200's user avatar
-1 votes
1 answer
74 views

Why no one thought of the concept of joins before the 70s? [closed]

So, let me see if I have my history right: In 1976, the first system sold as an RDBMS was Multics Relational Data Store. I don't know if it had JOIN, but let's assume it did. The UNIX command join ...
Sebastian Carlos's user avatar

1
2 3 4 5
167