Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

8
votes
2answers
1k views

Parsing XML's , JSON's and newer data file formats in UNIX using command line utilities

Unix environment has some excellent tools for parsing text in various forms. However off late the data is not in the regular set of formats(CSV,TSV,Record based or some other delimiter based) it used ...
4
votes
4answers
885 views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
3
votes
2answers
146 views

How to write this pseudocode where condition

This question is extension of this question about listing questions of directories. I get a problem that I need to understand the original section of the text. Current output where you see the ...
2
votes
6answers
4k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
10
votes
2answers
503 views

/usr/bin/ptx: Can you provide a use case or two?

I was going through the list of files included in coreutils and I was able to come up with an example of how I could personally use all of the commands provided except for ptx. Can you give one or two ...
8
votes
5answers
553 views

Convert file contents to lower case

I have temp file with some lower-case and upper-case contents. Input Contents of my temp file: hi Jigar GANDHI jiga I want to convert all upper to lower. Command I tried the following command: ...
8
votes
2answers
306 views

How can I delete all text between nested curly brackets in a multiline text file?

This question comes from How can I delete all text between curly brackets in a multiline text file? (just the same, but without the requirements for nesting). Example: This is { {the multiline ...
8
votes
5answers
2k views

`highlight` command

Is there a command which can be used to easily highlight (with colors, bold text or inverted back-/foreground) specific strings in stdin while passing the entire string to stdout? I'm thinking of ...
7
votes
4answers
234 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
4
votes
4answers
5k views

Concatenate multiple files with same header

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other ...
4
votes
1answer
2k views

How does one find and replace text in all open files with geany?

How does one find and replace text in all open files with geany?
3
votes
1answer
192 views

What is the difference between > and >> (especially as it relates to use with the cat program)?

Suppose that I have a file called temp.txt. Using the cat program, I would like to add the contents of this file to the end of myfile.txt -- creating myfile.txt if it does not exist and appending to ...
2
votes
3answers
187 views

how to create a data file with a column of date from day1 to dayN?

I was wondering if I could generate a hourly or daily time step date in a column starting from some year in the past to some day in the near past or today. to be more clear, I want to create one ...
8
votes
2answers
486 views

wc -L reports a line-length of 8 for a tab-char. bug or feature?

-L is a useful feature of wc, or so I thought. It prints the length of the longest line. For some reason it expands a single-byte tab-char to a length of 8. Is there some way to set this to not ...
6
votes
2answers
453 views

sort every field numerically, varying field counts

I'm trying to sort some data using sort. I noticed it was sorting by digit rather than number, so I added the -n flag. It then seemingly only numerically sorts on the first field though. Breaking it ...
6
votes
3answers
29k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
6
votes
2answers
3k views

grep surrounding characters of a match

I'm looking to do a find and replace within a giant database dump, and it's not doing what I think should happen. I'd like to grep for my target string in the file, and then see the surrounding 8 ...
4
votes
3answers
2k views

Split file into multiple small files, separate by the newline symbol

Is there a utility that split file by newline symbol? e.g if a file contains the following lines, aa bbb cccc If I want to split it to 3 files, the desired output would be: aa, bbb And cccc (in 3 ...
4
votes
2answers
285 views

tac command's option creates strange output [comprehension question]

Say I have this file, containing nothing but a b c b a Using tac --separator=a file in BASH [on a Debian based Linux], I get this: # empty line # empty line b c ...
4
votes
3answers
832 views

Delete the matching line and several more from a file

I have a text file called file_a.txt. My first command is grep -A 12 ".production =" file_a.txt The output is a few block. Each block of string contains 13 rows I specifically want to delete all ...
3
votes
2answers
111 views

How can I partition a set of words given pairs that must end up together?

I have a space or comma separated table with two columns, each row representing equivalence of the two words. A B B C B D C E F G What I want is a table with each row listing all mutually ...
3
votes
3answers
259 views

Rearrange text file by prepending data rows with header row

I have a file whose content looks like this: 2009 150 0 0 0 0000 75.316 0.0400390625 0.00007 0.00000 0.8980 76.216 0.0400390625 0.00007 1.00000 0.9046 77.217 0.0400390625 ...
2
votes
3answers
254 views

Remove spaces and headers from a dumped database table

I have dumped the database table data into a flat file and below is how the data looks like : (Kindly copy from below ;metier_code ; ;-------------------------; (0 rows affected) ...
2
votes
5answers
2k views

vlookup function in unix

How to make something similar to Excel's vlookup function in Unix? excerpt from office website, VLOOKUP The V in VLOOKUP stands for vertical. Use VLOOKUP instead of HLOOKUP when your comparison ...
2
votes
2answers
9k views

How can I replace text after a specific word using sed?

I have a file named .ignore. In need to replace the projdir. For example: ignore \..* ignore README projdir Snake I need to replace Snake with, for example, "PacMan". I read the man page, but I ...
1
vote
3answers
120 views

using awk command to compare the two files and print all the columns then missing display as NA

File 1: 123, 234 345, 789 678, 900 File 2 123, abc 345, dnc Desired output (all columns in file1 and missing value in file two as NA): 123,234,abc 345,789,dnc 678,900,NA
1
vote
4answers
836 views

Replace text between brackets

I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like: SOMETHING [BAR, WIBBLE] SOMETHING [BAR] This gives the desired result of: SOMETHING [FOO, FOO] SOMETHING [FOO] ...
-1
votes
1answer
41 views

replace pattern matching

I want to replace part of a file's data, with data from another file. Suppose file1 has data as is written below and file2 has some data I want to store replace file1 data from file2 from starting ...
39
votes
5answers
24k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
54
votes
7answers
56k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid off those messages?
27
votes
14answers
56k views

How to count the number of a specific character in each line?

I was wondering how to count the number of a specific character in each line by some text processing utilities? For example, to count " in each line of the following text "hello!" Thank you! The ...
30
votes
4answers
7k views

How can I wrap text at a certain column size?

I know that I can use something like cat test.txt | pr -w 80 to wrap lines to 80 characters wide, but that puts a lot of space on the top and bottom of the printed lines and it does not work right on ...
12
votes
11answers
11k views

How can I split a text file into multiple text files?

I have a text file called entry.txt that contains the following: [ entry1 ] 1239 1240 1242 1391 1392 1394 1486 1487 1489 1600 1601 1603 1657 1658 1660 2075 2076 2078 2322 2323 2325 2740 2741 2743 ...
20
votes
5answers
6k views

echo a file without the first and last lines

Is there a simple way I can echo a file, skipping the first and last lines? I was looking at piping from head into tail, but for those it seems like I would have to know the total lines from the ...
19
votes
5answers
11k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
12
votes
6answers
23k views

How to convert TXT to PDF?

I want to convert .txt files to .pdf. I'm using this: ls | while read ONELINE; do convert -density 400 "$ONELINE" "$(echo "$ONELINE" | sed 's/.txt/.pdf/g')"; done But this produces one "error" -- ...
6
votes
5answers
20k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
16
votes
2answers
12k views

How do I remove certain lines (using line numbers) in a file?

There are specific lines that I want to remove from a file. Let's say it's line 20-37 and then line 45. How would I do that without specifying the content of those lines?
15
votes
3answers
16k views

How to find unmatched brackets in a text file?

Today I learned that I can use perl -c filename to find unmatched curly brackets {} in arbitrary files, not necessarily Perl scripts. The problem is, it doesn't work with other types of brackets () [] ...
8
votes
4answers
5k views

Efficient in-place header removing for large files using sed?

The commands below may takes minutes depends on the file size. Is there any more effient method? sed -i 1d large_file
7
votes
5answers
5k views

Compare two files with first column and remove duplicate row from 2nd file in shell script

I will ask my question with an example. I have 2 files: File #1: 118D FC300_R5_TP FX.B 32775 2112 6 2038 6 2112 0 118E FC300_R5_TP FX.B 32775 2136 7 ...
14
votes
10answers
3k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
11
votes
4answers
47k views

Finding text between two specific characters or strings

Say I have lines like this: *[234]* *[23]* *[1453]* where * represents any string (except a string of the form [number]). How can I parse these lines with a command line utility and extract the ...
7
votes
6answers
2k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
6
votes
2answers
6k views

How do I insert a space every four characters in a long line?

I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...
4
votes
4answers
16k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
12
votes
2answers
26k views

How do I delete the first n lines of an ascii file using shell commands?

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so ...
10
votes
2answers
1k views

Where has the trailing newline char gone from my command substitution?

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 ...
9
votes
3answers
3k views

Process last line first using awk

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally. The ...
8
votes
1answer
386 views

Is there a way to make perl -i not clobber symlinks?

A friend of mine points out that if you do: perl -pi.bak -e 's/foo/bar/' somefile when "somefile" is actually a symlink, perl does just what the docs say it will do: It does this by renaming ...