Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

18
votes
5answers
23k views

Shell: How to read the bytes of a binary file and print as hexadecimal?

In shell, how can I read the bytes of a binary file I have, and print the output as hexadecimal numbers?
8
votes
4answers
19k views

How to use bash script to read binary file content?

I want to read a character and then a fixed length of string (the string is not null terminated in the file, and its length is given by the preceding character). How can I do this in a bash script? ...
4
votes
2answers
4k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
4
votes
1answer
4k views

How to numerical sort by last column?

I have this input: sdkxyosl 1 safkls 2 asdf--asdfasxy_asd 5 dkd8k jasd 29 sdi44sw 43 asasd afsdfs 10 rklyasd 4 I need this output: sdi44sw 43 dkd8k jasd 29 asasd afsdfs 10 asdf--asdfasxy_asd 5 ...
4
votes
1answer
285 views

sed : insert sth before subsequent lines that begin the same, but are not the same

I have a LaTeX file with one glossary entry per line: ... \newglossaryentry{ajahn}{name=Ajahn,description={\textit{(Thai)} From the Pali \textit{achariya}, a Buddhist monk's preceptor: `teacher'; ...
1
vote
3answers
336 views

Switch from grep to sed

I have a files students.txt, with lines of the form: Surname, Forename: Day.Month.Year: Degree For example: Smith, John: 15.01.1986: MSc IT Taylor, Susan: 04.05.1987: MSc IT Thomas, Steve: ...
2
votes
3answers
3k views

How can I loop through lines of a file and find files matching each line?

In a BASH shell, I would like to take the lines of a file (eg pattern.txt) and find the files on my system whose names contain the patterns in each line of my file. So, I have the following for loop ...
7
votes
4answers
332 views

Getting text from last marker to EOF in POSIX.2

I have a text with marker lines like: aaa --- bbb --- ccc I need to get a text from the last marker (not inclusive) to EOF. In this case it will be ccc Is there an elegant way within POSIX.2? ...
19
votes
3answers
18k views

How to find unmatched brackets in a text file?

Today I learned that I can use perl -c filename to find unmatched curly brackets {} in arbitrary files, not necessarily Perl scripts. The problem is, it doesn't work with other types of brackets () [] ...
27
votes
3answers
13k views

How can I make iconv replace the input file with the converted output?

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT. Since adding the -o parameter ( in my experience ) actually writes ...
6
votes
5answers
22k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
1
vote
1answer
881 views

Keep newlines in output of cut

Let's imagine we have a file: foo bar foo1 bar1 I want to cut every second word in every line, so I want file to look like this: foo foo1 I am trying to use cut -d ' ' -f -1 but what I get is: ...
3
votes
4answers
268 views

Remove empty configuration section

Files like ~/.config/vlc/vlcrc are 99% junk if you want to version control only the configuration options. I've got a script to remove the comments, but there's a ton of empty configuration sections ...
2
votes
2answers
351 views

non-case-sensitive sed - OpenWrt

the input would be like this in a file: <sCripT and the parsing looks like this: sed -i "s/<script//g" SOMETHING.html the "/gi" is not good, because OpenWrt doesn't knows it (busybox..): ...
2
votes
3answers
5k views

Printing unique lines

Is there some better solution for printing unique lines other than a combination of sort and uniq?
2
votes
4answers
11k views

How can I find and replace with a new line?

I have a CSV delimited by commas and I want to delimit it by newlines instead. Input: a, b, c Output: a b c I've written Java parsers that do this stuff, but couldn't this be done with vim or ...
5
votes
7answers
6k views

get numeric ASCII value for a character

I'm trying to write a shell script which asks for an ASCII character in the range A-Z or a-z and returns its equivalent numerical value. For example, the output might look like the following: ...
43
votes
8answers
61k views

Looping through files with spaces in the names?

I wrote the following script to diff the outputs of two directores with all the same files in them as such: #!/bin/bash for file in `find . -name "*.csv"` do echo "file = $file"; diff ...
0
votes
2answers
445 views

Get last set of non-empty lines

I'm running a cron job which should get only the last result of iostat -d 1 2. This requires some parsing: What is the simplest way to get either the last set of non-empty lines from standard input to ...
8
votes
5answers
3k views

`highlight` command

Is there a command which can be used to easily highlight (with colors, bold text or inverted back-/foreground) specific strings in stdin while passing the entire string to stdout? I'm thinking of ...
10
votes
6answers
7k views

How can I print lines from file backwards (without using “tac”)?

I have a simple question (I think). I want to print lines from a file backwards without using tac command. Is there any other solution to do such thing with bash?
2
votes
1answer
123 views

The cursor to a match, edit, the cursor to the next match, edit — like in a directory?

Algorithmic blackout: problem: find the files that match problem: stack/buffer the matches move the cursor to the next file of the stack and thereby to the location of the match repeat step 2 ...
8
votes
1answer
420 views

Is there a way to make perl -i not clobber symlinks?

A friend of mine points out that if you do: perl -pi.bak -e 's/foo/bar/' somefile when "somefile" is actually a symlink, perl does just what the docs say it will do: It does this by renaming ...
7
votes
5answers
605 views

How intelligent are unix utilities?

Unix utilities like sort, find, grep, diff et al are very handy to perform quick tasks, sometimes without writing any code at all. I wanted to know what algorithms do they internally use and how do ...
16
votes
2answers
4k views

Horizontal file concatenation

Is there a Linux command like cat that joins files with the same number of lines horizontally?
5
votes
1answer
781 views

Is there a search utility which can binary split search a large sorted (sequential) text file?

Yes, I know it sounds odd. Sequential and Binary-splits don't mix.. That is unless the sequence is the byte offset within the file itself... I've scrambled together a binary split search in bash ...
12
votes
6answers
8k views

Looking at logs from bottom upwards

Is there a linux command or some way to look at logs from bottom up rather than from top towards bottom. I know about tail -n <number of lines>, but is there something that I can actually scroll ...
3
votes
2answers
2k views

sed or tr one-liner to delete all numeric digits

So, I have this textfile, and it consists of mostly alphanumeric characters. It's a standard document. But since I copied it and pasted it from a PDF, there are page numbers in there. I don't much ...
7
votes
2answers
2k views

How can I cut a large file in place? [duplicate]

I have a very big text file, about 80GB, and I need to cut a part form it that lies between two given lines. The part I need is not big, and I have not enough space left on the hard drive to do things ...
2
votes
1answer
1k views

How can I add a line to multiple files in a directory tree structure?

Adding a line to multiple files in the same structure is not very hard, but I am stumped on how to do it over a possibly large directory tree structure. So lets say we have your project directory, ...
8
votes
2answers
1k views

Parsing XML's , JSON's and newer data file formats in UNIX using command line utilities

Unix environment has some excellent tools for parsing text in various forms. However off late the data is not in the regular set of formats(CSV,TSV,Record based or some other delimiter based) it used ...
3
votes
2answers
730 views

How is the octal 2-byte output calculated from od

I'm struggling to work out what the octal 2-byte output from the od command is. I understand the octal output (-b flag) but the octal 2-byte is a mystery to me (-o) Can someone shed some light on how ...
7
votes
4answers
6k views

Is it possible in bash, to start reading a file from an arbitary byte count offset?

I want to locate a date which is somewhere in an 8 GB log (text). Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem ...
1
vote
1answer
135 views

svn command to see a tally of my changes

Is there an SVN command that I can run to see a tally of all of my contributions? Something like: added 1800 deleted 15
7
votes
2answers
750 views

What governs the limits of shell brace expansion?

In this example I refer to the expansion of a sequence of integers, but perhaps(?) the limits would be relevant to all aspects of brace expansion.. This more general view is also of interest to me. ...
5
votes
3answers
1k views

Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format

I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...
14
votes
6answers
28k views

How to cut part from log file?

I have an 8 Gb log file (Rails production log). I need to cut it between some dates (lines). Which command could I use to do this?
7
votes
2answers
146 views

How does one find and replace text in all open files with jed?

How does one find and replace text in all open files with jed?
2
votes
1answer
339 views

How does one find and replace text in all open files with LibreOffice?

How does one find and replace text in all open files with LibreOffice?
2
votes
2answers
1k views

How does one find and replace text in all open files with gedit?

How does one find and replace text in all open files with gedit?
4
votes
1answer
2k views

How does one find and replace text in all open files with geany?

How does one find and replace text in all open files with geany?
2
votes
3answers
3k views

Extracting tokens from a line of text

Using bash scripting and grep/awk/sed, how can I split a line matching a known pattern with a single character delimiter into an array, e.g. convert token1;token2;token3;token4 into a[0] = token1 … ...
11
votes
4answers
4k views

command to layout tab separated list nicely

Sometimes, I'm getting as an input tab separated list, which is not quite aligned, for instance var1 var2 var3 var_with_long_name_which_ruins_alignment var2 var3 Is there an easy way to render ...
20
votes
11answers
10k views

Is there a robust command line tool for processing csv files?

I work with CSV files and sometimes need to quickly check the contents of a row or column from the command line. In many cases cut, head, tail, and friends will do the job; however, cut cannot easily ...
9
votes
2answers
3k views

How to remove multiple lines per occurrence in a file?

Say I have this 857835 line file, containing stuff like this: a1 rubbish1 rubbish2 rubbish3 rubbish4 a1 rubbish5 rubbish6 rubbish7 rubbish8 And I wish to remove all occurences of a1 and the next ...
14
votes
10answers
3k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
3
votes
2answers
1k views

vim - but with scrolling, clickable text, and copy-paste

I like vim. I think touching mice is dirty. But sometimes when my hand is already on a mouse, it sure would be nice to be able to scroll the text or click a location in the text and have the cursor ...
23
votes
5answers
13k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
12
votes
2answers
11k views

Parse html on Linux

I have a couple of hundred html source code files. I need to extract the contents of a particular <div> element from each of these file so I'm going to write a script to loop through each file. ...
8
votes
3answers
8k views

How do I insert a space every four characters in a long line?

I've got a long line that I want to insert a space every 4 characters, on a single lone line of solid text to make it easier to read, what's the simplest way to do this? also I should be able to input ...