The tag has no wiki summary.

learn more… | top users | synonyms

34
votes
5answers
14k views

How to remove duplicate lines inside a text file?

A huge (up to 2 GiB) text file of mine contains about 100 exact duplicates of every line in it (useless in my case, as the file is a CSV-like data table). What I need is to remove all the repetitions ...
12
votes
6answers
3k views

Which extension to use for text files? (Unix/Linux)

I noticed that I can read text files without an extension .txt just fine. How come? Should I save these files with or without the .txt extension? Also, what about .ini files? I usually use them like ...
12
votes
11answers
8k views

Grep huge number of patterns from huge file

I have a file that's growing about 200,000 lines a day, and it is all formed with blocks of three lines as such: 1358726575123 # key Joseph Brunner # name carpenter # job ...
11
votes
2answers
5k views

How can I find out how many lines a text file contains without viewing it?

How can I find how many lines a text file contains without opening the file in an editor or a viewer application? Is there a handy Unix console command to see the number?
11
votes
4answers
346 views

How can I delete all english lines from a textfile?

I have this text file: 714 01:11:22,267 --> 01:11:27,731 Auch wenn noch viele Generationen auf einen Wechsel hoffen, Even if it takes many generations hoping for a change, 715 01:11:27,732 --> ...
10
votes
3answers
2k views

Is there some sort of PDF to text -converter?

I need PDF files to text so I can search over them in bulk from commandline. Is there some converter for Ubuntu, OBSD or similar distro? Perhaps related post, OCR with ubuntu here.
7
votes
4answers
2k views

creating a sequence of numbers, one per line in a file

Is there a way to create out of thin air, a file that is a sequence of numbers, starting at a given number, one per line? something like magic_command start 100 lines 5 > b.txt and then, b.txt ...
7
votes
2answers
360 views

Does OS X have its own line format?

Obviously there are at least two newline types: DOS and Unix. But does OS X have its own plaintext 'format'? I opened a text file in nano and was surprised to see: [ Read 26793 lines (Converted from ...
7
votes
2answers
2k views

Finding all “Non-Binary” files

Is it possible to use the find command to find all the "non-binary" files in a directory? Here's the problem I'm trying to solve. I've getting an archive of files from a windows user. This archive ...
6
votes
3answers
372 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
5
votes
5answers
972 views

Script to remove spaces and lowercase in file names

I am trying to write a script that will replace spaces with "-" and make all letters lower case for all files in the current directory. for x in 'ls' do if [ ! -f $x ]; then ...
5
votes
4answers
478 views

Look for English word in a file via terminal

How to find and print out English words contained in a file via linux command line?
5
votes
1answer
206 views

How to remove all instances of a particular symbol from a text file?

I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going ...
5
votes
2answers
444 views

How to convert UTF-8 txt files to all uppercase in bash?

I have some UTF-8 .txt files which I would like to convert to all uppercase. If it was just ASCII, I could use: tr [:lower:] [:upper:] But since I'm working with diacritics and stuff, it doesn't ...
5
votes
1answer
158 views

Synonyms (from a thesaurus) in Emacs

I use Emacs 23 on Debian sid. I can check the spelling of a word in Emacs with M-$. How can I obtain synonyms (using a thesaurus?) for a word in Emacs? None of the following were useful: one, two, ...
4
votes
3answers
737 views

Colorized `cat` for source and script files? [duplicate]

Possible Duplicate: How can I display syntax highlighting on a file? Is there any application like cat, but with color output? I'd like to view my files with colors like in any IDE's ...
4
votes
2answers
854 views

Character count in Unix wc command

When I give wc command on a file with contents given below, it gives the number of characters as 30. Does it include the end of file character? Since including the space and newline there are only 29 ...
4
votes
1answer
130 views

Why isn't there a djvu2text?

This has been bothering me for a while. .djvu files are often searchable, in my experience. So somewhere in there, all the text is just one big string, right? But there's no djvu2text program. And ...
4
votes
5answers
1k views

extract fields from “tail -f” of a syslog stream

Example line from syslog file: Aug 1 10:25:50 10.10.10.1 id=firewall sn=XXXX time="2012-08-01 14:35:18 UTC" fw=x.x.x.x pri=6 c=1024 m=537 msg="Connection Closed" f=11 n=195273698 ...
4
votes
3answers
450 views

How to skip file in sed if it contains regex?

I currently use the following simplified command to remove trailing whitespace and add a newline at end of file where needed: find . -type f -exec sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' {} \+ As ...
4
votes
1answer
877 views

Replace variable read from a text file (so they can be interpreted)

I tried several stuff with not luck so maybe one of you experts can help me (I hope!). I want to store in a text file some select statement that contain variables like a date: $ cat res.txt select ...
4
votes
1answer
105 views

Is there a Unix command that searches for similar strings, based mostly on how they sound when spoken?

I have a file of names, and I want to search within it, not caring too much about whether I have spelled the name ( that I am searching for ) correctly. I know that grep has quite a bit of ...
3
votes
3answers
138 views

Transforming text to tabular form

I have a text file with the following structure: aaa bbb ccc ddd eee fff 1 2 3 4 5 6 1.1 1.2 1.3 1.4 1.5 1.6 ggg hhh iii jjj kkk lll 7 8 9 10 11 12 2.1 2.2 2.3 2.4 2.5 2.6 and I want the ...
3
votes
5answers
103 views

How to prepend a particular text file contents to every text file in a directory and its subdirectories?

I hate to clutter my source code files with licensing/attribution comments but some times it is a requirement. So having dozens of source code files (organized in a tree of subdirectories) written I ...
3
votes
2answers
2k views

AWK-Find maximum value in a row & print with header

I need to read a file, find the maximum value in each row, and print the ID of the row, the column the maximum value came from (sno#), and the value from the associated lc# column. If the maximum ...
3
votes
1answer
62 views

Read the files in shell script doesn't print out anything?

I am trying to read a file from shell script line by line. Below is my file content - hello=world And below is my shell script that I have written - #!/bin/bash while read -r line; do ...
3
votes
2answers
167 views

How to know if a text file is a subset of another

I am trying to find a way to predict if a text file is a subset of another.. For example: foo bar is a subset of foo bar pluto While: foo pluto and foo bar are not one a subset of each ...
3
votes
2answers
66 views

Get “Text Content” of Files

I was wondering if there is a tool to scrape text content of a file. Basically I would like to put in a pdf, word document, compressed file or anything else and get the text it contains out. There ...
3
votes
1answer
259 views

Is there something like a lorem ipsum generator?

Is there something like a 'lorem ipsum generator'? I know it exists in Latex, and even in LibreOffice, but I'm looking for a terminal command. I would like to type something like loremipsum 10 ...
3
votes
1answer
129 views

Is the last new line on fstab important?

What happens if the last line of fstab is not terminated by a newline? Why do people get a warning when that last line is not terminated by a newline?
3
votes
1answer
218 views

How to fix xdg-mime for text formats

My xdg-mime makes mistakes. It recognizes an .html file containing Java code as a C++ file. I don't quite know the internal mechanism of xdg (eg. how it determines filetype). And I want to know how ...
3
votes
0answers
108 views

Mint 13 bug that prevents right clicking on desktop icons and numerous other symptoms

I use Mint 13 with Gnome Shell 3.4. For the last month or two I have had an issue that randomly occurs, and once it does only logging out or rebooting fixes it. I randomly have a problem that causes ...
2
votes
3answers
2k views

What is the purpose of -e in sed command?

I can't find any documentation about the sed -e switch, for simple replace, do I need it? e.g. sed 's/foo/bar/' VS sed -e 's/foo/bar/'
2
votes
4answers
205 views

How to multiply a data file with another index data file?

I want your help in multiplying my data file with another index data file. Both data files have the same number of rows. But the data file have more than 1000 columns while the index data has only one ...
2
votes
4answers
721 views

how to word wrap, align text like the output of man?

what is the command that word wraps and justifies a text file so that the output looks like that of a man page: All of these system calls are used to wait for state changes in a child ...
2
votes
2answers
390 views

view file containing ISO-8859 text, CRLF line terminators, and escape sequences

I have stumbled upon a strange text file in an archive of an old dos game downloaded from some abandonware site. The text file is named WHY_NOT.TXT. I assume the file is some sort of .NFO file from ...
2
votes
3answers
513 views

How can I turn ugly output into pretty, useful information?

How can I turn this ugly output into pretty, useful data? The output: /* ---------- TA#box#AbC_p ---------- */ insert_job: TA#box#AbC_p job_type: a #owner: bob permission: gx date_conditions: 1 ...
2
votes
1answer
714 views

Keyboard Shortcut To Send Text Strings To Program

How can I configure a shortcut key to send a text string to the current program? The purpose is to type common entries quicker (email address, street address, phone number, username, favorite quote, ...
2
votes
2answers
1k views

How can I find and install pacman-color or cope?

I'm using Arch-Linux and I'm trying to install something that colors the text in the terminal (xterm), but not necessarily just for xterm. It would be nice to find something for text without ...
2
votes
2answers
538 views

How can I get entries for a given time period from a 400MB+ log file?

I have a 400MB+ Tomcat log file (catalina.out). How can I pull out entries for a given time period?
2
votes
1answer
312 views

Search and Filter Text on large log files

I use the tail, head and grep commands to search log files. Most of the time the combination of these 3 commands, in addition to using pipe, gets the job done. However, I have this one log that many ...
2
votes
1answer
227 views

Compare all text files in a directory, and sort by similarity

In Unix, is there any way to compare every text file in a directory to every other text file in the directory, and then sort each pair of files by similarity (using the diff utility)? There are ...
2
votes
1answer
147 views

Expanding text format in VI

The following code is taken from the superfish module for websites: (function($){$.fn.hoverIntent=function(f,g){var ...
1
vote
2answers
2k views

How to select, copy and paste some columns from different files and create one file from them?

I want to create and organize data in a file from a number of files by selecting parts of the columns of the given files. I have more than 10 file to copy the second, third and forth columns of each ...
1
vote
2answers
101 views

How to bury an invisible mark into lines of text?

How can I bury an invisible mark into random lines of text? Such a mark has to be there, though it will be invisible to someone reading that text printed out on the console. I want to identify those ...
1
vote
3answers
235 views

Open a text file in a terminal and auto-refresh it whenever it is changed

How do I open a text file in a terminal with instant auto-refresh every time it is changed? I've looked at vim with :set autoread, but it requires some elementary input (such as a keypress inside ...
1
vote
1answer
321 views

Remove first instance of lines with duplicate field value

I have sha1 summed all the image files on my storage server and put the results in a text file in the form of: sha1sum filename I've sorted the file and removed all all unique sha1sum entries. So ...
1
vote
2answers
710 views

Should I end my text/script files with a newline? [duplicate]

Possible Duplicate: What's the point in adding a new line to the end of a file? What is considered better behaviour: to leave the last letter of the last line to be the last byte of a ...
1
vote
3answers
922 views

How to change encoding from Non-ISO extended-ASCII text, with CRLF line terminators to UTF-8?

I have a txt file : $ file -i x.txt x.txt: text/plain; charset=unknown-8bit $ file x.txt x.txt: Non-ISO extended-ASCII text, with CRLF line terminators And there are some characters that are ...
1
vote
1answer
191 views

Selecting/highlighting Text Problem

I am using Backtrack 5. As it is based on Ubuntu 10.04 I decided to ask my question to here: I'm having problems selecting text with my mouse. For example when I try to rename a folder, sometimes I ...