Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

11
votes
10answers
8k views

Decoding URL encoding (percent encoding)

I want to decode URL encoding, is there any built-in tool for doing this or could anyone provide me with a sed code that will do this? I did search a bit through unix.stackexchange.com and on the ...
2
votes
3answers
47 views

Undo letterspacing with sed

I have a source text file containing text where some words are l e t t e r s p a c e d like the word "letterspaced" in this question (i.e., there is a space character between the letters of the word. ...
1
vote
1answer
172 views

How to copy rows of different files and paste column-wise?

I want to create a new file by copying rows of each file and pasting columnwise according to their file name orders. Here is the sample input and required output files below. file1.txt 1234 5678 ...
0
votes
2answers
22 views

“head” only printing one line?

I've got a CSV file that's generated by saving as CSV from Excel. If I do "head" (or indeed "grep" or anything else) it only prints the first line: head -n 10 messy.csv 10,15,11,21 But if I open ...
-3
votes
1answer
26 views

Extract single information from a column in a file

I have a file with tab-separated fields in the following format: 2-micron 251 1523 R0010W . + SGD gene . ...
2
votes
4answers
52 views

Print alternating lines with conditional

Still struggling with messy logfiles. What I want is to find lines that first matching string X and then string Y. I then want to print them together. The problem is that, sometimes there is X, but no ...
7
votes
4answers
3k views

tail -f, but with line numbers

I'm trying to see how many times foo bar appears in /var/log/foo.log within an arbitrary amount of time on a remote server, but nothing that I've tried so far has worked. I've already got a timer ...
7
votes
4answers
271 views

How to grep-inverse-match and exclude “before” and “after” lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except ...
2
votes
1answer
21 views

Move up every other line to end of line containing character

I have ran 2 xpath queries piped together to retrieve certain information. However the results came out formatted on separate lines when I want them to be on the same line. So for example the lines ...
0
votes
0answers
14 views

I have two arrays roles added and roles deleted, want to merge them and write in this order in perl [migrated]

I have another scenario. My file has the following fields: Username | Roles | Type |date | abc|admin |added | 01072015 abc|developer |deleted |01072015 abc|deploy |added |01072015 xyz |admin |deleted ...
8
votes
2answers
1k views

Parsing XML, JSON, and newer data file formats in UNIX using command line utilities

The Unix environment has some excellent tools for parsing text in various forms. However, of late, the data is not in the traditional (historical) formats (CSV, TSV, record-based or some other ...
0
votes
3answers
54 views

How to sort/reformat text information in a file?

I have a file (called customers) with information that needs to be sorted; example: qweasd,asdqwd@asd.org,256.62394383 qweasdczasd,asdghjghqwd@assssd.org,256.62394383 ...
1
vote
4answers
102 views

Delete everything before / on every line

I am trying to delete certain that appears before / on every line. I have something like: testing.db.com/7fad416d-f2b3-4259-b98d-2449957a3123 testing.db.com/8a8589bf-49e3-4cd7-af15-6753067355c6 ...
4
votes
1answer
50 views

How can I repair a tar file where stderr got mixed with stdout?

I have a tar file that I'm trying to repair. The problem is that tar's stderr got mixed in with its stdout, so the file looks like this: % head android-1435613730.tar tar: removing leading '/' from ...
1
vote
3answers
91 views

Merge duplicated lines which have the same fist three filed

I have the below input (thousand lines). I want a sed command to merge duplicated lines with the same fist three filed but adds only different fields and delete N/A: D04005;4;279;0;0;SSM-4-1 ...
24
votes
5answers
15k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...
0
votes
1answer
54 views

What is the cause of the sed error?

In my grunt file I have the command like this "sed -i index 's/API_CONTEXT_URI/http:\\/\\/localhost:5557/g' www/index.js" This is not a file I have written; this has to come to me as a maintenance ...
3
votes
2answers
316 views

Extract only chinese characters

I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this? I tried: [The range of Chinese Unicode chars is 4E00 thru 9FFF (344 ...
4
votes
6answers
76 views

Join multiple files with two blank lines?

I'm trying to join Markdown files with two newlines between each file. I've tried using the following command: paste -sd '\n\n' file1.md file2.md This joins the files with a newline, but not two ...
4
votes
4answers
167 views

How can I convert two-valued text data to binary (bit-representation)

I have a text file with two (2) only possible characters (and maybe new lines \n). Example: ABBBAAAABBBBBABBABBBABBB (Size 24 bytes) How can I convert this to a binary file, meaning a bit ...
5
votes
5answers
488 views

Grouped sorting of continuous paragraphs (separated by blank line)?

I think I'm pretty experienced now in sorting by columns; however, I haven't found anything so far how to sort continuous rows. Supposing we have a text file that looks like this: (very simplified, ...
0
votes
1answer
44 views

How to extract particular fields from lines using shell commands?

File1: 27/07/2014 10.10.03 Breaker1 was closed by Mr. A.Basu Spectrum ui6dms 29/07/2014 04.50.53 Breaker1 was opened by Mr. D.Gupta Spectrum ui6dms and one note added 30/08/2014 03.20.01 EarthSwitch ...
4
votes
2answers
184 views

How to append the lines of a file to the end of the lines of other file?

File #1: I have foofoo You have foobar she/he has foo File #2: bar foobar barfoo Final: I have foofoobar You have foobarfoobar she/he has foobarfoo
-6
votes
1answer
64 views

Shell script to get common elements from two files [on hold]

My problem is as follows: I have two files in the following format: File 1: 1000 LSI Logic / Symbios Logic 0001 53c810 0021 53c1010 66MHz Ultra3 SCSI Adapter 1000 1000 ...
2
votes
6answers
639 views

Adding a character to every other text line

I have a huge (ca. 20G) text file which contains millions of passages (a.k.a. paragraphs) with headers. Headers and paragraphs are always one line each, e.g., Sunshine This is a sunny day. ...
69
votes
10answers
72k views

How to add a newline to the end of a file?

Using version control systems I get annoyed at the noise when the diff says No newline at end of file. So I was wondering: How to add a newline at the end of a file to get rid off those messages?
2
votes
2answers
34 views

Sort a list and separate item with commas instead of newlines

I want to list all the filesystems in a single comma-delimited list, in dictionary order. I'm using this command: cat /proc/filesystems | sed 's/nodev//' | sed 's/,$//' And my output looks like ...
7
votes
6answers
183 views

How to efficiently split up a large text file wihout splitting multiline records?

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
12
votes
7answers
9k views

Looking at logs from bottom upwards

Is there a linux command or some way to look at logs from bottom up rather than from top towards bottom. I know about tail -n <number of lines>, but is there something that I can actually scroll ...
2
votes
2answers
33 views

Workaround for busybox head that drops NUL from data

TL;DR: Latest busybox (1.23.2) provided head is buggy and will happily drop NUL characters in the provided data. Replacing the binaries is not an option. I'm using head -[nc] -X in my scripts, is ...
1
vote
2answers
36 views

Extract lines from a file using vim or teminal

I have the following in a file. I'd like to use Vim or terminal to extract title and its content like this. How can I achieve this? Topic 1 - Core: Algebra 1.1 Arithmetic sequences and series; sum ...
2
votes
1answer
33 views

Why should you avoid “stringently columnar” input formats?

Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition, had this to say at the time [McIlroy78]: (ii) Expect the output of every program to become the input to ...
2
votes
2answers
65 views

Compare 2 Files if it falls between a range of numbers

I have 2 files, File 1: 4 7 2 File 2: 1 5 Alpha 6 10 Bravo 11 15 Charlie 16 21 Delta 22 30 Echo I want to compare every entry in file 1 if it lies between col 1 and col 2 of file 2 (should be ...
1
vote
2answers
143 views

Another awk(ward) question - mapping substrings with multi-keys

I have two files that look like this: mapping.txt: 100178 L08/2015 100180 L08/2015 103822 J03/2012 103823 A06/2012 103823 J02/2012 103823 J03/2012 ... input.txt: 100180011|225| ...
-1
votes
2answers
23 views

To sed match and some text in newlines

You want to visualize maximum 10 marks but minimum 1 mark after the sequence 65 72 62 65 in a newline where the dummy data source is hexdump -v /usr/bin/telnet and the less is a viewer. Code hexdump ...
7
votes
7answers
6k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" ...
-3
votes
2answers
40 views

Transform text records using only a “for” loop

I have an input file that is text, consisting of lines that look like: 352|C|PAID|7036|VOICE|01-FEB-12 I need to split each line into fields at the | (vertical bar) characters and output each field ...
2
votes
3answers
73 views

Single record of a file getting splitted over multiple lines

We are using flat text files to store in teradata table after transformations from Informatica. The file contains 14 fields (~ separated). But as the records are getting scattered over multiple lines ...
-1
votes
3answers
75 views

sed - Remove lines that contain “hotmail” in the third field

I am trying to delete all emails that contain the word hotmail in the third field from my .txt file. I am currently trying to use the following but it only deletes some lines. I am not sure why sed ...
2
votes
3answers
68 views

Removing duplicates every X number of lines

Is it possible to remove only the AWD015F11 duplicate using Awk? I have a large file with one column. This happens every 24 rows. I have tried counting the rows in search for a pattern that I could ...
8
votes
4answers
5k views

Concatenate lines by first column by awk or sed

How can I use awk in the following situation? I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh). The file may be ...
6
votes
1answer
105 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
6
votes
2answers
744 views

BASH base conversion from decimal to hex

In Bash, how does one do base conversion from decimal to another base, especially hex. It seems easy to go the other way: $ echo $((16#55)) 85 With a web-search, I found a script that does the ...
1
vote
2answers
46 views

Using df and trying to awk a specific column

Among other problems with this script I am trying to make (this is the first unix/linux script I have ever made), I am trying to use df to get the % of disk space used and email it to the root. Our ...
7
votes
7answers
345 views

Removing numeric values in certain columns whilst keeping minus signs?

I put my hands up to being a complete beginner, so apologies if I'm doing this wrong. Say I have the following data frame that continues indefinitely horizontally and vertically with negative numbers ...
1
vote
2answers
76 views

Extracting information from entries in a LDIF file

I've a file that contains entries like this: ddn: cn=312548,ou=Entities,ou=Active,ou=Vault,o=abc member: cn=312548001,ou=Unnamed,ou=Identities,ou=Active,ou=Vault,o=abc member: ...
-1
votes
1answer
17 views

Matching 2 different files by first column

So I have 2 different files. file1.txt is: abcdefjeiireiiioe:data:otherdata rijirjeiwrieeoooe:datamore:otherdatamore file2.txt is: abcdefjeiireiiioe:data:otherotherdata ...
1
vote
1answer
51 views

Bash script to edit text in csv-format

As part of an inventory bash script I am checking SFP modules and adding them to a .csv-file. The format currently looks like this, with varying lengths: RO01;1000BaseSX SFP;1000BaseSX SFP ...
210
votes
5answers
75k views

Why is printf better than echo?

I have heard that printf is better than echo. I can recall only one instance from my experience where I had to use printf because echo didn't work for feeding some text into some program on RHEL 5.8 ...
2
votes
4answers
44 views

Sum of alternate values in a column using either sed or nawk

foo.txt: 1 10 11 2 20 22 3 30 32 4 40 42 5 50 52 6 60 62 7 70 72 8 80 82 9 90 92 10 100 110 Desired Out.txt : 1 10 11 2 20 22 3 ...