The tag has no usage guidance.

learn more… | top users | synonyms

3
votes
2answers
195 views

Merging files by rows

Is it possible to copy the whole rows of File1 in a new File3 following the instruction given by File2 by using a simple bash script using sed or awk? File1: /*two or more columns*/ AC 456324 DC ...
3
votes
2answers
230 views

Remove rows from tab delimited files based on a common column with another file

I am having trouble processing a tab delimited file based on the common column Column_4 with another file. One file is likely to be very small (less than 100 rows), the second one, however, will be ...
3
votes
2answers
354 views

How do I insert new columns into a .csv file?

Example Data: "Monday","12345","Friday" In the above file, I want to add a new column in between 12345 and Friday. How can I do that ?
3
votes
2answers
198 views

Compare 2 delimited files and output differences

I have asked similar questions here a couple times in the past with great success, but now my needs have slightly changed and I am struggling to get the exact output I am looking for. I would like to ...
3
votes
2answers
1k views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
3
votes
2answers
158 views

How can I compare two files based on the value of the first column?

I want to compare two files based on the value of their first column: file1 aaa 23 3 bbb 56 6 ccc 45 2 file2 bbb m a aaa a m ccc m m The output file should be: aaa a m bbb m a ccc m m ...
3
votes
2answers
2k views

Merging multiple lines based on column 1

I have a file like below.. abc, 12345 def, text and nos ghi, something else jkl, words and numbers abc, 56345 def, text and nos ghi, something else jkl, words and numbers abc, 15475 def, ...
3
votes
2answers
2k views

Cut command not extracting fields properly in aligned columns

I have a text file in which I must cut the fields 3,4,5 and 8: 219 432 4567 Harrison Joel M 4540 Accountant 09-12-1985 219 433 4587 Mitchell Barbara C 4541 Admin Asst ...
3
votes
1answer
184 views

Sort only on the second column [duplicate]

I'm trying to sort a file where typically (not always) the lines are whatever:[A-Ba-b0-9_]: values where fields are separated with :. Some lines do not follow this pattern, I'm not interested in ...
3
votes
1answer
48 views

Why should you avoid “stringently columnar” input formats?

Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition, had this to say at the time [McIlroy78]: (ii) Expect the output of every program to become the input to ...
3
votes
2answers
1k views

Select certain column of of each file, paste to a new file

I have 20 tab delimited files with the same number of rows. I want to select every 4th column of each file, pasted together to a new file. In the end, the new file will have 20 columns with each ...
3
votes
3answers
648 views

How can I get the first column of a 300GB txt file?

let me first explain my problem in detail. It's actually pretty simple. I have a huge .txt file, 300GB to be more preciese, and I would like to put all the distinct strings from the first column, that ...
3
votes
3answers
3k views

How to reorder a large number of columns?

I'm looking for a pipeable one-liner to reorder a large number of columns (where manually entering the column numbers in e.g. an awk command such as awk '{print $3,$2,$1}' is not feasible). The order ...
3
votes
1answer
310 views

zsh and COLUMNS

I cannot understand the behavior of the $COLUMNS variable under zsh. Here's an example of the sort of behavior I'm referring to. The script $HOME/cols.sh is the following: echo "COLUMNS=$COLUMNS" ...
2
votes
3answers
234 views

How do I reorder or rearrange the output below

Here in my INPUT: 0015:"IDPTS3O" 0091:"ODMBS1I" 0015:"IDPTS3O" 0091:"ODMBS1I" 0015:"IDPTS3O" 0091:"ODMBS1I" 0015:"IDPTS3O" 0034:"ADTSS1I" 0015:"IDPTS3O" 0034:"ADTSS1I" ...
2
votes
3answers
1k views

changing column separators in a file

I have a large file, that I need to process and after writing some scripts that don't seem to work properly, I have discovered that a small subset of the lines in the file are actually space separated ...
2
votes
2answers
2k views

Bash: append columns to table

If you want to append a column (not a line) to a table, and the column has the same number of elements for sure, how can you do it? Given a set of files, which keeps changing each day, you want to ...
2
votes
5answers
6k views

Sum the values in a column except the header

I have a file as given below -------------------------------------------------------------- Name_Customer Item_Purchased Item_Amount Credit ...
2
votes
3answers
1k views

How does this awk align the spacing?

Can someone please tell me what does this do? cat infile| awk ' { for(i=3;i<=NF;i++){ $2=$2" "$i } printf "%-4s %s\n", $1,$2 ; }' > outfile.txt infile contains: 1 John Smith 2 ...
2
votes
2answers
311 views

Create an ASCII art table from tabular data

Given perhaps comma- or tab-delimited input, I'd like to present a series of appropriately padded columns to stdout, so I can easily scan columnar information which would otherwise present rather ...
2
votes
2answers
179 views

“Fill columns before rows” option in unix column command

According to man column: -x Fill columns before filling rows. This option doesn't seem to do anything. Any idea how to use it?
2
votes
2answers
2k views

Align columns in ASCII file

I have a text file that looks like his: #c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 4599 995,274 2523,658 ...
2
votes
4answers
104 views

Fill empty lines in specific column with values

I have data of the following format: 2342 1 6473 2 7463 2 3647 2734 1 9343 The lines that do not have a value in the 2nd column are empty in that spot (i.e. there is no space, tab etc.). I am ...
2
votes
2answers
84 views

Append a column

I've trawled through here and a few other places to try and find an answer to this, but it seems so stupidly simple that no one has answered it. All I need to do is add a single value in a new column ...
2
votes
4answers
2k views

Extract lines from a text file with one column equal to the maximum

I have a text file (with header line) called file.txt. I am trying to extract the lines equal to the maximum value of a specific column (I do not know what the max value is): ID t1 q1 t2 q2 q3 1 f ...
2
votes
4answers
369 views

Adding values of the column that have the same data ID?

I want to prepare a uniform daily data from a mixed daily and sub-daily data by adding the sub-daily values and producing a uniform daily data. My sample input and desired out put data looks like ...
2
votes
1answer
4k views

i/o format with printf in bash-shell

I am trying to output the results of my shell script using printf. To keep the results aligned in column, I am using tab as: printf "%s\t %s\t %s\t %s\t %s\t %s\t %s\n" $i $est $var_m $jk $mst $var_M ...
2
votes
2answers
73 views

How do I echo a line with linebreak at the end at window border? [duplicate]

I would like to echo a list all in one line, TAB separated (like ls does with files in one folder) for i in one two some_are_very_long_stuff b c; do echo $i; done will print one line per word: one ...
2
votes
2answers
1k views

Print some fields of /etc/passwd

I have a question concerning the usage of wild cards in Linux. Here I have some lines from a long list of lines in the /etc/passwd -file. root:x:0:0:root:/root:/bin/bash ...
2
votes
1answer
78 views

Produce aligned columns in a shell script

In shell how do I have an even length of spaces by the length of words on my menu? I've tried to get the length of the longest character string and then create each column length by the longest ...
2
votes
3answers
2k views

Trouble in adding a column with awk

I'm trying to add a column to a csv file by using: awk -F "," '{print $0"new_col"}' file.csv but I got : new_col8,0.405,0.101 new_col4,0.374,0.100 new_col23,0.342,0.100 new_col130,0.298,0.100 ...
2
votes
2answers
161 views

Extract a column from whitespace-separated fields that are not aligned

How can we extract a column from output in which whitespace is not in a regular format? For example, when we type ps -ef, it shows all the process, process id's.... etc. How can we get only the last ...
2
votes
2answers
2k views

Data pivot with awk

I'm trying to pivot a file using awk. This is an example of my input File: VarName;TimeString;VarValue;Validity;Time_ms A;23.11.201215:03:53;1;1;41236627696,7593 ...
2
votes
2answers
52 views

Output distribution to a maximal number of columns

I have a bash script foo that provides me with many short lines of output. Think ls -1. To get a more compact presentation, I would like to distribute the output over multiple columns, like ls -C. I ...
2
votes
3answers
354 views

Extracting part from a file column

I want to extract part from a log file column which is like this: xx.xxx.xx.xx#59796: Edit: This is the actual log line: Jan 10 17:38:11 server named[747]: client 21x.x0.x8x.xxx#40649: view ...
2
votes
1answer
264 views

Combining files with 4 columns each, into 4 files for each column [closed]

So I have a bunch of text files which look like this: T1 r1 a b c d r2 d e f g ... T2 r1 h i j k r2 l m n o ... And I want to make FOUR different text files in the following ...
2
votes
2answers
907 views

How do you truncate all columns in a file to a specific length?

Given columned text in a file that is delimited by a certain character, how do you truncate the width of either a specific column(s) or all columns? The use case is that I am reading some log files ...
2
votes
3answers
64 views

How to add a new line for a missing entry between two fields

I have this file. As you can see in column1, entries TEST01N03, TEST01N06, and TEST01N18 are missing (total of 18 entries). Is there a way to check the fields of this column and fill in the missing ...
2
votes
1answer
196 views

Compare the values of a column with all the values in another column

I have 2 input files. Each line of File1 should be compared to each and every line of File2. Logic is: If Column1 of File1 does not match Column1 (all the values under it) of File2, then print the ...
2
votes
1answer
47 views

How do you implement totals from usage output generated by a program in a script so that they carry on sanely through repeated execution?

I have a script with a pipeline where the data is throttled by a program. Every 20 minutes, a throughput status is sent on stderr, and I simply redirect this to a file: command | cstream -t -512k -T ...
2
votes
1answer
249 views

text processing-perl

I am a biologist. I would like help of computer experts to help in text file editing. Here is what the input file looks like: ##dsfsd2 ##sdf-sdf sasg 5.6.3 gi34_ex Gen CDS 161 317 . ...
1
vote
3answers
960 views

removing spaces from first column

My input file has positions in first column with different number of spaces (or no space) 16504 16516 1650811 16520 1651 16524 16516111 16528 165204 16532 ...
1
vote
4answers
95 views

lines to columns with awk

I have to following sample output: <HARDWARE> <NAME>WIN1</NAME> <OS>Windows 7</OS> <IP>1.2.3.4</IP> <DOMAIN>contoso.com</DOMAIN> ...
1
vote
3answers
177 views

Find records present in file 1 also in both file 2 and file 3

I have three files file1.txt, file2.txt, file3.txt and they are of same format. I want to select records from file1.txt which are present in both file2.txt and file3.txt based on column 2 and column ...
1
vote
3answers
270 views

ps output differs when run from command line and from script

I have to create list of PID's of the running processes. Here is the script ps echo "fields are " ps | cut -d' ' -f1 ps is showing PID TTY TIME CMD 2168 pts/4 00:00:00 sh 2177 ...
1
vote
3answers
295 views

sort CSV by number of column in rows?

I have multiple CSV files with many records. The total number of rows should be 134. I have many files, and each rows has its own number of columns (from 15 to 200). I need to sort them according to ...
1
vote
1answer
186 views

Why doesn't this unexpand seem to work?

It's my understanding that unexpand can replace multiple spaces with tabs. I have a file, x.x with contents 1 2 3 4 5 and I type expand x.x but the output is still 1 2 3 4 5 and ...
1
vote
2answers
374 views

Filter Large Data Sets on Dates with Command Line

I have a huge data sets that consists of tab separated values that are stored in rows. An example row looks like this: Dec 4 14:37:36.381651 algorc1 [27751:l@27932]/error: [] - [T0000A124M5] ...
1
vote
3answers
8k views

Shell script read separately columns from file

When we have file which is a kind of list like : Mary 34 George 45 John 56 Josh 29 using the awk command $1 refers to the first column and $2 to the second column.But once we write a shell script ...
1
vote
3answers
643 views

Find matches for the 5th columns in a pipe delim files

File 1: Connect|20130320000023|UTC|PPP|test3@test3.co.uk|test1@test1.co.uk|0BCBE578| File 2: Connect|20130320000023|UTC|PPP|test1@test1.co.uk|test1@test1.co.uk|0BCBE578| ...