0

I have a 1 line file ,contains non graphic characters and I want to split it based on a pattern. My pattern is \[0-9][0-9][0-9]; how can I do it? For example, I want to split this line:

\001abd \002pqr \003xyz

to:

\001abd 
\002pqr 
\003xyz

I'm using /bin/sh as default shell.

Another example input:

CHANGE^\039^OE@ MORE^\040^L^[[00m^OAC DEPOSIT TO WHICH ACCOUNT^N020^^\055^L^[(1^[[00m^OAA PAYMENT FROM WHICH ACCOUNT^N020

Desired output:

CHANGE^
\039^OE@ MORE^
\040^L^[[00m^OAC DEPOSIT TO WHICH ACCOUNT^N020^^
\055^L^[(1^[[00m^OAA PAYMENT FROM WHICH ACCOUNT^N020

The size of a one line file is 80KB and I'm using GNU sed version 4.2.1 and the OS is Red Hat Enterprise Linux Server release 6.5 (Santiago)

closed as unclear what you're asking by G-Man, Anthon, Archemar, don_crissti, chaos Oct 11 '15 at 12:05

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    is that your real data because that is easily split by spaces? – cas Oct 10 '15 at 5:41
  • your example output does not represent your input split according to your pattern. – mikeserv Oct 10 '15 at 6:08
  • 2
    It would probably be informative to see the first ten lines or so of output from od -c run on a file with the second sample output (the all shouting one in the comments). For example: od -c file | sed 10q. Add that to your question so you can format it. – Jonathan Leffler Oct 10 '15 at 14:14
  • 1
    Please edit your question and add an exact sample of the input and an exact sample of the output without any semplifications, with the ambiguity introduced by the sample input / output in the question and what you later posted in the comment it's impossible to extabilish what you want exactly. – kos Oct 10 '15 at 14:35
  • 2
    But mostly when you use SE sites you give anyone a chance and post a sample of your input file on your question, and you don't mail single users your real input file. -1. (let me clarify that this is nothing against Jonathan Leffler, who was kind enough to help you and who clearly put a great effort into writing his answer). – kos Oct 11 '15 at 11:58
2

There was an initial problem statement, and various solutions were proposed. It turned out that those did not work because the actual data did not match the description.

Workable Answer

My pattern is: FS[0-9][0-9][0-9], (FS is 'field separator'); how use that in the sed script?

With Bash and BSD sed, you can use this (which is closely based on what John1024 proposed and uses the BSD and Mac OS X sed notation -E to enable extended regular expressions):

sed -E $'s/(.)(\x1C[[:digit:]]{3})/\\1\\\n\\2/g' file1

The $'…' notation is Bash's ANSI C quoting mechanism. The FS has byte value 28, hex 0x1C or octal 038. The doubled backslashes represent a backslash seen by sed; the \n preceded by \\ satisfies the sed manual which says (in the s/// section):

A line can be split by substituting a newline character into it. To specify a newline character in the replacement string, precede it with a backslash.

Check on what works with GNU sed.

I also observe that FS is sometimes coded as Control-Backslash (because Control-A has code 1, but A has code 65 = 64 + 1; backslash \ has code 92 = 64 + 28); this probably accounts for alwaystudent's confusion in the question.

Note that GNU sed uses -r to do what BSD does with -E; POSIX sed doesn't recognize either notation.

Investigational Background

I've been sent the file by email, and if what I got is accurate, then we need a different characterization of what's required.

Word count output:

$ wc file1
       1    8804   80106 file1
$

Here's the output from a hex dump:

$ odx file1 | sed 20q
0x0000: 33 1C 1C 1C 31 31 1C 30 30 31 0E 32 30 31 1C 30   3...11.001.201.0
0x0010: 30 32 0E 32 30 31 1C 30 30 33 0E 32 30 33 1C 30   02.201.003.203.0
0x0020: 30 34 24 20 1C 30 30 35 0E 30 30 32 1C 30 30 36   04$ .005.002.006
0x0030: 0E 30 30 32 1C 30 31 31 0C 1B 28 32 0F 45 40 20   .002.011..(2.E@ 
0x0040: 20 20 59 4F 55 52 20 43 41 52 44 20 49 53 20 4E     YOUR CARD IS N
0x0050: 4F 54 20 20 53 45 52 56 49 43 45 44 0F 46 40 20   OT  SERVICED.F@ 
0x0060: 20 20 20 20 20 20 20 20 20 42 59 20 20 54 48 49            BY  THI
0x0070: 53 20 41 54 4D 20 0F 47 40 20 20 20 20 20 50 4C   S ATM .G@     PL
0x0080: 45 41 53 45 20 54 41 4B 45 20 20 59 4F 55 52 20   EASE TAKE  YOUR 
0x0090: 43 41 52 44 1B 28 37 0F 49 40 20 20 20 20 20 20   CARD.(7.I@      
0x00A0: 20 5C 26 20 2D 28 23 58 3E 3D 20 5C 25 22 40 22    \& -(#X>= \%"@"
0x00B0: 20 41 22 20 0F 4A 40 20 20 20 20 20 20 20 30 57    A" .J@       0W
0x00C0: 5F 40 5B 3F 4A 58 20 2D 28 40 23 51 20 59 5F 22   _@[?JX -(@#Q Y_"
0x00D0: 20 0F 4B 40 20 20 30 3E 5F 40 22 3E 40 26 20 22    .K@  0>_@">@& "
0x00E0: 40 20 3E 5B 3D 20 20 2D 28 40 23 51 20 23 4D 47   @ >[=  -(@#Q #MG
0x00F0: 55 1B 28 32 1C 30 31 34 0C 1B 28 3E 0F 43 40 20   U.(2.014..(>.C@ 
0x0100: 20 20 45 53 50 2D 4C 49 4E 4B 2F 46 54 53 0F 45     ESP-LINK/FTS.E
0x0110: 40 20 20 20 20 20 20 20 41 54 4D 0F 47 40 20 4D   @       ATM.G@ M
0x0120: 41 52 4B 45 54 49 4E 47 20 4D 45 53 53 41 47 45   ARKETING MESSAGE
0x0130: 20 45 32 1C 30 31 35 0C 1C 30 31 38 0C 1C 30 32    E2.015..018..02
$

Here's the same data from od -c:

$ od -c file1 | sed 20q
0000000    3 034 034 034   1   1 034   0   0   1 016   2   0   1 034   0
0000020    0   2 016   2   0   1 034   0   0   3 016   2   0   3 034   0
0000040    0   4   $     034   0   0   5 016   0   0   2 034   0   0   6
0000060  016   0   0   2 034   0   1   1  \f 033   (   2 017   E   @    
0000100            Y   O   U   R       C   A   R   D       I   S       N
0000120    O   T           S   E   R   V   I   C   E   D 017   F   @    
0000140                                        B   Y           T   H   I
0000160    S       A   T   M     017   G   @                       P   L
0000200    E   A   S   E       T   A   K   E           Y   O   U   R    
0000220    C   A   R   D 033   (   7 017   I   @                        
0000240        \   &       -   (   #   X   >   =       \   %   "   @   "
0000260        A   "     017   J   @                               0   W
0000300    _   @   [   ?   J   X       -   (   @   #   Q       Y   _   "
0000320      017   K   @           0   >   _   @   "   >   @   &       "
0000340    @       >   [   =           -   (   @   #   Q       #   M   G
0000360    U 033   (   2 034   0   1   4  \f 033   (   > 017   C   @    
0000400            E   S   P   -   L   I   N   K   /   F   T   S 017   E
0000420    @                               A   T   M 017   G   @       M
0000440    A   R   K   E   T   I   N   G       M   E   S   S   A   G   E
0000460        E   2 034   0   1   5  \f 034   0   1   8  \f 034   0   2
$

And here's a character frequency analysis of the data:

  =   3:      1
  =  10:      1
  =  12:    648
  =  14:    883
  =  15:   3461
  =  27:   1384
  =  28:    722
  =  32:  15248
! =  33:    178
" =  34:   1236
# =  35:   1847
$ =  36:      2
% =  37:     44
& =  38:    945
' =  39:    197
( =  40:   2096
) =  41:   1434
* =  42:    695
+ =  43:     25
, =  44:    446
- =  45:   1431
. =  46:     92
/ =  47:     29
0 =  48:   2453
1 =  49:   1279
2 =  50:   1052
3 =  51:    758
4 =  52:    427
5 =  53:    565
6 =  54:    299
7 =  55:    862
8 =  56:    414
9 =  57:    423
: =  58:     78
; =  59:    330
< =  60:      3
= =  61:    313
> =  62:   1683
? =  63:     60
@ =  64:   3472
A =  65:   2268
B =  66:    791
C =  67:   2034
D =  68:   1480
E =  69:   2862
F =  70:    732
G =  71:    692
H =  72:    799
I =  73:   1202
J =  74:    360
K =  75:    358
L =  76:    963
M =  77:    823
N =  78:   1483
O =  79:   1726
P =  80:    588
Q =  81:    507
R =  82:   1411
S =  83:   1624
T =  84:   1905
U =  85:   1172
V =  86:    151
W =  87:    372
X =  88:   1063
Y =  89:    647
Z =  90:    758
[ =  91:   1026
\ =  92:    665
] =  93:    275
^ =  94:    397
_ =  95:   1179
a =  97:      1
c =  99:      1
d = 100:      1
m = 109:    240
o = 111:      2
p = 112:      2
q = 113:      4
r = 114:      2
s = 115:      2
t = 116:      4
u = 117:      1
w = 119:      1
y = 121:      1
z = 122:     15

The sum of the numbers in the last column is 80106, which agrees with wc.

As you can see, there is just one newline (code 10) and it appears at the very end of the file. There are very few lower-case letters, a lot of upper-case letters, a moderate number of backslashes, but (what you can't see from the data shown so far is that) none of the backslashes are followed by a digit. Note that there are no character codes outside the ASCII range (none with the high bit set), and the coverage of the ASCII range is not complete either.

I wrote a quick analysis program to see what characters follow a backslash:

#include <stdio.h>

int main(void)
{
    int c;
    int count[256] = { 0 };
    while ((c = getchar()) != EOF)
    {
        if (c == '\\')
        {
            c = getchar();
            count[c]++;
        }
    }

    for (int i = 0; i < 255; i++)
    {
        if (count[i] != 0)
            printf("%3d = %5d\n", i, count[i]);
    }

    return 0;
}

When run on the file, it produced:

 12 =     3
 14 =    58
 15 =     3
 27 =    25
 34 =    10
 35 =    51
 37 =    14
 38 =   126
 40 =     9
 44 =    51
 45 =    26
 47 =     2
 59 =    17
 62 =   118
 64 =    46
 65 =     2
 66 =     2
 67 =    17
 69 =     1
 71 =     4
 72 =     5
 74 =    15
 79 =     1
 81 =     9
 83 =     1
 85 =     5
 88 =    18
 90 =    12
 91 =     6
 95 =     8

The sum of the counts is 665, which agrees with the number of backslashes in the original character analysis.

The digits have codes 48..57; not one of the characters after a backslash is a digit.

This is why the various solutions shown fail — they never stood a chance because the data doesn't even begin to resemble what is described.

  • it seems sed solutions failed because of long line, is there tr solution for this? – alwaystudent Oct 11 '15 at 17:12
  • 1
    @alwaystudent No, it has nothing to do with line length. What Jonathan's detailed analysis found was that the pattern that you asked about, \[0-9][0-9][0-9], occurs nowhere in the file. That is why the sed solution did not break up the lines. – John1024 Oct 11 '15 at 18:59
  • @John1024 : you are right , i get it, now, so my pattern is: FS [0-9][0-9][0-9] , (FS: filed separator) , how use that in your sed script – alwaystudent Oct 12 '15 at 4:30
  • @alwaystudent: With Bash and BSD sed, you can use sed -E $'s/(.)(\x1C[[:digit:]]{3})/\\1\\\n\\2/g' file1. The $'…' notation is ANSI C quoting. The doubled backslashes represent a backslash seen by sed; the \n preceded by \` satisfies the sed` manual which says: A line can be split by substituting a newline character into it. To specify a newline character in the replacement string, precede it with a backslash. Check on what works with GNU sed (this was under the s/// entry). – Jonathan Leffler Oct 12 '15 at 5:12
  • 1
    @mikeserv — It has two purposes. (1) It shows that I was (unexpectedly) sent the actual data file, which then allowed me to show everyone else who was trying to answer what we were actually dealing with. (2) It gives an actual sed script that works directly on some Unix machines (those running BSD or Mac OS X) and that can readily be adapted to other machines (those with GNU sed on board), and which fixes the data as the OP wanted. It also points out one of the sources of confusion — the distinction between backslash and control-backslash. – Jonathan Leffler Oct 14 '15 at 0:27
2

Using sed:

$ cat file
\001abd \002pqr \003xyz
$ sed -E 's/(.)(\\[[:digit:]]{3})/\1\n\2/g' file
\001abd 
\002pqr 
\003xyz

Using the same sed command but your other data example:

$ cat file2
CHANGE^\039^OE@ MORE^\040^L^[[00m^OAC DEPOSIT TO WHICH ACCOUNT^N020^^\055^L^[(1^[[00m^OAA PAYMENT FROM WHICH ACCOUNT^N020
$ sed -E 's/(.)(\\[[:digit:]]{3})/\1\n\2/g' file2
CHANGE^
\039^OE@ MORE^
\040^L^[[00m^OAC DEPOSIT TO WHICH ACCOUNT^N020^^
\055^L^[(1^[[00m^OAA PAYMENT FROM WHICH ACCOUNT^N020

Update: splitting on FS-digit-digit-digit

The ASCII "file separator" (FS) character is hexadecimal 1C. Using GNU sed:

sed -E 's/(.)(\x1c[[:digit:]]{3})/\1\n\2/g'

To demonstrate this, let's create a test file:

$ echo $'One\x1c123Two\x1c456Three\x1c7none' >newfile

Now, let's run sed:

$ sed -E 's/(.)(\x1c[[:digit:]]{3})/\1\n\2/g' newfile
One
123Two
456Three7none

The lines are successfully split.

Discussion

On my terminal, as shown above, the FS characters are invisible. The do become visible when running less. For example, running less newfile results in the display:

One^\123Two^\456Three^\7none

Here we can see that the FS character is displayed as ^\. This is consistent with the second example input shown in the question:

CHANGE^\039^OE@ MORE^\040^L^[[00m^OAC DEPOSIT TO WHICH ACCOUNT^N020^^\055^L^[(1^[[00m^OAA PAYMENT FROM WHICH ACCOUNT^N020
  • i suspect that the real data isn't text, and those \039, \0E@ etc aren't literal characters but representations of control codes. \040 is probably space character. ^L is a form-feed. ^[[00m is part of some sort of ansi or vt100 code. in fact, it looks a lot like a script log. – cas Oct 10 '15 at 7:07
  • @John1024 it work for my example, but unfortunately failed for real -long-data so i think this script fail for long line, is there a adjustment for huge data? – alwaystudent Oct 10 '15 at 8:38
  • 1
    @alwaystudent: How long is the 'one line file'? Kilobytes, Megabytes, bigger? Which version of sed are you running (which o/s are you on — what's available on HP-UX is very different from what's available on Linux, BSD, Mac OS X). Please add such information to the question. – Jonathan Leffler Oct 10 '15 at 14:46
  • @alwaystudent If you are using GNU sed, large data needs no adjustment. As Jonathan Leffler asked, what OS and sed version are you using? How big is your "huge data"? Also, please do a hexdump, e.g. od -c file | sed 10q, so we can be sure what is actually in the file. – John1024 Oct 10 '15 at 15:54
  • 1
    Also note that when you ask a question, or when people are leaving comments, it is helpful if you stick around to answer such issues. People are expending effort to try and help you; the very least you can do is expend a little effort to help them help you. – Jonathan Leffler Oct 11 '15 at 6:42
-1

If you want to go from:

\001abc \002jkl \003xyz

...to...

\001abc
\002jkl
\003xyz

...then the most simple and performant solution is to do:

tr \  \\n <in >out

That, however, is not the same as splitting out your input on \newlines according to your \[0-9][0-9][0-9] pattern, which could be effected like:

sed 's/\\[0-9]\{3\}/&\n/g' <in >out

...and would result in...

\001
abc \002
jkl \003
xyz

...although I suppose you could mean you want to insert a \newline before each occurrence of \[0-9][0-9][0-9], in which case you could do:

sed 's/\\[0-9]\{3\}/\n&/g' <in >out

...to get...


\001abc 
\002jkl
\003xyz

...with a trailing space on the end of each line.

But both of those might have problems dealing with very long input lines. If your entire input file is only the one line, then we can reliably do:

{   tr '\\' \\n | 
    sed -e:t \
        -e'$!N;/\n[0-9]\{3\}/!s/\n/\\/;tt' \
        -e's/\n/&\\/;P;D'
}   <infile >outfile

The above command chain will transliterate all backslashes in input to \newline characters then pipe the results to sed which will recursively test the head of each input line for three digits. tr won't have any issues dealing with long input lines at all, and, by the time it is through with it, its output should contain at least many \newline characters as you need. If a \newline character is not immediately followed by three digits then it is simply replaced with a backslash, else if it is then a backslash is inserted between the \newline character and the three digits.

The results for your second example are:


CHANGE^
\039^OE@ MORE^
\040^L^[[00m^OAC DEPOSIT TO WHICH ACCOUNT^N020^^
\055^L^[(1^[[00m^OAA PAYMENT FROM WHICH ACCOUNT^N020

Not the answer you're looking for? Browse other questions tagged or ask your own question.