I am trying to have the file command detect some Windows text files that were never meant to be classified by file... The best choice seems to use regex to match the line content, but I cannot find a single example of its use (the commonality of the keywords 'file', 'magic', and 'regex' does not help in a google-centric world). The man page does not help.

Furthermore, I cannot get the ^ $ to work.

Both files start with

Project Units: <stuff>
Units & Scale - <stuff>
<blank line>

Next line is a header that starts 4a) Object point ID,Photo #, 4b) Id,Name,

My attempt at a magic rules for this is:

0  string Project\040Units:
>2 regex ^Object\040point\040ID,Photo\040#, PhotoModeler 2D export table

0  string Project\040Units:
>2 regex ^Id,Name, PhotoModeler 3D export table

i.e. match 'Project Units:' on the first line, then do a regex attempt to maximum 2+1 lines. Anchor regex to beginning of line for speed.

This is on Ubuntu 14.04, file-5.14.

Example of file type 1 (first 10 lines only):

Project Units: meters
Units & Scale - Active, Translate - Active, Rotate - Active

Object Point ID,Photo #,X (pixels),Y (pixels),Residual X,Residual Y,Residual Vector,Mark Type,Layer,Material,Tagged
2,1,1429.187065,1456.427823,-0.164541,0.182824,0.245964,LSM Circular,Default,White, 
2,2,666.583514,1126.807078,-0.168174,0.109780,0.200833,LSM Circular,Default,White, 
2,3,716.264669,1196.788962,0.152059,0.082258,0.172882,LSM Circular,Default,White, 
2,4,674.145595,442.969428,0.119315,-0.050084,0.129401,LSM Circular,Default,White, 
2,5,330.056929,836.292587,0.048372,-0.022235,0.053238,LSM Circular,Default,White, 
2,6,1147.101715,39.253316,0.475434,-0.189514,0.511814,LSM Circular,Default,White, 

Example of file type 2 (first 10 lines only):

Project Units: meters
Units & Scale - Active, Translate - Active, Rotate - Active

Id,Name,Photos (used),X (project units),Y (project units),Z (project units),X Precision,Y Precision,Z Precision,Precision Vector Length,Tightness (percent),Tightness (project units),Angle (deg.),Control Name,RMS Residual (pixels),Largest Residual (pixels),Photo Largest Residual,Material,Layer,Tagged,Type,Use In Processing,Frozen,#Constraints,Target Code,Target Bits,Ref. Check Tag,Photos (marked),Color (R),Color (G),Color (B)
2," ","1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",0.285721,1.143037,-0.000990,0.000044,0.000043,0.000075,0.000097,0.037511,0.000682,85.604862, ,0.261467,0.511814,6,White,Default, ,Regular,yes,no,0,n/a,n/a, ,"1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",255,255,255
3," ","1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",0.428622,1.143108,-0.000230,0.000044,0.000042,0.000074,0.000096,0.033814,0.000615,86.326354, ,0.222883,0.475602,6,White,Default, ,Regular,yes,no,0,n/a,n/a, ,"1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",255,255,255
4," ","1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",0.142979,1.143124,-0.000840,0.000045,0.000044,0.000078,0.000100,0.030045,0.000546,84.468461, ,0.239445,0.374918,16,White,Default, ,Regular,yes,no,0,n/a,n/a, ,"1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",255,255,255
5," ","1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",0.571353,1.143164,0.000784,0.000044,0.000042,0.000074,0.000096,0.027194,0.000494,86.593419, ,0.213540,0.430629,6,White,Default, ,Regular,yes,no,0,n/a,n/a, ,"1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",255,255,255
6," ","1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",0.000141,1.143101,-0.000885,0.000046,0.000045,0.000081,0.000103,0.035513,0.000646,82.937166, ,0.291437,0.465014,16,White,Default, ,Regular,yes,no,0,n/a,n/a, ,"1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",255,255,255
7," ","1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",0.714058,1.143134,0.000247,0.000044,0.000043,0.000075,0.000097,0.030057,0.000547,86.326626, ,0.221009,0.426056,6,White,Default, ,Regular,yes,no,0,n/a,n/a, ,"1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21",255,255,255
  • 1
    Please edit your post and include an example of the files you need to identify. – terdon Jun 22 '16 at 9:37
  • 1
    The caret needs to be escaped: \^ . See eg. the make tests. – JigglyNaga Jun 22 '16 at 11:54
  • @terdon: Added first 10 lines of example files. Solution was to escape the caret as posted below. :) – Niclas Börlin Jun 22 '16 at 13:05
  • 1
    Oh, cool, both an edit and an answer, thanks! It would be great if you remember to come back and accept the answer once the system lets you do so. – terdon Jun 22 '16 at 13:06

The file(1) manpage only tells you how to run the command. For a description of the magic patterns, see magic(5). However, the section on regex isn't especially detailed. A wide range of examples of its use can be found in the pattern files that come with it: https://github.com/file/file/tree/master/magic/Magdir

Your main problem was that the caret needs escaping: \^ for beginning-of-line, \\^ for a literal ^. I haven't worked out what special meaning the unescaped ^ has. Spaces can also be escaped, making the pattern slightly more readable.

You intend to restrict the match to a narrow line range. regex takes a /<length> option (after the word regex, not after the pattern), so that places a limit on where the search ends. If the length is followed by an l, it means lines instead of bytes. In my tests, /1l can only match an empty line -- a non-empty line, even when using the exact start offset, requires at least /2l.

For the start of the search, offset is interpreted as a byte count, even with regex. (Pre-version 5.19, the documentation suggests that it is interpreted as a "line count", but that statement was removed with no matching code change, so I doubt that it was accurate even before then.) You could use offset &0 to start the search from the end of the previous match, but that won't make a lot of difference when the previous match ended in the middle of the first line.

Furthermore, "beginning of line" also matches "beginning of search range" (ie. from offset), regardless of whether that was the start of a line in the file.

So to match things more strictly, you could use a full-line regexp on every line, and use offset &1 on the next match, to skip the previous newline, and be in the right place for the \^ to work as expected. This may be overkill for identifying your custom file types.

Finally, you don't need to repeat the common parts. The level of > indentation means that a pattern should be tried when previous patterns at the same level failed.

Joining that all together:

0       regex/2l        \^Project\ Units:.*$
>&1     regex/2l        \^Units\ &\ Scale.*$
>>&1    regex/1l        \^$
>>>&1   regex/2l        \^Object\ Point\ ID     Photo Modeler 2D export table   
>>>&1   regex/2l        \^Id,Name,Photos        Photo Modeler 3D export table

One solution was due to @JigglyNaga - escape the caret. Snippet below is now part of my .magic file.

0   string  Project\040Units:
>2  regex   \^Id,  PhotoModeler 3D export table

0   string  Project\040Units:
>2  regex   \^Object\040Point\040ID,  PhotoModeler 2D export table

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.