6

The question indicates my preference to use emacs, but the overriding issue is that I want to be able to do a normal text search and somehow see/copy-paste the byte-offset of the matched text.

To be clear, by byte-offset, I do not mean emac's point value, which shows the number of characters from the start of the buffer, eg. in UTF-16LE, point considers \x0d\x00\x0a\x00 as 1 character, whereas I'm interested in it as 4 bytes.

Any other editor (or viewer) which presents this basic information while displaying the text in a "normally" readable and searchable fashion is worthwile.

Even a hex view with a synchronized normal-text view would be okay, but a typical Hex-dump viewer/editor is not what I'm after, as they (typically) only display ASCII chars, and I haven't found a FOSS Hex-dump viewer/editor which can perform a simple text-mode search for non ASCII UTF-8 or for any UTF-16 strings.

I'm primarily concerned with legibility and search-ability of the text, so a "normal" Hex dump program is only a fallback (which I'm already using).

Improve this question

4 Answers 4

Reset to default
6

First of all, in case you don't know about it, Emacs has hexl-find-file which opens up a file in hex editing mode. I know that it's not what you asked for, but if you're already using one, and you're comfortable with Emacs, then it's good to know about it for future needs.

Second, for this kind of "raw" editing of a file (which I tend to do often), find-file-literally is really great. It does what you'd expect it to do, and pretends to be a pre-unicode version of itself and open the file with escapes showing up for non-ascii characters (and control chars etc). This is likely to do what you want, though it does have the obvious disadvantage of not being able to actually read the text if you have a lot of non-ascii content.

So going down further into primitive support, there's the enable-multibyte-characters variable and the set-buffer-multibyte function that is used to toggle it. The nice thing about this is that it changes the buffer presentation dynamically -- for example, try this:

(defun my-multi-toggle ()
  (interactive)
  (set-buffer-multibyte (not enable-multibyte-characters)))
(global-set-key (kbd "C-~") 'my-multi-toggle)

and you now have a key that toggles the raw mode dynamically. It also has the nice property of leaving the cursor in the same place. But this raw mode shows you the internal representation (which looks like UTF-8) and not whatever the file happens to be using as its encoding. It should be possible to do what you're talking about with some hack (for example, using find-file-literally on an open file will ask you about revisiting it, but that resets the location and reloads the file too) -- but it sounds like the above is already fine. (That is, my guess is that you're trying to edit some text field in an otherwise binary file...)

Improve this answer
4
  • 1
    Your answer is not quite on track to what I'm looking for, but it is certainly of value. I love that toggling ability, and I'll definitely be using it! However the main thing I want is a simple display (preferably in the mode-line (or mini-buffer) of the byte-offset /count of the current cursor/point position (from the start of the buffer).. The idea of using a full hex view is only an option because such displays typically show the byte-offsets... but such a hex-view would need to display (or at least, search for)**all** text in a readable form (which is what you tapped into).
    – Peter.O
    Jun 1, 2012 at 5:35
  • BTW: Although your multibyte toggle is great, it is a bit funky in how it displays the underlying \octal value: In a UTF-16 file it displays the values in UTF-8 forma!. BTW, can that octal view be switched to display Hex values?.. Thanks for your clear and useful answer...
    – Peter.O
    Jun 1, 2012 at 5:35
  • (1) Displaying the position is easy, but it will work only if you're willing to open files in a literal mode. (2) Re the multibyte toggle: see the emphasized "But" in my answer. (3) You can change how things get displayed, but it's not an easy+quick thing to do (and it makes me think that you really want a hex editor -- did you try that hexl-mode?).
    – Eli Barzilay
    Jun 1, 2012 at 17:58
  • I have used hexl mode. I've been aware of it for a while now. I tend to use hexview instead (for viewing) as it doesn't upset the undo buffer as hexl does (because it edits I assume).. thanks for your help..
    – Peter.O
    Jun 1, 2012 at 22:00
2

I've managed to throw together an elisp script to show the byte offset at point (poff) (via a shortcut-key).

Its presentation is currently very rough but it works fine for UTF-16LE/CR-LF (tested at start of file and end of file. UTF-16 was the format that got me onto this theme, and has actually been the mimplest to work with..

UTF-8 will be a bit trickier. as it will need some file I/O...

Here is the script. 

(defun poff-zap ()
  "Get the byte offset of point - A prototye, tested minimally only with UTF-16LE" 
  (interactive)

  (let ((linect (- (line-number-at-pos) 1)) ;; line count to point
        (choncl (- (point) (point-at-bol))) ;; characters to point on current line
        (chrpnl 0) ;; chars per newline
        (bytpch 0) ;; bytes per char
        (bytpnl 0) ;; bytes per newline
        (offset 0) ;; the byte offset   
        (coding  (car (split-string (symbol-name buffer-file-coding-system) "-")))
        (format (cadr (split-string (symbol-name buffer-file-coding-system) "-"))))

    (case (coding-system-eol-type buffer-file-coding-system)
      ('0 (setq chrpnl 1)) ;; unix 
      ('1 (setq chrpnl 2)) ;; dos
      ('2 (setq chrpnl 1)) ;; mac
      (t))

    (if (> chrpnl 0) 
        (cond
        ((string= "utf" coding) 
            (cond
            ((string= "8" format) 
                (progn
                (setq bytpch -1)
                ;; need to do an actual byte count
                ;;   using a UTF-8 parser
                ;; ...plus a BOM check(?)
            ))
            ((or 
             (string= "16" format)  
             (string= "16le" format)) 
                (progn 
                (setq bytpch 2)
                (if (= 2 chrpnl) (setq offset linect))
                (setq offset (+ offset (point)))
                (setq offset (* offset bytpch))
            ))
            (t)))
        (t)))

     (message (concat 
      "poff-zap: " (number-to-string bytpch)  " bytes-per-char\n" 
      "          " (number-to-string chrpnl)  " chars-per-newliner\n"
      "          " (number-to-string bytpnl)  " bytes-per-newliner\n"
      "          " (number-to-string (point)) " point-emacs\n"
      "          " (number-to-string offset)  " offset poff-zap\n"
      "          " (symbol-name buffer-file-coding-system) ))  
))
(global-set-key (kbd "C-#") 'poff-zap)
Improve this answer
7
  • 3
    Since Emacs 23, Emacs uses UTF-8 internally. For an UTF-8 file, all you need is (position-bytes (point)).
    – Gilles 'SO- stop being evil'
    Jun 2, 2012 at 0:19
  • This cannot work in the general case, since you need to deal with encodings that don't have fixed width for all bytes. For a concrete example, try (insert #x1D11E) in some utf-16 file to insert a "G CLEF" character, and you'll see that it takes four bytes. The bottom line is that for a proper answer you need to be able to map the contents of the (decoded) buffer to the raw contents of the file on the disk, and I don't think that Emacs keeps that after the text was decoded. If you know that you only need to deal with 2-byte/char then that's hackable, but not what you asked for...
    – Eli Barzilay
    Jun 2, 2012 at 4:16
  • Oh, and BTW, my guess is that making up the missing information requires reading the raw contents of the file and find the current point in it -- an exercise that is expensive enough to be impractical to hack into the modeline. (Not impossible though, if you really want to go that way...)
    – Eli Barzilay
    Jun 2, 2012 at 4:18
  • @Gilles, great! thanks!.. That solves UTF-8. Now a general comment about the above script for UTF-16.. It doesn't handle Surrogate Pairs well, as each pair is considered by emacs as a single character (as it should do). It does however work with the BMP (Basic Multilingual Plane) including U+200B ZERO WIDTH SPACE... so, it seems that UTF-16 is the one that needs parsing, but I'll defer doing that until I encounter my first Astral Plane characters..
    – Peter.O
    Jun 2, 2012 at 5:36
  • @Eli Barzilay.. yes, exactly... You wrote your comment as I was writing mine.. So, I'll need to forego the mode-line (for Astral Plane vaues of UTF-16), but it will suit occasional light usage, which is basically what I intended it for... I could do a one-time scan, on visiting a file, and have the routine start up hexl-mode or hexview instead when the file contains surrogate-pairs (or, my fallback-fallback.. use WinHex in wine, which has got WYSIWIG text seach, ie. no need to specially handle \x00 chars for ASCII text. WinHex works, but I'd much prefer to use a *nix tool).
    – Peter.O
    Jun 2, 2012 at 5:54
2

Sounds like you want position-bytes. You can see the byte offset of the cursor ('point' in Emacs terminology):

M-: (1- (position-bytes (point)))

Note that position-bytes is 1-indexed, hence the 1-. You can wrap this in a convenience command:

(defun wh/byte-offset-at-point ()
  "Report the byte offset (0-indexed) in the file
corresponding to the position of point."
  (interactive)
  (message "byte offset: %d" (1- (position-bytes (point)))))
Improve this answer
0

Vim

In vim, g^G displays at the bottom of the screen something like:

Col 1 of 3; Line 2 of 2; Word 2 of 2; Char 5 of 8; Byte 7 of 10

To have the byte offset of the cursor always in the status line, add %o to the statusline option:

:set statusline+=\ %o

(you may need to throw in a :set laststatus=2 for the status line to always be visible even when the screen is not split. To have the offset displayed on the right size, put %o on the right of a %= in statusline).

See also :go 123 or 123go to position the cursor nearest the 123rd byte.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .