2

I have 62 GB RAM in my Linux server and my Python code is trying to load 20 GB file in the memory. However, it is crashing and throwing MemoryError.

I am not sure why it would be like this?

I know I can load the file incrementally and this error can be mitigated.

But my question is more fundamental, why python cannot load this file into memory even if so much memory is available.

When I type ulimit, i shows unlimited.

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256797
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Then why my process cannot access memory.

free -mg:

              total        used        free      shared  buff/cache   available
Mem:             62           1          52           0           8          61
Swap:            31           0          31
def read_from_file(file_name):
    with open(file_name, mode='rt', encoding='utf-8') as reader:
        text = reader.read()
        return text
10
  • 1
    How are you checking that you have enough memory? Presumably, there are other things running on the server, all of which will also be consuming memory.
    – terdon
    Jan 22, 2021 at 18:30
  • free -m was showing 61GB free
    – Exploring
    Jan 22, 2021 at 18:41
  • That doesn't make sense. You mean there was nothing at all running? Can you show us the output of free -m you mention? Were you maybe looking at the "total" column instead of the "available" column?
    – terdon
    Jan 22, 2021 at 18:42
  • I rebooted the machine. So only the OS is running.
    – Exploring
    Jan 22, 2021 at 18:43
  • 3
    First thing I'd check is how Python represents what you read internally. If each character gets converted to a unicode word, you'd already need 4*20GB = 80 GB for this file. (But it may equally well be another reason). Try to mmap it instead (but then you yourself will have to deal with the encoding).
    – dirkt
    Jan 22, 2021 at 20:50

1 Answer 1

3

Reading a text file with 20G text data (let's assume encoded as utf-8) would still first require to read the file content as binary chunks before the data can be decoded as Python string.

Assume we have a text file called utf-8_text.txt whose content are the following 4 utf-8 characters without newline:

aä猫👾

The following script should clear things up a bit.

import codecs
import os
import sys
import unicodedata

# some unicode characters from different codepoint ranges
unicode_characters = dict(
    character="a",
    umlaut="ä",
    cat="猫",
    monster="👾"
)

# add "utf-16", "utf-32" if you want to test with other encodings
encodings = ["utf-8"]

for key, character in unicode_characters.items():
    character_mul = character * 1000
    print(
        f"{key}: {repr(character)} "
        f"(len={len(character)} {'U+%04X' % ord(character)} "
        f"{unicodedata.category(character)} "
        f"{unicodedata.name(character)}) "
        f"/ {repr(character)}*1000 (len={len(character_mul)})"
    )
    for encoding in encodings:
        encoded_character = character.encode(encoding=encoding)
        print(
            f"\t{encoding} of {repr(character)}: "
            f"{encoded_character} "
            f"(len={len(encoded_character)} "
            f"sizeof={sys.getsizeof(encoded_character)})"
        )
        encoded_character_mul = character_mul.encode(encoding=encoding)
        print(
            f"\t{encoding} of {repr(character)}*1000: "
            f"{encoded_character_mul[:12]}... "
            f"(len={len(encoded_character_mul)} "
            f"sizeof={sys.getsizeof(encoded_character_mul)})"
        )
        print()

# python text needs to be decoded after it was read,
# this causes memory exhaustion
for encoding in encodings:
    with open(f"{encoding}_test.txt", mode='rb') as reader:
        stat_result = os.fstat(reader.fileno())
        binary = reader.read()
        binary_mul = binary * 1000
        text = codecs.decode(binary, encoding=encoding)
        text_mul = text * 1000
        print(
            f"{encoding}_test.txt: {binary} = {repr(text)}\n"
            f"\tfile_size: {stat_result.st_size}\n"
            f"\t{repr(text)} len: {len(binary)} as bytes / {len(text)} as str\n"
            f"\t{repr(text)} sizeof: {sys.getsizeof(binary)} as bytes / "
            f"{sys.getsizeof(text)} as str\n"
        )
        print(
            f"\t1000*{repr(text)} len: {len(binary_mul)} as bytes / "
            f"{len(text_mul)} as str\n"
            f"\t1000*{repr(text)} sizeof: {sys.getsizeof(binary_mul)} as bytes / "
            f"{sys.getsizeof(text_mul)} as str"
        )

When we run this script, it will first print some memory usage information about each individual unicode character including a copy of that character repeated 1000 times. Then we see the memory usage when reading utf-8_text.txt in binary mode.

character: 'a' (len=1 U+0061 Ll LATIN SMALL LETTER A) / 'a'*1000 (len=1000)
        utf-8 of 'a': b'a' (len=1 sizeof=34)
        utf-8 of 'a'*1000: b'aaaaaaaaaaaa'... (len=1000 sizeof=1033)

umlaut: 'ä' (len=1 U+00E4 Ll LATIN SMALL LETTER A WITH DIAERESIS) / 'ä'*1000 (len=1000)
        utf-8 of 'ä': b'\xc3\xa4' (len=2 sizeof=35)
        utf-8 of 'ä'*1000: b'\xc3\xa4\xc3\xa4\xc3\xa4\xc3\xa4\xc3\xa4\xc3\xa4'... (len=2000 sizeof=2033)

cat: '猫' (len=1 U+732B Lo CJK UNIFIED IDEOGRAPH-732B) / '猫'*1000 (len=1000)
        utf-8 of '猫': b'\xe7\x8c\xab' (len=3 sizeof=36)
        utf-8 of '猫'*1000: b'\xe7\x8c\xab\xe7\x8c\xab\xe7\x8c\xab\xe7\x8c\xab'... (len=3000 sizeof=3033)

monster: '👾' (len=1 U+1F47E So ALIEN MONSTER) / '👾'*1000 (len=1000)
        utf-8 of '👾': b'\xf0\x9f\x91\xbe' (len=4 sizeof=37)
        utf-8 of '👾'*1000: b'\xf0\x9f\x91\xbe\xf0\x9f\x91\xbe\xf0\x9f\x91\xbe'... (len=4000 sizeof=4033)

utf-8_test.txt: b'a\xc3\xa4\xe7\x8c\xab\xf0\x9f\x91\xbe' = 'aä猫👾'
        file_size: 10
        'aä猫👾' len: 10 as bytes / 4 as str
        'aä猫👾' sizeof: 43 as bytes / 92 as str

        1000*'aä猫👾' len: 10000 as bytes / 4000 as str
        1000*'aä猫👾' sizeof: 10033 as bytes / 16076 as str

The function sys.getsizeof reports the size of a Python object in memory, and as we can see, a simple utf-8 encoded file with 4 unicode characters may have just 10 bytes on disk, but eventually a total of 92 bytes in memory as str, quite inflated.

This of course is not the real increase for the above use case and probably is caused by some Python book keeping or string interning. More realistically, we have a larger input (like aforementioned 20G file), and to address that we report memory usage of a 4000 unicode character version of the file (1000 repetitions of aä猫👾).

Now 1000 times aä猫👾 translates to 10,033 bytes in memory as raw bytes (i.e., 10,000 bytes on disk as aä猫👾 in utf-8 has 10 bytes + book-keeping) and 16,076 bytes in memory as str.

Generalizing, 20G of raw utf-8 bytes on disk increases roughly by 60% memory in this example to 32G as str, which would not fill up the memory but both of them together must be held in memory plus some decoder book-keeping when decoding raw bytes to utf-8 str which is probably the cause for using up all memory for the machines specs given above.

Inspecting the implementation of io.TextIOWrapper.read() seems to confirm my guess.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .