2

Until now I use pyenv to install python versions under a user name pydev in directory /home/pydev. I use the Debian package direnv to link the project directory with a python version installed using pyenv. I use pip-tools to store the custom python environment for each project in the project directory.

Edit - The rationale for isolated development environments is given in this link

https://pwal.ch/posts/2019-11-10-sane-python-environment-2020-part-1-isolation/

Edit - The author of the link above developed his own isolated python environment tools here (I don't want to install one python version per project so I use Pyenv and pip-tools to link the python version with the env library):

https://github.com/pwalch/lonesnake

I am authoring python applications for use in development, potential patent disclosure, and hopefully to launch in production on a Linux Debian server. The Filesystem Hierarchy Standard (FHS) specifies storage of data in /srv and software applications in /opt.

https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/Linux-Filesystem-Hierarchy.html#opt

/opt/pythonapp01/code0000

/srv/pythonapp01/eod-data

The application I am working on now uses end-of-day data in csv format that does not update in development but would update every night in production. The python code would transform the csv to other csv formats for storage as files and transform those modified csv files into file formats for serving Highcharts stockcharts.

My question is how to organize directories under Linux Filesystem Hierarchy? How to store files for development, to support a possible patent disclosure, and to put the resulting python application(s) into production on a Linux Debian server? Should development and documentation files be located in a user folder? Should eod data go into /srv/pythonapp01 and code go into /opt/pythonapp01/code0000? Should pyenv and pip-tools installed environment also go into the application directory under /opt in production? I have read that applications should avoid using the system python in production and I have never setup a python app in production so it would help to get some guidance early in the development phase.

4
  • If your program is intended to be installed on a debian server, then make a debian package. And use the standard system directories - /etc, /bin or /usr/bin, /lib or /usr/lib, /usr/share, /var and so on. /opt and /srv are for third-party applications, usually - but not always (e.g. maptool is available as a .deb package but installs into /opt) - installed as a tar file or by compiling the source, without apt or dpkg.
    – cas
    Commented Feb 8, 2023 at 2:55
  • BTW, the pythonista attitude of avoiding system python/libraries/etc and despising the OS is complete bullshit - it's an evolution of the unfortunately common programmer idiocy of believing that "running important stuff out of my personal home directory with my weird and idiosyncratic setup is perfectly fine and normal". It isn't, it's disaster in waiting. Python devs need to learn that the OS their code runs on ISN'T a hostile thing to be avoided, worked around, and subverted. It's a tool to use, an environment you need to integrate your code into in order to take full advantage of.
    – cas
    Commented Feb 8, 2023 at 3:04
  • 1
    I added a link to Part 1 of the articles (four part series) that convinced me to use isolated python versions and isolated virtual environments during development. The main rationale is to minimize the frequency of events that break your code during development and then reduce the effort to remedy whatever events broke the code. I think an update of the server can impact system python, and that can break code in production, so the idea is to isolate the application python from the system python. But I have not used python in production only on my local raspberry pi and Debian desktop box. Commented Feb 8, 2023 at 17:30
  • Yeah, I get why it makes things easier with the way python is now but if they stopped breaking compatibility with every minor point release, it wouldn't be necessary. That was what made me give up on python. Incompatibility between python 2 & 3 was fair enough - although the switch to 3 dragged on forever (and technically still isn't over as there's still lots of python2 stuff that hasn't been ported), but breaking compatibility between 3.1 & 3.2, 3.2 & 3.3, ..., 3.10 & 3.11 is completely unacceptable - but to the python devs and community, breaking stuff all the time is OK - "This is fine."
    – cas
    Commented Feb 9, 2023 at 0:18

1 Answer 1

1

This is several questions, which I abstract to three:

  1. How do I organize Python source code and Documentation for development?

  2. How do I package and deploy a Python app to Linux?

  3. Where do I store data that's generated daily?

Organize Development and Documentation Files

Use a source code/version control system, probably the modern version control tool, git, with GitLab or GitHub or similar git hosting service for safe, shared remote storage.

So you will have a repository, a special directory named for your project or application.

Within your repository, the arrangement of the documents and source code will depend on what you're building. I like to keep documentation in .md or .rst files in a docs directory. It's customary to have a README.md that describes the project/repo.

Kenneth Reitz created an example structure for a Python module, which is here: https://github.com/navdeep-G/samplemod. It's not fully minimal for a beginner, so you can skip the parts you don't understand, but it's a good starting point. https://docs.python-guide.org/writing/structure/ describes the structure.

I'm not 100% sure that pyenv works with the deployment method I describe below because I use venv or virtualenv, but all you really need for deployment is a requirements.txt listing the packages you need to install in a new python environment. You can create a requirements.txt while within your working environment by typing pip freeze to all the packages you have installed. Redirect that to a requirements.txt and save it.

How Do I Package and Deploy a Python Application?

For "single file" deployment to Linux and Unix (Mac) based systems, pex works well and is fairly simple. https://pypi.org/project/pex/. There's a short, clear tutorial at https://medium.com/ovni/pex-python-executables-c0ea39cee7f1.

There are many choices other than pex, and you might start at https://packaging.python.org/en/latest/ if you want to investigate them.

To explicitly answer your question "Should pyenv and pip-tools installed environment also go into the application directory under /opt in production?", you won't need to install those because the pex files are self-contained.

Where Do I Store Daily Data?

You asked "Should eod data go into /srv/pythonapp01 and code go into /opt/pythonapp01/code0000?"

The data location seems OK, but your application will look more like /opt/bin/pythonapp01.pex (or whatever binary/executable/packaged format you chose) in /opt/bin (and then you'll need to add /opt/bin to the PATH of any user who wants to run it). As an example, the homebrew package for Mac OS installs to /opt/homebrew/bin. /usr/local/bin is another popular install location.

There are many schemes you might use to save End-of-Day data, for example by day-of-year number 1 through 365, or by date, with hierarchical directories. I save daily temperature readings in, e.g., /mnt/data/2023/02/06/ and create a new daily directory every day, a new monthly directory every month, and a new yearly directory every year.

Keep backups!

I hope this is helpful!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .