Which Linux directories should be used for python applications?

Question

Until now I use pyenv to install python versions under a user name pydev in directory /home/pydev. I use the Debian package direnv to link the project directory with a python version installed using pyenv. I use pip-tools to store the custom python environment for each project in the project directory.

Edit - The rationale for isolated development environments is given in this link

https://pwal.ch/posts/2019-11-10-sane-python-environment-2020-part-1-isolation/

Edit - The author of the link above developed his own isolated python environment tools here (I don't want to install one python version per project so I use Pyenv and pip-tools to link the python version with the env library):

https://github.com/pwalch/lonesnake

I am authoring python applications for use in development, potential patent disclosure, and hopefully to launch in production on a Linux Debian server. The Filesystem Hierarchy Standard (FHS) specifies storage of data in /srv and software applications in /opt.

https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/Linux-Filesystem-Hierarchy.html#opt

/opt/pythonapp01/code0000

/srv/pythonapp01/eod-data

The application I am working on now uses end-of-day data in csv format that does not update in development but would update every night in production. The python code would transform the csv to other csv formats for storage as files and transform those modified csv files into file formats for serving Highcharts stockcharts.

My question is how to organize directories under Linux Filesystem Hierarchy? How to store files for development, to support a possible patent disclosure, and to put the resulting python application(s) into production on a Linux Debian server? Should development and documentation files be located in a user folder? Should eod data go into /srv/pythonapp01 and code go into /opt/pythonapp01/code0000? Should pyenv and pip-tools installed environment also go into the application directory under /opt in production? I have read that applications should avoid using the system python in production and I have never setup a python app in production so it would help to get some guidance early in the development phase.

Answer 1

This is several questions, which I abstract to three:

1. How do I organize Python source code and Documentation for development?

2. How do I package and deploy a Python app to Linux?

3. Where do I store data that's generated daily?

Organize Development and Documentation Files

Use a source code/version control system, probably the modern version control tool, git, with GitLab or GitHub or similar git hosting service for safe, shared remote storage.

So you will have a repository, a special directory named for your project or application.

Within your repository, the arrangement of the documents and source code will depend on what you're building. I like to keep documentation in .md or .rst files in a docs directory. It's customary to have a README.md that describes the project/repo.

Kenneth Reitz created an example structure for a Python module, which is here: https://github.com/navdeep-G/samplemod. It's not fully minimal for a beginner, so you can skip the parts you don't understand, but it's a good starting point. https://docs.python-guide.org/writing/structure/ describes the structure.

I'm not 100% sure that pyenv works with the deployment method I describe below because I use venv or virtualenv, but all you really need for deployment is a requirements.txt listing the packages you need to install in a new python environment. You can create a requirements.txt while within your working environment by typing pip freeze to all the packages you have installed. Redirect that to a requirements.txt and save it.

How Do I Package and Deploy a Python Application?

For "single file" deployment to Linux and Unix (Mac) based systems, pex works well and is fairly simple. https://pypi.org/project/pex/. There's a short, clear tutorial at https://medium.com/ovni/pex-python-executables-c0ea39cee7f1.

There are many choices other than pex, and you might start at https://packaging.python.org/en/latest/ if you want to investigate them.

To explicitly answer your question "Should pyenv and pip-tools installed environment also go into the application directory under /opt in production?", you won't need to install those because the pex files are self-contained.

Where Do I Store Daily Data?

You asked "Should eod data go into /srv/pythonapp01 and code go into /opt/pythonapp01/code0000?"

The data location seems OK, but your application will look more like /opt/bin/pythonapp01.pex (or whatever binary/executable/packaged format you chose) in /opt/bin (and then you'll need to add /opt/bin to the PATH of any user who wants to run it). As an example, the homebrew package for Mac OS installs to /opt/homebrew/bin. /usr/local/bin is another popular install location.

There are many schemes you might use to save End-of-Day data, for example by day-of-year number 1 through 365, or by date, with hierarchical directories. I save daily temperature readings in, e.g., /mnt/data/2023/02/06/ and create a new daily directory every day, a new monthly directory every month, and a new yearly directory every year.

Keep backups!

I hope this is helpful!