virtualenv & virtualenvwrapper
A Virtual Environment enables us to keep the dependencies required by different projects in separate places, by creating virtual Python environments.
In other words, virtualenv is a tool to create isolated Python environments. The virtualenv creates a folder which contains all the necessary executables to use the packages that a Python project would need.
So, each project can have its own dependencies, regardless of what dependencies every other project has.
The site packages (3rd party libraries) installed using easy_install or pip are typically placed in one of the directories pointed to by site.getsitepackages:
>>> import site >>> site.getsitepackages() ['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages'] >>>
$ python3 Python 3.5.2 (default, Jul 5 2016, 12:43:10) >>> import site >>> site.getsitepackages() ['/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.5/dist-packages']
Pip is a tool that fetched Python packages from the Python package Index and its mirrors. We use it to manage and install Python packages. It is similar to easy_install but pip was originally written to improve on easy_install. So, it has more features, the key feature being support for virtualenv.
To install it with pip:
$ sudo pip install virtualenv Password: Downloading/unpacking virtualenv Downloading virtualenv-13.0.1-py2.py3-none-any.whl (1.7MB): 1.7MB downloaded Installing collected packages: virtualenv Successfully installed virtualenv Cleaning up...
When we install a package from PyPI using the copy of pip that's created by the virtualenv tool, it will install the package into the site-packages directory inside the virtualenv directory. We can then use it in our program just as before.
We only need the virtualenv tool itself when we want to create a new environment. This is really simple. Start by changing directory into the root of our project directory, and then use the virtualenv command-line tool to create a new environment:
$ mkdir myproject $ cd myproject $ virtualenv env New python executable in env/bin/python Installing setuptools, pip, wheel...done.
Here, env is just the name of the directory we want to create our virtual environment inside. It's a common convention to call this directory env, and to put it inside our project directory (so, say we keep our code at ~/myproject/, the environment will be at ~/myproject/env/ - each project gets its own env). But we can put it whatever we like.
If we look inside the env directory we just created, we'll see a few subdirectories:
The one we care about the most is bin. This is where the local copy of the python binary and the pip installer exists. Let's start by using the copy of pip to install requests into the virtualenv (rather than globally):
$ env/bin/pip install requests /home/k/myproject/env/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: Collecting requests /home/k/myproject/env/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: Downloading requests-2.7.0-py2.py3-none-any.whl (470kB) 100% |------------------| 471kB 783kB/s Installing collected packages: requests Successfully installed requests-2.7.0
Notice that we didn't need to use sudo this time, because we're not installing requests globally, we're just installing it inside our home directory.
Now, instead of typing python to get a Python shell, we type env/bin/python:
$ env/bin/python Python 2.7.5 (default, Mar 9 2014, 22:15:05) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.get('http://bogotobogo.com') <response > >>> </response>
Instead of typing env/bin/python and env/bin/pip every time, we can run a script to activate the environment. This script, which can be executed with source env/bin/activate, simply adjusts a few variables in our shell (temporarily) so that when we type python, we actually get the Python binary inside the virtualenv instead of the global one:
$ which python /usr/bin/python $ source env/bin/activate (env)$ which python /home/k/myproject/env/bin/python (env)$
So, now we can just run pip install requests (instead of env/bin/pip install requests) and pip will install the library into the environment, instead of globally. The adjustments to our shell only last for as long as the terminal is open, so we'll need to remember to rerun source env/bin/activate each time you close and open our terminal window. If we switch to work on a different project (with its own environment), we can run deactivate to stop using one environment, and then source env/bin/activate to activate the other.
virtualenvwrapper is a set of extensions to virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing our development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.
To install it:
$ sudo pip install virtualenvwrapper ... Successfully installed stevedore-1.18.0 virtualenv-15.1.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.2
Using virtualenv without virtualenvwrapper is a little bit painful because everytime we want to activate a virtual environment, so we have to type long command like this:
$ source ~/myproject/env/bin/activate
But with virtualenvwrapper, we only need to type:
$ workon myproject
Note that we haven't installed virtualenvwrapper.sh, this may not work yet.
As initialization steps, we will want to add the command to source /usr/local/bin/virtualenvwrapper.sh to our shell startup file, changing the path to virtualenvwrapper.sh depending on where it was installed by pip:
$ vi ~/.bashrc export WORKON_HOME=~/Envs
Run the script:
$ source ~/.bashrc
Then, execute the following:
$ mkdir -p $WORKON_HOME $ echo $WORKON_HOME /home/k/Envs
$ source /usr/local/bin/virtualenvwrapper.sh virtualenvwrapper.user_scripts creating /home/k/Envs/premkproject virtualenvwrapper.user_scripts creating /home/k/Envs/postmkproject virtualenvwrapper.user_scripts creating /home/k/Envs/initialize virtualenvwrapper.user_scripts creating /home/k/Envs/premkvirtualenv virtualenvwrapper.user_scripts creating /home/k/Envs/postmkvirtualenv virtualenvwrapper.user_scripts creating /home/k/Envs/prermvirtualenv virtualenvwrapper.user_scripts creating /home/k/Envs/postrmvirtualenv virtualenvwrapper.user_scripts creating /home/k/Envs/predeactivate virtualenvwrapper.user_scripts creating /home/k/Envs/postdeactivate virtualenvwrapper.user_scripts creating /home/k/Envs/preactivate virtualenvwrapper.user_scripts creating /home/k/Envs/postactivate virtualenvwrapper.user_scripts creating /home/k/Envs/get_env_details
We may want to run the script from ~/.bashrc:
$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
Let's create our first virtualenv:
$ mkvirtualenv env1 New python executable in /home/k/Envs/env1/bin/python Installing setuptools, pip, wheel...done. virtualenvwrapper.user_scripts creating /home/k/Envs/env1/bin/predeactivate virtualenvwrapper.user_scripts creating /home/k/Envs/env1/bin/postdeactivate virtualenvwrapper.user_scripts creating /home/k/Envs/env1/bin/preactivate virtualenvwrapper.user_scripts creating /home/k/Envs/env1/bin/postactivate virtualenvwrapper.user_scripts creating /home/k/Envs/env1/bin/get_env_details (env1) k@laptop:~$
We are not limited to a single virtualenv, and we can add another env:
(env1) k@laptop:~$ mkvirtualenv env2
List the envs:
(env1) k@laptop:~$ ls $WORKON_HOME env1 env2 ...
Now, we can switch between envs with workon command:
(env2) k@laptop:~$ workon env1 (env1) k@laptop:~$ echo $VIRTUAL_ENV /home/k/Envs/env1 (env1) k@laptop:~$
Now we can install some software into the environment:
(env1) k@laptop:~$ pip install django Collecting django Using cached Django-1.10.3-py2.py3-none-any.whl Installing collected packages: django Successfully installed django-1.10.3 (env1) k@laptop:~$
We can check the new package with lssitepackages:
(env1) k@laptop:~$ lssitepackages django pip setuptools-29.0.1.dist-info Django-1.10.3.dist-info pip-9.0.1.dist-info wheel easy_install.py pkg_resources wheel-0.29.0.dist-info easy_install.pyc setuptools (env1) k@laptop:~$
How to leave/exit/deactivate a python virtualenv?
(env1) k@laptop:~$ deactivate k@laptop:~$ workon env1
In this section, we have a sample of using virtualenv for Django application. The source is available from sfvue2.
We get the following from the source:
The following is from install.mb, and we can see how it works:
*Make sure you have Python2.7.3, virtualenv, pip and sqlite3 installed*
- Download or clone this repo.
- Go to project home folder and run these commands:
$ cp sfvue/example_local.py sfvue/local_settings.py $ virtualenv venv $ source venv/bin/activate
- This will create a virtual environment and activate it.
(note) We may want to install mariadb-devel to avoid
vertualenv EnvironmentError: mysql_config not found
Now use pip to install dependencies with:
$ pip install -r dev-requirements.txt(note) --allow-external PIL --allow-unverified PIL PIL==1.1.7
- Now we have to prepare a database:
$ python manage.py syncdb
- It will ask you to provide username, email and password. Give them and run following migrations:
$ python manage.py migrate guardian $ python manage.py migrate resources $ python manage.py migrate profiles
- Run django server
$ python manage.py runserver
- Go to [http://127.0.0.1:8000/admin/](http://127.0.0.1:8000/admin/)
- Create Resource Types named Book, Ebook, Tutorial, Online Course, Other.
- Go to home page.
Running Python Programs (os, sys, import)
Modules and IDLE (Import, Reload, exec)
Object Types - Numbers, Strings, and None
Strings - Escape Sequence, Raw String, and Slicing
Strings - Methods
Formatting Strings - expressions and method calls
Files and os.path
Traversing directories recursively
Regular Expressions with Python
Object Types - Lists
Object Types - Dictionaries and Tuples
Functions def, *args, **kargs
map, filter, and reduce
Sets (union/intersection) and itertools - Jaccard coefficient and shingling to check plagiarism
Hashing (Hash tables and hashlib)
Dictionary Comprehension with zip
The yield keyword
Generator Functions and Expressions
Classes and Instances (__init__, __call__, etc.)
if__name__ == '__main__'
@static method vs class method
Private attributes and private methods
bits, bytes, bitstring, and constBitStream
json.dump(s) and json.load(s)
Python Object Serialization - pickle and json
Python Object Serialization - yaml and json
Priority queue and heap queue data structure
Graph data structure
Dijkstra's shortest path algorithm
Prim's spanning tree algorithm
Functional programming in Python
Remote running a local file using ssh
SQLite 3 - A. Connecting to DB, create/drop table, and insert data into a table
SQLite 3 - B. Selecting, updating and deleting data
MongoDB with PyMongo I - Installing MongoDB ...
Python HTTP Web Services - urllib, httplib2
Web scraping with Selenium for checking domain availability
REST API : Http Requests for Humans with Flask
Blog app with Tornado
Python Network Programming I - Basic Server / Client : A Basics
Python Network Programming I - Basic Server / Client : B File Transfer
Python Network Programming II - Chat Server / Client
Python Network Programming III - Echo Server using socketserver network framework
Python Network Programming IV - Asynchronous Request Handling : ThreadingMixIn and ForkingMixIn
Python Interview Questions I
Python Interview Questions II
Python Interview Questions III
Python Interview Questions IV
Python Interview Questions V
Image processing with Python image library Pillow
Python and C++ with SIP
PyDev with Eclipse
Redis with Python
NumPy array basics A
NumPy Matrix and Linear Algebra
Pandas with NumPy and Matplotlib
Batch gradient descent algorithm
Longest Common Substring Algorithm
Python Unit Test - TDD using unittest.TestCase class
Simple tool - Google page ranking by keywords
Google App Hello World
Google App webapp2 and WSGI
Uploading Google App Hello World
Python 2 vs Python 3
virtualenv and virtualenvwrapper
Uploading a big file to AWS S3 using boto module
Scheduled stopping and starting an AWS instance
Cloudera CDH5 - Scheduled stopping and starting services
Removing Cloud Files - Rackspace API with curl and subprocess
Checking if a process is running/hanging and stop/run a scheduled task on Windows
Apache Spark 1.3 with PySpark (Spark Python API) Shell
Apache Spark 1.2 Streaming
bottle 0.12.7 - Fast and simple WSGI-micro framework for small web-applications ...
Flask app with Apache WSGI on Ubuntu14/CentOS7 ...
Fabric - streamlining the use of SSH for application deployment
Ansible Quick Preview - Setting up web servers with Nginx, configure enviroments, and deploy an App
Neural Networks with backpropagation for XOR using one hidden layer
NLP - NLTK (Natural Language Toolkit) ...
RabbitMQ(Message broker server) and Celery(Task queue) ...
OpenCV3 and Matplotlib ...
Simple tool - Concatenating slides using FFmpeg ...
iPython - Signal Processing with NumPy
iPython and Jupyter - Install Jupyter, iPython Notebook, drawing with Matplotlib, and publishing it to Github
iPython and Jupyter Notebook with Embedded D3.js
Downloading YouTube videos using youtube-dl embedded with Python
Machine Learning : scikit-learn ...
Django 1.6/1.8 Web Framework ...
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization