Bundle-python

Version 42 (Adrian Georgescu, 03/04/2018 03:05 pm)

1 1 Saúl Ibarra Corretgé
h1. Building a Python Framework to bundle inside Blink
2 1 Saúl Ibarra Corretgé
3 1 Saúl Ibarra Corretgé
In order to avoid using the system Python a custom Framework build is needed. Using a bundled Python version will make the package bigger in size, but all package versions are controlled and not up to the environment. Also, we can use the latest Python version, with latest bugfixes and features, since Apple only updates the system Python version on every major OS release.
4 1 Saúl Ibarra Corretgé
5 13 Saúl Ibarra Corretgé
The following instructions only apply for 64bit builds, 32bit builds are no longer supported.
6 7 Adrian Georgescu
7 22 Adrian Georgescu
Blink dependencies must be installed under the following directory structure:
8 22 Adrian Georgescu
9 22 Adrian Georgescu
* Distribution/Frameworks/
10 22 Adrian Georgescu
* Distribution/Resources/lib
11 1 Saúl Ibarra Corretgé
 
12 7 Adrian Georgescu
h2. Building the Python Framework itself
13 1 Saúl Ibarra Corretgé
14 18 Saúl Ibarra Corretgé
* Install it using Homebrew
15 1 Saúl Ibarra Corretgé
16 1 Saúl Ibarra Corretgé
<pre>
17 40 Adrian Georgescu
brew install python (this may install python 3)
18 40 Adrian Georgescu
19 40 Adrian Georgescu
To install 2.7:
20 40 Adrian Georgescu
21 41 Adrian Georgescu
brew install python2
22 41 Adrian Georgescu
23 41 Adrian Georgescu
or
24 40 Adrian Georgescu
https://stackoverflow.com/questions/18671253/how-can-i-use-homebrew-to-install-both-python-2-and-3-on-mac
25 40 Adrian Georgescu
26 40 Adrian Georgescu
brew install pyenv
27 40 Adrian Georgescu
pyenv install 2.7
28 1 Saúl Ibarra Corretgé
</pre>
29 1 Saúl Ibarra Corretgé
30 18 Saúl Ibarra Corretgé
The framework will be installed and linked with Homebrew supplied OpenSSL and SQLite versions. Those libraries will need to be copied too.
31 1 Saúl Ibarra Corretgé
32 18 Saúl Ibarra Corretgé
*NOTE*: Be careful when copying the framework around, it contains symlinks and if @cp -r@ is used the size will we doubled, use @cp -a@ instead.
33 22 Adrian Georgescu
34 22 Adrian Georgescu
The Python framework is found in
35 22 Adrian Georgescu
36 22 Adrian Georgescu
<pre>
37 27 Adrian Georgescu
cp -a /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework ~/work/blink/Distribution/Frameworks/
38 22 Adrian Georgescu
</pre>
39 22 Adrian Georgescu
40 2 Saúl Ibarra Corretgé
* Reduce the size of the Python Framework:
41 2 Saúl Ibarra Corretgé
42 1 Saúl Ibarra Corretgé
There are a number of things that can (and must when submitting a sandbox app to Mac App Store) be removed from the framework directory to make it smaller in size:
43 1 Saúl Ibarra Corretgé
44 1 Saúl Ibarra Corretgé
<pre>
45 27 Adrian Georgescu
cd ~/work/blink/Distribution/Frameworks//Python.framework
46 42 Adrian Georgescu
cd Versions/2.7
47 42 Adrian Georgescu
ln -sf Current 2.7
48 42 Adrian Georgescu
cd ~/work/blink/Distribution/Frameworks//Python.framework
49 42 Adrian Georgescu
ln -sf Versions/Current/Headers .
50 42 Adrian Georgescu
ln -sf Versions/Current/Python .
51 42 Adrian Georgescu
ln -sf Versions/Current/Resources .
52 26 Adrian Georgescu
find . -name *.pyc -exec rm -r "{}" \; 
53 26 Adrian Georgescu
find . -name *.pyo -exec rm -r "{}" \; 
54 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/config/python.o
55 24 Adrian Georgescu
rm -r Versions/Current/bin
56 24 Adrian Georgescu
rm -r Versions/Current/Resources/*
57 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/test
58 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/plat-*
59 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/idlelib
60 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/curses
61 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib2to3
62 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-tk
63 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/bsddb
64 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-dynload/gdbm.so
65 24 Adrian Georgescu
rm -r Versions/Current/lib/python2.7/lib-dynload/readline.so
66 35 Adrian Georgescu
rm -r Versions/2.7/lib/python2.7/site-packages
67 1 Saúl Ibarra Corretgé
</pre>
68 3 Saúl Ibarra Corretgé
69 1 Saúl Ibarra Corretgé
Replace @Versions/Current/lib/python2.7/site.py@ with an empty file.
70 28 Adrian Georgescu
71 28 Adrian Georgescu
<pre>
72 38 Adrian Georgescu
rm ~/work/blink/Distribution/Frameworks//Python.framework/Versions/Current/lib/python2.7/site.py
73 38 Adrian Georgescu
touch ~/work/blink/Distribution/Frameworks//Python.framework/Versions/Current/lib/python2.7/site.py
74 28 Adrian Georgescu
</pre>
75 5 Saúl Ibarra Corretgé
76 34 Adrian Georgescu
Python Framework needs file a Info.plist file under Resources in order to be compatible with latest OSX bundle structure:
77 34 Adrian Georgescu
78 34 Adrian Georgescu
<pre>
79 34 Adrian Georgescu
cp build_scripts/PythonFramework.plist Distribution/Frameworks/Python.framework/Resources/Info.plist           
80 34 Adrian Georgescu
</pre>
81 34 Adrian Georgescu
82 34 Adrian Georgescu
83 5 Saúl Ibarra Corretgé
h2. Compiling PyObjC
84 5 Saúl Ibarra Corretgé
85 13 Saúl Ibarra Corretgé
In order to get a PyObjC version that will work with the framework created above (Python 2.7, 64bits) an equivalent Python must be used to compile it. That is, if has to be a Python 2.7 version (it doesn't have to be the exact version) and it has to be a 64bit version. The MACOSX_DEPLOYMENT_TARGET must also be set to the appropriate value.
86 5 Saúl Ibarra Corretgé
87 5 Saúl Ibarra Corretgé
PyObjcC can be installed with easy_install or pip. We install it in 2 steps to save some compilation time due to a bug in the build system:
88 1 Saúl Ibarra Corretgé
89 5 Saúl Ibarra Corretgé
<pre>
90 5 Saúl Ibarra Corretgé
pip install pyobjc-core
91 5 Saúl Ibarra Corretgé
pip install pyobjc
92 23 Adrian Georgescu
pip install pycrypto
93 1 Saúl Ibarra Corretgé
</pre>
94 1 Saúl Ibarra Corretgé
95 5 Saúl Ibarra Corretgé
When compiling PyObjC a Python package will be created for every system framework, but not all of them are needed (at the moment), so just pick the ones we use:
96 5 Saúl Ibarra Corretgé
97 5 Saúl Ibarra Corretgé
<pre>
98 5 Saúl Ibarra Corretgé
AddressBook
99 5 Saúl Ibarra Corretgé
AppKit
100 1 Saúl Ibarra Corretgé
Cocoa
101 5 Saúl Ibarra Corretgé
CoreFoundation
102 5 Saúl Ibarra Corretgé
Foundation
103 13 Saúl Ibarra Corretgé
JavaScriptCore
104 5 Saúl Ibarra Corretgé
LaunchServices
105 1 Saúl Ibarra Corretgé
PyObjCTools
106 1 Saúl Ibarra Corretgé
Quartz
107 13 Saúl Ibarra Corretgé
ScriptingBridge
108 13 Saúl Ibarra Corretgé
StoreKit
109 1 Saúl Ibarra Corretgé
WebKit
110 8 Adrian Georgescu
objc
111 6 Saúl Ibarra Corretgé
</pre>
112 6 Saúl Ibarra Corretgé
113 1 Saúl Ibarra Corretgé
114 21 Adrian Georgescu
For example this is the content of a Resources/lib bundled with Blink Cocoa as of November 3rd, 2016 (including sipsimple dependencies & all):
115 20 Adrian Georgescu
116 1 Saúl Ibarra Corretgé
<pre>
117 1 Saúl Ibarra Corretgé
AVFoundation
118 1 Saúl Ibarra Corretgé
AddressBook
119 1 Saúl Ibarra Corretgé
AppKit
120 1 Saúl Ibarra Corretgé
Cocoa
121 1 Saúl Ibarra Corretgé
CoreFoundation
122 1 Saúl Ibarra Corretgé
Crypto
123 1 Saúl Ibarra Corretgé
Foundation
124 1 Saúl Ibarra Corretgé
LaunchServices
125 1 Saúl Ibarra Corretgé
PyObjCTools
126 1 Saúl Ibarra Corretgé
Quartz
127 1 Saúl Ibarra Corretgé
ScriptingBridge
128 1 Saúl Ibarra Corretgé
WebKit
129 1 Saúl Ibarra Corretgé
_cffi_backend.so
130 1 Saúl Ibarra Corretgé
_ldap.so
131 1 Saúl Ibarra Corretgé
_markerlib
132 1 Saúl Ibarra Corretgé
application
133 1 Saúl Ibarra Corretgé
cffi
134 1 Saúl Ibarra Corretgé
cjson.so
135 1 Saúl Ibarra Corretgé
cryptography
136 1 Saúl Ibarra Corretgé
cryptography-1.5.1.dist-info
137 1 Saúl Ibarra Corretgé
dateutil
138 1 Saúl Ibarra Corretgé
dns
139 1 Saúl Ibarra Corretgé
dsml.py
140 1 Saúl Ibarra Corretgé
enum
141 1 Saúl Ibarra Corretgé
eventlib
142 1 Saúl Ibarra Corretgé
formencode
143 1 Saúl Ibarra Corretgé
gmpy2.so
144 1 Saúl Ibarra Corretgé
gnutls
145 1 Saúl Ibarra Corretgé
greenlet.so
146 1 Saúl Ibarra Corretgé
idna
147 1 Saúl Ibarra Corretgé
ipaddress.py
148 1 Saúl Ibarra Corretgé
ldap
149 1 Saúl Ibarra Corretgé
ldapurl.py
150 1 Saúl Ibarra Corretgé
ldif.py
151 1 Saúl Ibarra Corretgé
lxml
152 1 Saúl Ibarra Corretgé
msrplib
153 1 Saúl Ibarra Corretgé
objc
154 1 Saúl Ibarra Corretgé
otr
155 1 Saúl Ibarra Corretgé
pkg_resources
156 1 Saúl Ibarra Corretgé
pyasn1
157 1 Saúl Ibarra Corretgé
pycparser
158 1 Saúl Ibarra Corretgé
pydispatch
159 1 Saúl Ibarra Corretgé
pytz
160 1 Saúl Ibarra Corretgé
service_identity
161 1 Saúl Ibarra Corretgé
sipsimple
162 1 Saúl Ibarra Corretgé
six.py
163 1 Saúl Ibarra Corretgé
sqlobject
164 1 Saúl Ibarra Corretgé
twisted
165 1 Saúl Ibarra Corretgé
xcaplib
166 21 Adrian Georgescu
</pre>
167 21 Adrian Georgescu
168 21 Adrian Georgescu
169 21 Adrian Georgescu
*NOTE:* The _objc_ package is located inside a _PyObjC_ directory, just copy it from there, without the parent directory.
170 21 Adrian Georgescu
171 21 Adrian Georgescu
<pre>
172 21 Adrian Georgescu
__import__('pkg_resources').declare_namespace(__name__)
173 21 Adrian Georgescu
</pre>
174 21 Adrian Georgescu
175 25 Adrian Georgescu
h2. Fix library paths
176 25 Adrian Georgescu
177 25 Adrian Georgescu
All libraries must have their relative path change to the Framework path bundled within Blink.app
178 25 Adrian Georgescu
179 25 Adrian Georgescu
<pre>
180 25 Adrian Georgescu
#!/bin/sh
181 25 Adrian Georgescu
182 25 Adrian Georgescu
old_path="local/lib/\|local/Cellar/\|/usr/local/opt/libmpc/lib/\|/usr/local/opt/mpfr/lib/\|Frameworks/Frameworks/\|/Users/adigeo/work/ag-projects/video/local/lib/"
183 25 Adrian Georgescu
new_path="@executable_path/../Frameworks/"
184 25 Adrian Georgescu
185 25 Adrian Georgescu
for library in $@; do
186 25 Adrian Georgescu
  install_name_tool -id $new_path$library $library
187 25 Adrian Georgescu
  dependencies=$(otool -L $library | grep $old_path | awk '{print $1}')
188 25 Adrian Georgescu
  for dependency in $dependencies; do
189 25 Adrian Georgescu
      new_basename=$(basename $dependency)
190 25 Adrian Georgescu
      new_name="$new_path$new_basename"
191 25 Adrian Georgescu
      echo $dependency $new_name $library
192 25 Adrian Georgescu
      install_name_tool -change $dependency $new_name $library
193 25 Adrian Georgescu
  done
194 1 Saúl Ibarra Corretgé
done
195 1 Saúl Ibarra Corretgé
</pre>
196 1 Saúl Ibarra Corretgé
197 1 Saúl Ibarra Corretgé
A script is available in ./build_scripts/ directory
198 1 Saúl Ibarra Corretgé
199 1 Saúl Ibarra Corretgé
<pre>
200 1 Saúl Ibarra Corretgé
./build_scripts/change_lib_names.sh Distribution/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/*.so
201 1 Saúl Ibarra Corretgé
chmod +w Distribution/Frameworks/Python.framework/Versions/Current/Python
202 1 Saúl Ibarra Corretgé
./build_scripts/change_lib_names.sh Distribution/Frameworks/Python.framework/Versions/Current/Python
203 1 Saúl Ibarra Corretgé
</pre>
204 1 Saúl Ibarra Corretgé
205 33 Adrian Georgescu
*NOTE*: Python.framework as well as all other libraries must be signed using command line tools. Make sure when building Blink that "Code sign on copy" option is disabled for Python.framework. This script can be used to sign all libraries and frameworks
206 33 Adrian Georgescu
207 33 Adrian Georgescu
<pre>
208 33 Adrian Georgescu
sos=`find ./Resources/lib -name *.so`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
209 33 Adrian Georgescu
sos=`find ./Frameworks -name *.dylib`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
210 33 Adrian Georgescu
sos=`find ./Frameworks -name *.so`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
211 33 Adrian Georgescu
sos=`find ./Frameworks -name *.o`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
212 33 Adrian Georgescu
sos=`find ./Frameworks -name *.a`; for s in $sos; do codesign -f -s '3rd Party Mac Developer Application: AG Projects' $s; done
213 33 Adrian Georgescu
</pre>
214 33 Adrian Georgescu
215 33 Adrian Georgescu
A script is available in ./build_scripts/ directory
216 33 Adrian Georgescu
217 33 Adrian Georgescu
<pre>
218 33 Adrian Georgescu
./build_scripts/codesign.sh 
219 33 Adrian Georgescu
</pre>
220 29 Adrian Georgescu
221 21 Adrian Georgescu
h2. Module exceptions
222 21 Adrian Georgescu
223 21 Adrian Georgescu
When copying built Python modules into the distribution folder, care must be taken with the 2 following packages:
224 21 Adrian Georgescu
225 1 Saúl Ibarra Corretgé
* zope: an empty @__init__.py@ file must be created in the @zope@ directory
226 1 Saúl Ibarra Corretgé
* cryptography: the @*-dist.info@ must be copied too
227 39 Adrian Georgescu
* _PyObjCTools_ is not a valid Python package, as it lacks a @__init__.py@ file, an empty one needs to be manually created with this content:
228 39 Adrian Georgescu
229 21 Adrian Georgescu
230 21 Adrian Georgescu
h1. Creating a sandbox (Python virtualenv)
231 21 Adrian Georgescu
232 21 Adrian Georgescu
<pre>
233 21 Adrian Georgescu
sudo easy_install pip
234 21 Adrian Georgescu
sudo pip install virtualenv virtualenvwrapper
235 21 Adrian Georgescu
</pre>
236 21 Adrian Georgescu
237 21 Adrian Georgescu
Add to ~.bashrc
238 21 Adrian Georgescu
239 21 Adrian Georgescu
<pre>
240 21 Adrian Georgescu
# Virtualenv
241 21 Adrian Georgescu
export WORKON_HOME=$HOME/.virtualenvs
242 21 Adrian Georgescu
export PIP_VIRTUALENV_BASE=$WORKON_HOME
243 21 Adrian Georgescu
export PIP_RESPECT_VIRTUALENV=true
244 21 Adrian Georgescu
export VIRTUALENVWRAPPER_SCRIPT=/usr/local/bin/virtualenvwrapper.sh
245 21 Adrian Georgescu
[[ -f /usr/local/bin/virtualenvwrapper_lazy.sh ]] && source /usr/local/bin/virtualenvwrapper_lazy.sh
246 21 Adrian Georgescu
</pre>
247 21 Adrian Georgescu
248 21 Adrian Georgescu
Creating a sandbox:
249 21 Adrian Georgescu
250 21 Adrian Georgescu
<pre>
251 21 Adrian Georgescu
mkvirtualenv -p $(which python2.7) sandbox
252 21 Adrian Georgescu
</pre>
253 21 Adrian Georgescu
254 21 Adrian Georgescu
Exiting the sandbox:
255 21 Adrian Georgescu
256 21 Adrian Georgescu
<pre>
257 21 Adrian Georgescu
deactivate
258 21 Adrian Georgescu
</pre>
259 21 Adrian Georgescu
260 21 Adrian Georgescu
Entering the sandbox:
261 21 Adrian Georgescu
262 21 Adrian Georgescu
<pre>
263 21 Adrian Georgescu
workon sandbox
264 19 Adrian Georgescu
</pre>