What's new in dispy 4.12.3
Sep 10, 2020
- dispynode under Windows doesn't force daemon mode. Command line now can be used to see the status of computations, change CPUs, quit disypynode etc.
- It is possible now to use client and nodes with different versions of Python 3 even if pickle protocol versions are different. See pycos configuration variable PickleProtocolVersion; dispy uses pycos for communication so setting this variable appropriately will allow serialization /deserialization even if different Python versions use different default protocol versions (e.g., Python 3.7 and 3.8 use different protocol versions). Note that this will not work between Python 3 and Python 2.
- This version requires pycos 4.9.0 due to changes in pycos's Task exit value semantics. If dispy is installed manually, please upgrade pycos to 4.9.0 as well.
New in dispy 4.12.2 (May 14, 2020)
- Fixed white space issues in httpd module in Python 3 so it works (same as httpd in Python 2).
New in dispy 4.12.1 (Apr 11, 2020)
- Fixed dispynode in version 4.12.0 to run setup only client uses this feature; otherwise, dispynode crashes. With this programs that don't use setup, includingsample.py in examples, work now.
New in dispy 4.10.5 (Mar 27, 2019)
- DispyNode instances now have tx and rx attributes that maintain amount data sent to / received from that node. print_status method and web interface now show this information.
New in dispy 4.10.4 (Mar 27, 2019)
- Added support for Python 3.7.
New in dispy 4.10.3 (Mar 27, 2019)
- Fixed job scheduler to handle jobs submitted with submit_node.
- Added discover_nodes method to JobCluster that clients can use to establish communication with nodes that may not be found when cluster initialized.
New in dispy 4.10.2 (Mar 27, 2019)
- Added option force_cleanup to dispynode so all files transferred or created by computation are removed when computation is closed, even if computation disables cleanup.
- Fixed processing of provisional results so jobs are not dropped after first such result.
- Fixed close_node method.
New in dispy 4.10.1 (Mar 27, 2019)
- Added support for suid and sgid features in Unix so a computation can not access files of other computations. See "Isolate Computation Files" for details and how to use this feature.
- id attribute of jobs is now set by dispy to successeive integers starting with 1.
- dispynode checks if any job process is dead at zombie_interval interval and if so, sends reply for it.
- If client / scheduler determines that a node is dead (or disconnected), jobs scheduled on it are not discarded now, so if a node comes back, client can accept replies for those jobs.
- set_node_cpus, deallocate_node etc. now accept DispyNode instances as arguments (in addition to host name or IP address, as done before).
New in dispy 4.10.0 (Mar 27, 2019)
- Added support for *BSD (FreeBSD, NetBSD etc).
- Run setup and cleanup functions in a subprocess (instead of in main dispynode process). This prevents computations from interfering each other (i.e., there are no side effects from one computation loading modules etc.).
- Semantics of return value from setup function has changed. See setup function description in dispy JobCluster for details.
- Added --unsafe_setup option to dispynode to run setup and cleanup functions in main dispynode process, as used to be the case until 4.10.0 release.
- Fixed SharedJobCluster so accept remote address as scheduler_node parameter.
- Improved process termination when client cancels jobs and when clean option is used to terminate processes in prior run. If psutil module is available (which is strongly recommended), dispynode makes additional checks to make sure processes being terminated are from previous dispynode run (when clean option is used).
New in dispy 4.8.7 (May 8, 2018)
- Fixed shutdown method to delete fault recovery file when SharedJobCluster used (and client doesn't crash).
- In some cases with IPv6 sockets, attribute IPV6_V6ONLY is not defined, causing crash during start; this is now fixed.
- Fixed parsing of ip_addrs and ext_ip_addrs fields saved in config file with dispynode.
- Fixed dispynode so changes to os.environ by one client don't interfere other clients.
New in dispy 4.8.6 (Apr 20, 2018)
- Added deallocate_node and close_node methods to cluster.
- Added controls to web interface to allocate / deallocate / close node(s).
- Added "release" command to dispynode to release from current client if client is not active.
New in dispy 4.8.5 (Mar 26, 2018)
- Fixed dispyscheduler crash when started in 'daemon' mode.
- When a node is rediscovered (i.e., after a node is lost and comes alive), use it again for any active clusters.
- When jobs are rescheduled or abandoned after a node is lost, call 'status' function (if provided) with appropriate job argument.
- Fixed issue with binding to UDP with Windows
New in dispy 4.8.4 (Jan 11, 2018)
- Fixed client crash when SharedJobCluster is closed with Python 3.
- Improved warning messsages when client's UDP server binding to its port fails.
New in dispy 4.8.3 (Aug 1, 2017)
- Fixed parsing of ext_ip_addr option to dispynode.
- Fixed recover_jobs function (which can be used for fault-recovery in case client crashes or loses network connection to nodes).
- recover_file option to recover_jobs is now optional; if this option is not given, latest file with prefix _dispy_ is used (default falut-recovery files are of the form _dispy_*).
New in dispy 4.8.2 (Jun 28, 2017)
- License has been changed to Apache 2.0 (from MIT).
- If netifaces is not available, appropriate address is used for binding to UDP for broadcasting.
New in dispy 4.8.1 (Jun 14, 2017)
- dispy version 4.8.1 has been released. In this version crash with dispyscheduler (due to conflicts of names of variables and methods with change from asyncoro to pycos) has been fixed.
New in dispy 4.7.6 (May 4, 2017)
- dispy version 4.7.6 has been released. In this version httpd and dispynetrelay have been fixed to support IPv6.
New in dispy 4.7.5 (Apr 21, 2017)
- Fixed IPv6 for Windows and OS X. IPv6 with Python 2.7 under Windows needs package win_inet_ptorn (it is not required for Python 3.6+). For IPv6 under OS X, package netifaces is required. Even if not required with other platforms, it is recommended to install netifaces to select suitable interface from multiple configured interfaces.
- Fixed dispyscheduler's httpd functionality (to monitor computations scheduled with SharedJobScheduler)
New in dispy 4.7.1 (Mar 14, 2017)
- SSL has been fixed. The fix is in asyncoro 4.4.0 version. If dispy is installed with pip, asyncoro will automatically be upgraded. Otherwise, please upgrade asyncoro to latest version.
- Python function computations can now refer to 'dispy_node_name' and 'dispy_node_ip_addr'. These give name and IP address of node where computation is being executed.
- dispynode will remove any files generated by computation and left behind when computation is closed. Earlier such files were left behind by dispynode, taking up disk space.
- dispyscheduler has new option cleanup_nodes that if set (to True), will cause dispynode to cleanup (i.e., remove any files transferred and generated by computations) even if computations set cleanup=False. This can be used when nodes are being shared by multiple users and computations don't leave beind any files, which may take up disk space.
- Added support for IPv6. It is strongly recommended that netifaces module is installed to use IPv6.
New in dispy 4.6.17 (Sep 29, 2016)
- NodeAllocate's allocate method is now called with additional parameter platform, which is value returned by platform.platform() on the node. This can be used to filter / allocate nodes depending on platform when cluster consists of nodes with different platforms.
- Fixed dispynode to terminate when 'quit' or 'exit' commands are given.
New in dispy 4.6.16 (Aug 17, 2016)
- Added --client_shutdown option to dispynode. If this option is given, client program can call dispynode_shutdown() in cleanup function to shutdown dispynode.
- --save_config option in dispynode now takes filename argument to save configuration in. In earlier releases --config option (which is used to load configuration) had to be specified to give the filename to save configuration.
- dispynode shutdown improved - if dispynode is terminated (killed or interrupted with Ctrl+C), signal handler cleans up before quitting. In earlier releases (4.6.15 in particular), dispynode left behind some files that prevented next dispynode to start.
- Fixed a race condition with dispyscheduler and client. In earlier releases if a client submitted jobs that took very little time, the scheduler may create new jobs with same ID as some job that finished at dispyscheduler, but not finished at client, causing client to ignore such jobs.
New in dispy 4.6.15 (Jul 29, 2016)
- Modules and files transferred to remote servers are saved with same relative paths as at client. With this, modules with multiple files, and submodules etc. can be sent with depends options. The paths are preserved only for files relative to client's current working directory; files with paths not under client's working directory don't preserve paths, as saving them at remote server with such paths may not be possible (due to permissions), or unwarrnated.
- Saving and restoring configuration to initialize options to dispynode (to avoid having to give options every time / on every node) have been fixed.
New in dispy 4.6.2 (Dec 16, 2015)
- The changes since last release are:
- Fixed transferring job dependencies (specified with dispy_job_depends) when using with SharedJobCluster.
- Fixed dispyscheduler so it can now be used with separate "secret"s for nodes and clusters; i.e., options node_secret and cluster_secret can be different)..
New in dispy 4.6.1 (Dec 12, 2015)
- dispy version 4.6.1 has been released. The changes since last release are:
- Fixed SharedJobCluster in submitting jobs with arguments (broken in 4.6.0 release).
- SharedJobCluster now uses one port for getting all replies from (remote) scheduler. By giving specific port, and port forwarding (e.g., with SSH), scheduler in remote network can now be used.
New in dispy 4.6.0 (Dec 12, 2015)
- Added service_start, service_stop and service_end options to dispynode to allow a node to run jobs during specific hours of the day.
- Added save_config and config optoins to dispynode to save given options in a file and use that file to start dispynode in all machines in a cluster with those options (instead of having to list options for each run).
- Added submit_node option to JobCluster and SharedJobCluster to schedule a job to given node. This method, along with cluster_status callback can be used to implement custom job schedulers in the client itself.
- Added exclusive option to SharedJobCluster. If this option is set to True, that cluster is executed exclusively, without sharing nodes with other clusters.
- An issue since 4.5.3 release with cluster not detecting nodes (apparently seems to occur with Google Cloud) has been fixed; thanks to Stylianos Kyriacou.
New in dispy 4.5.5 (Nov 5, 2015)
- This version fixes Issue #21 - Using same cluster more than once.
New in dispy 4.5.4 (Aug 17, 2015)
- This version supports setup and cleanup parameters under Windows with some limitations (compared to Linux, OS X and other Posix systems, where there are no limitations); these should be Python functions or partial functions.
New in dispy 4.5.3 (Aug 10, 2015)
- dispy version 4.5.3 has been released. This version supports passing class instances in client to servers with Python 3 under Windows. Processes started with multiprocessing in Python 3 under Windows use __mp_main__ namespace, so user defined code is executed in that namespace so unpickling/deserialization of objects works.
New in dispy 4.5.2 (Aug 10, 2015)
- Version 4.5 introduced a feature to keep a computation's global variables in a (dictionary) variable in that computation. This feature works with Linux, OS X and other Unix variants, but not with Windows. The implementation of this feature broke Windows, as the variable couldn't be sent as argument to multiprocessing.Process. Version 4.5.2 fixes this issue so dispynode works with Windows again.
New in dispy 4.5.1 (Aug 10, 2015)
- dispy version 4.5.1 release fixes issue in 4.5 version with loading httpd module in dispyscheduler.
New in dispy 4.5 (Aug 10, 2015)
- Short summary of changes in version 4.5:
- Fixed dispyscheduler to import httpd module.
- dispynode stores global variables of each computation in its own namespace. With this, if more than once cluster (from same scheduler) is being executed, global variables initialized by one computation doesn't interfere with those of another. Moreover, a computation doesn't need to worry about having to remove global variables in 'cleanup' - they will be thrown away when computation is closed.
- Fixed 'del_cluster' to remove cluster from all the nodes before removing cluster. Otherwise, because 'yield' is used to close node, if multiple clusters are used,
New in dispy 4.4 (Aug 10, 2015)
- Short summary of changes in version 4.4:
- 'setup' and 'cleanup' can be partial (Python) functions (in addition to being normal functions). This way nodes can be setup with different functions (i.e.,. partial functions can be called with different arguments for different nodes).
- httpd module and web pages support multiple clusters in the same program. The web pages show each cluster separately, as well as a special cluster that combines information for all clusters in a cluster named '* Combined'.
- dispyscheduler program supports '--httpd' option. When this option is given, httpd server is started so all the clusters currently using the scheduler can be monitored in web browser. The names of clusters have ' @ ' appended.
- Added 'timeout' and 'terminate_pending' to 'recover_jobs' function in dispy so fault recovery can be used to close the nodes, if necessary.
- 'nodes' parameter for JobCluster and SharedJobCluster can be given as a list of 'NodeAllocate' objects. This gives more flexibility to allocate nodes, as well as customize allocation (for example, to use fewer CPUs during specific time interval).
- dispynode and dispyscheduler programs have '--msg_timeout' option to adjust socket timeout used to send/receive messages. Default is 5 seconds. If necessary, for example, to send large files over slow network, the timeout can be adjusted with this option. dispy module also has MsgTimeout variable that can be adjusted in the client program appropriately.
New in dispy 4.3 (Aug 10, 2015)
- dispy version 4.3 has been released. Most of the changes since last release are fixes. The one change is simplification of fault recovery. Now the clients store recovery information always and in case of crash, function recover_jobs in dispy module can be used to retrieve the results of jobs and release the nodes. See Fualt Recovery for more details.
New in dispy 4.2 (Aug 10, 2015)
- dispy version 4.2 has been released. The changes are:
- Fixes issue with distributing computations as programs
- Added 'close_node' function in dispy module so client can close currently running computations on the node after recovering from lost connection / crash. See doc strings for 'close_node' on how to use it.
- Added '--clean' option to dispynode and dispyscheduler. If this option is given, dispynode and dispyscheduler remove any files that are left behind by previous runs. The files are now stored by dispynode in /tmp/dispy/node and by dispyscheduler in /tmp/dispy/scheduler.
New in dispy 4.1 (Aug 10, 2015)
- dispy version 4.1 has been released with following changes from version 4.0:
- httpd module now includes interface to manage cluster (changing CPUs when JobCluster is used, adding nodes), monitoring jobs on nodes, terminating jobs etc.
- httpd can be used with SSL so https protocol can be used to monitor and manage cluster with web browsers.
- Using httpd module is slightly different from the way it was in version 4.0; see httpd module and example.
New in dispy 4.0 (Aug 10, 2015)
- dispy version 4.0 has been released. In this release, cluster_status parameter has been added to JobCluster and SharedJobCluster. This should be set to a function or method. Whenever there is a change in status of a job (such as created, submitted to a node, finished executing etc.) or a change in status of a node (initialized for given cluster, closed), given function is called. See dispy page for details.
- It includes httpd module that can be used to easily create a web interface to view cluster status. See View Cluster Status for information.
- asyncoro module is no longer included in dispy package; it must be installed separately. If Python's package index (PyPI / pip) is used for installing dispy, asyncoro will be automatically installed from PyPI. If dispy is installed manually, then asyncoro must be installed at appropriate dist-packages.
- dispy package includes basic documentation (in text form) and a fews examples. If installed with PyPI, these are stored under the directory where dispy module is stored (e.g., /usr/local/lib/python2.7/dist-packages/dispy).
New in dispy 3.22 (Aug 10, 2015)
- dispy version 3.22 has been released. With this release, dependencies can be distributed for each job (in addition to distributing dependencies for the whole computation). Job's dependencies (functions, classes, modules and files if computations are Python functions, and modules and files if computations are standalone programs) are sent with the job to the node and removed after the job is done. Job's dependencies should be given as a list to 'cluster.submit()' when creating a job. See Cluster methods for details.