Diederik Greveling: Building a Multi-Core Apply Function for Pandas | PyData Amsterdam 2019

Most people who worked with pandas used the pandas apply method and might have encountered it to be slow. Vectorizing your function will improve the speed issues, however, this can sometimes be troublesome and tedious. As an alternative, one can use a multiprocessing counterpart to pandas, for example, dask or modin. While these are perfectly fine alternatives, they might be overkill for executing a simple parallel apply function.

In this talk, we explore how hard it would be to create a multi-core apply method for pandas which results in increased execution speed but keeps memory overhead to a minimum. We go into the technical side of python memory management and explore copy on write memory whilst also exploring the way pandas data frames are built. In the end, this effort is combined into a simple pandas extension.

Some python code examples of how to invoke copy on write memory in python multiprocessing will be shown, as well as how the multi-core apply extension for pandas was build. Overall, this will be an interesting introduction to python multiprocessing and learning some of the internals in pandas.

www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

Python,Tutorial,Education,NumFOCUS,PyData,Opensource,download,learn,syntax,software,python 3,pandas,multi core apply function,vector,