Hyukjin Kwon
Hyukjin is a Databricks software engineer as the tech-lead in OSS PySpark team, ASF member, Apache Spark PMC member and committer, working on many different areas in Apache Spark such as PySpark, Spark SQL, SparkR, infrastructure, etc. He is the top contributor in Apache Spark, and leads efforts such as Project Zen, Pandas API on Spark, and Python Spark Connect.
Sessions
In this talk, we'll explore effective strategies for scaling pandas workloads using PySpark. We'll delve into techniques such as the Pandas API on Spark, Python UDFs, Pandas UDFs, and Pandas Function APIs. In addition, this talk covers how to manage dependencies and environment setup seamlessly when transitioning to distributed PySpark cluster, providing insights into optimizing performance and leveraging PySpark features for seamless integration with pandas workflows.