PyTorch 2.0 release accelerates open-source machine learning

Among the most widely used machine learning (ML) technologies today is the open source framework PyTorch.

PyTorch debuted on Facebook (now known as Meta) in 2016 with the release of version 1.0 in 2018. In September 2022, Meta moved the PyTorch project to the new PyTorch Foundation, which is managed by the Linux Foundation. Today, the developers of PyTorch took the next major step for PyTorch, announcing the first experimental release of PyTorch 2.0. The new release promises to help speed up ML training and development, while maintaining backwards compatibility with existing PyTorch application code.

“We’ve added an additional feature called `torch.compile` for users to insert into their codebases,” said PyTorch maintainer Soumith Chintala. says VentureBeat. “We call it 2.0 because we believe users will find it a great new addition to the experience.”

The new PyTorch compiler that makes all the difference for ML

There have been discussions in the past about when the PyTorch project should call for a new 2.0 release.

In 2021, for example, there was a brief discussion about whether PyTorch 1.10 should be labeled as a 2.0 version. Chintala said PyTorch 1.10 doesn’t have enough fundamental changes from 1.9 to warrant a major upgrade to 2.0.

The most recent generally available version of PyTorch is version 1.13, which was released at the end of October. A key feature of this release came from an IBM code contribution that allowed the machine learning framework to work more efficiently with basic Ethernet networks for large-scale workloads.

Chintala pointed out that the time has come for PyTorch 2.0 as the project introduces an additional new paradigm in the PyTorch user experience, called torch.compile, which brings users solid speedups that were not possible in the impatient mode by flaw of PyTorch 1.0.

He explained that out of about 160 open source models that the PyTorch project validated on early versions of 2.0, there was a 43% speedup and they worked reliably with the addition of a line to the base. of code.

“We expect that with PyTorch 2, people will change the way they use PyTorch on a daily basis,” Chintala said.

He said that with PyTorch 2.0, developers will begin experimenting with impatient mode and, once they have trained their models for long periods of time, will enable compiled mode for additional performance.

“Data scientists will be able to do with PyTorch 2.x the same things they did with 1.x, but they can do them faster and at scale,” Chintala said. “If your model trained in 5 days, and with the compiled mode of 2.x it now trains in 2.5 days, you can iterate on more ideas with that extra time, or build a model taller who trains in the same 5 days.”

More Python coming to PyTorch 2.x

PyTorch derives the first part of its name (Py) from the open-source Python programming language widely used in data science.

Modern versions of PyTorch, however, were not written entirely in Python, as parts of the framework are now written in the C++ programming language.

“Over the years, we’ve moved many parts of torch.nn from Python to C++ to optimize last-mile performance,” Chintala said.

Chintala said that in the latest 2.x series (but not 2.0), the PyTorch project plans to move code related to torch.nn to Python. He noted that C++ is generally faster than Python, but the new compiler (torch.compile) ends up being faster than running the equivalent code in C++.

“Moving these parts back to Python improves hackability and lowers the barrier for code contributions,” Chintala said.

Work on Python 2.0 will continue over the next few months and general availability is not expected until March 2023. Along with the development effort, PyTorch is transitioning from being governed and operated by Meta to its own independent effort .

“This is the start of the PyTorch Foundation, and you will hear more over a longer time horizon,” Chintala said. “The foundation is in the process of executing various transfers and setting goals.”