tl;dr modify the setOffScreenLimit dynamically. 

In order to keep a constant length ViewPager scrolls smooth, setOffScreenLimit(page.length) will keep all the views in memory. However, this poses a problem for any animations that involves calling View.requestLayout function (e.g. any animation that involves making changes to the margin or bounds). It makes them really slow (as per Romain Guy) because the all of the views that's in memory will be invalidated as well. So I tried a few different ways to make things smooth but overriding requestLayout and other invalidate methods will cause many other problems.

A good compromise is to dynamically modify the off screen limit so that most of the scrolls between pages will be very smooth while making sure that all of the in page animations smooth by removing the views when the user. This works really well when you only have 1 or 2 views that will have to make other views off memory.

@Override
public void onPageScrollStateChanged(int state) {
  if(state==ViewPager.SCROLL_STATE_IDLE){
    if(mViewPager.getCurrentItem()==INDEX_OF_ANIMATED_VIEW){
        mViewPager.setOffscreenPageLimit(1); 
    }else{
        mViewPager.setOffscreenPageLimit(OLD_PAGE_LENGTH);
    }
  }
}