Pulling a tar backup over adb

While TWRP custom recovery offers an excellent system for backing up the system and data partitions, it does not offer the same for the user data/internal storage partition. The lack of that option is because it has to act as the target locations for backups, so it can't store a backup of itself, and it is not erased under normal circumstances.

If you need to take a backup of the userdata partition, you can use adb to pull it from the device to a PC, but the way to do this is not immediately obvious.

My first attempt was to simply use adb pull /data/media ./ to store all the files into the current directory on my PC. This works to get the files on the PC, but it does not maintain all the permissions of the files, so some will not be accessible by their corresponding apps when you restore them. It's not impossible to go and set all the permissions by hand, but it would be ideal to have all the proper permissions stored with the files.

We do have access to the tar binary from adb, so my second attempt was using adb shell 'tar -cf - /data/media' > userdata.tar. This appears to work, but the tar file is invalid when you try to extract it.

In the end, I found that I had to make 2 changes to my command to end up with this: adb exec-out 'tar -cf - /data/media 2>/dev/null' | pv > userdata.tar. Using exec-out is equivalent to using shell, but adb doesn't set up a tty, so no bytes get mangled. The 2>/dev/null is needed because adb transmits stdout and stderr over the same channel, so any error messages that tar generates (particularly "removing leading '/' from member names") will end up in the output file and it will turn up unreadable. By sending the error messages to /dev/null on the phone, only the actual data gets into the file.

As you work with this stuff, I also suggest using tar -tvf on your backup file before you need it. The t option makes tar list the files, which can be used as a quick integrity check.