Thursday 11 April 2019

linux - Is there any limit on number of file name we can pass in tar


I am creating a tar and untar it at other location using:


$ tar -C sourcedir -cf - . | tar -C targetdir -xpf -

but it fails like this
Error I am getting is :Argument list too long - tar, when I am reducing the number of files I mean doing it in batches of files(like 1000 files at a time) it works.
Another point I noticed:
if file names are shorter than it works fine.

I am not sure what exactly is limiting this? please give suggestions.



Answer



Argument list too long is not an error specific to tar. It is an error (E2BIG) of the execve(2) syscall (given by the kernel, which has to put some limitations on execve to avoid spoiling memory). So your shell (which fork-s then execve-s the /bin/tar program) tells you that error message.


It could be difficult to increase that limit (perhaps some sysconf, I forgot the details). Or recompile your kernel and increase the ARG_MAX in its include/uapi/linux/limits.h.


GNU tar(1) accepts many interesting options (so please read the man page), in particular:



-T, --files-from=FILE


         Get names to extract or create from FILE.

So collect the file paths to be archived in some (e.g. temporary) file, then pass it with -T and you won't get any limitations.


However, the tar command has many interesting features, and you can use them to have a reasonably sized command.


See also find(1) and xargs(1). Consider also dar or afio (as altenatives to tar) or rsync


You might use GNU cp(1) as cp -va sourcedir destdir


No comments:

Post a Comment

How can I VLOOKUP in multiple Excel documents?

I am trying to VLOOKUP reference data with around 400 seperate Excel files. Is it possible to do this in a quick way rather than doing it m...