Cfrtjryk

I would like to run a series of commands against a set of files matched by a braces-and-globs pattern, without copy-pasting the pattern all over the place. I've been trying to do this by putting the pattern in a variable, but haven't been able to figure out how to make that variable work like the original pattern. How could I do that, or otherwise solve this problem?

For example, how can I run cat against files matching the pattern src/component {a,b,c}/*.c defined in the scalar variable component_source_code?

Example Context and Reproduction

set -euo pipefail;

mkdir "src/" "dist/";

trap 'rm -r "src/" "dist/"' EXIT;

I have a project whose structure resembles the following (though with more useful contents).

>"src/README.md"                 date;

mkdir "src/component a/";

>"src/component a/program.c"     date;

>"src/component a/tests.c"       date;

>"src/component a/budget$.txt"  date;

mkdir "src/component b/";

>"src/component b/program.c"     date;

>"src/component b/tests.c"       date;

>"src/component b/braces{}.txt"  date;

mkdir "src/component c/";

>"src/component c/program.c"     date;

>"src/component c/tests.c"       date;

>"src/component c/test data.txt" date;

mkdir "src/docs";

>"src/docs/test data.txt"        date;

I have build steps that need to target relevant files in multiple components.
I have defined variables with brace+glob patterns to match such sets of files.

readonly component_paths_pattern="src/component {a,b,c}";

readonly component_data_pattern="${component_paths_pattern}/*.txt";

readonly component_code_pattern="${component_paths_pattern}/*.c";

When I manually copy these patterns into an example command, they match the expected files.

>"dist/support.txt" cat src/component {a,b,c}/*.txt;

test -s "dist/all test data.txt";



>"dist/all.c" cat src/component {a,b,c}/*.c;

test -s "dist/all.c";

That would be okay if I only had to reference them once, but in reality I need to reference the same sets of files several times from different parts of the build script, hence my desire to reuse the patterns in a variable. However, I haven't been able to figure out how to make this work properly.

set -x;

Unsuccessfully Attempted Solutions

Unquoted Variable Expansion (Split+Globbing)

>"dist/support.txt" cat ${component_data_pattern};

I think this fails because the pattern contains a space, so it's split into two separate glob pattern arguments, neither of which match anything on their own.

+ cat 'src/component' '{a,b,c}/*.txt'

cat: src/component: No such file or directory

cat: {a,b,c}/*.txt: No such file or directory

Quoted Variable Expansion

>"dist/support.txt" cat "${component_data_pattern}";

I think this fails because brace expansion occurs before variable expansion, so the braces have no chance to be expanded here.

+ cat 'src/component {a,b,c}/*.txt'

cat: src/component {a,b,c}/*.txt: No such file or directory

Eval and Echo in Argument List

>"dist/support.txt" cat $(eval "echo ${component_data_pattern}");

Without quoting the subcommand expansion, I think this fails because some generated paths include spaces, causing them to be split into separate arguments.

++ eval 'echo src/component {a,b,c}/*.txt'

+++ echo 'src/component a/budget$.txt' 'src/component b/braces{}.txt' 'src/component c/test data.txt'

+ cat src/component 'a/budget$.txt' src/component 'b/braces{}.txt' src/component c/test data.txt

cat: src/component: No such file or directory

[...]

>"dist/support.txt" cat "$(eval "echo ${component_data_pattern}")";

If I do quote the subcommand expansion, I think it fails because this joins all of the paths into a single string, producing a long invalid path.

++ eval 'echo src/component {a,b,c}/*.txt'

+++ echo 'src/component a/budget$.txt' 'src/component b/braces{}.txt' 'src/component c/test data.txt'

+ cat 'src/component a/budget$.txt src/component b/braces{}.txt src/component c/test data.txt'

cat: src/component a/budget$.txt src/component b/braces{}.txt src/component c/test data.txt: No such file or directory

Eval and Printf %q in Argument List

Using printf '%q ' instead of echo fails for similar reasons.

>"dist/support.txt" cat "$(eval "printf '%q ' ${component_data_pattern}")";

++ eval 'printf '''%q ''' src/component {a,b,c}/*.txt'

+++ printf '%q ' 'src/component a/budget$.txt' 'src/component b/braces{}.txt' 'src/component c/test data.txt'

+ cat 'src/component a/budget$.txt src/component b/braces{}.txt src/component c/test data.txt '

cat: src/component a/budget$.txt src/component b/braces{}.txt src/component c/test data.txt : No such file or directory

>"dist/support.txt" cat $(eval "printf '%q ' ${component_data_pattern}");

++ eval 'printf '''%q ''' src/component {a,b,c}/*.txt'

+++ printf '%q ' 'src/component a/budget$.txt' 'src/component b/braces{}.txt' 'src/component c/test data.txt'

+ cat 'src/component' 'a/budget$.txt' 'src/component' 'b/braces{}.txt' 'src/component' 'c/test' data.txt

cat: src/component: No such file or directory

[...]

Use arrays, and don't store the filename globbing patterns in variables (let them expand into the matching pathnames instead):

component_dirs=( 'src/component '{a,b,c} )



component_data=()

component_code=()



for dir in "${component_dirs[@]}"; do

    component_data+=( "$dir"/*.txt )

    component_code+=( "$dir"/*.c   )

done

Then you could do, e.g.,

cat "${component_data[@]}"

unless that array contained many hundreds or thousands of pathnames.

Eval the Entire Command (Not Just The Arguments)

eval ">"dist/support.txt" cat ${component_data_pattern}";

test -s "dist/all.c";

I don't like this, but it works. Given that we're trying to expand a pattern that includes both braces and file globs, one of which takes place before variable expansion and one of which takes place after, there may be no alternative to something like this: manually expanding the variable into a string that includes the entire command call, and using that string as the argument to eval or bash -c. Don't forget to escape any inner quotes with ".

In the example above, there are no other arguments. If there are other arguments and those also use some kind of substitution, you'll need to escape those (with $, *, {, or }) so they're not expanded until the command is finally being evaluated and they can be interpreted in context.

readonly annoying_arg="$PWD/src/docs/test data.txt";

eval ">"dist/support.txt" cat ${component_data_pattern} "$annoying_arg"";

test -s "dist/all.c";