Ufdjrw

The awk below will create sub-directories in a directory (which is always the last line of file1, each block separated by an empty line), if the number in line 2 (always the first 6 digits in the format xx-xxxx) of file2 is found in $2 of file1. This is the current awk output.

If there is a match and a sub-directory is created in a directory then the corresponding line1 https in file2 will always be a link to a zip file for download. I can not seem to create that link in the sub-folder, download and extract the .zip. the download code executes and downloads the zip but has to manually added to the terminal. i apoogize for the long post, wanted to include all details to solve this

file1

xxx_006 19-0000_xxx-yyy-aaa
xxx_007 19-0001_zzz-bbb-ccc
FolderName_001_001

yyyy_0287 19-0v02-xxx
yyyy_0289 19-0v31-xxxx
yyyy_0293 19-0v05-xxxx
FolderName_002_002

file2

https://xx.yy.zz/path/to/file.zip
19-0v05-xxx_000_001
 cc112233
https://xx.yy.zz/path/to/download/file.zip
19-0v31-xxx-001-000
bb4456784
https://xx.yy.zz/path/to/file.zip
19-0v02-xxx_000_001
aaa331232

awk edit

cmd_fmt='mkdir -p "%s/%s"
# run the awk command
awk -v cmd_fmt="$cmd_fmt" '
# create an associative array (key/value pairs) based on the file1
NR==FNR for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next 

# retrieve the first 7-char of each line in file2 as the key to test 
 against the above hash
 k = substr($0, 1, 7) 

# if find k, then print
k in a print a[k] "t" $0 "t" l 
# save prev line to 'l' which is supposed to be the URL
 l = $0 
' RS= file1 RS='n' file2 | while IFS=$'t' read -r base_dir sub_dir link; 
do
echo "download [$link] to '$base_dir/$sub_dir'"
# bash command lines to make sub-folders and download files
 create the format text used in sprintf() to run the desired shell commands
cd "%s/%s" && curl -O -v -k -X GET %s -H "Content-Type:application/x- www-form-urlencoded" -H "Authorization:xxxx" && filename="%s"; unzip 
"$filename##*/"; '
done

desired awk output

FolderName_002_002 --- directory
 19-0v02-xxx_000_001 --- sub folder
 https://xx.yy.zz/path/to/file.zip --- zip and extracted downloaded to sub-folder
 19-0v05-xxx_000_001 --- sub-folder
 https://xx.yy.zz/path/to/file.zip --- zip and extracted downloaded to sub-folder
 19-0v31-xxx-001-000 --- sub-folder
 https://xx.yy.zz/path/to/file.zip --- zip and extracted downloaded to sub-folder

I believe your question is related to this one: Bash loop to make directory, if numerical id found in file

You can run all commands in one awk system() funcion, just organize them properly, for example:

# create the format text used in sprintf() to run the desired shell commands
cmd_fmt='mkdir -p "%s/%s" && cd "%s/%s" && curl -O -v -k -X GET %s -H "Content- Type:application/x-www-form-urlencoded" -H "Authorization:xxx" && filename="%s"; unzip "$filename##*/" && rm -f "$filename##*/"; '

# run the awk command
awk -v cmd_fmt="$cmd_fmt" '
 # create an associative array (key/value pairs) based on the file1
 NR==FNR for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next 

 # retrieve the first 7-char of each line in file2 as the key to test against the above hash
 k = substr($0, 1, 7) 

 # if find k, then run the system command 
 k in a cmd = sprintf(cmd_fmt, a[k], $0, a[k], $0, l, l); print(cmd) 

 # save prev line to 'l' which is supposed to be the URL
 l = $0 
' RS= file1 RS='n' file2

change print to system to execute the command.

Note: the above unzip and rm commands might not work if file names contains URL encoded chars.

Update based on your awk edit:

you can also just print the required info from awk line and then process them in bash, no need to do everything in awk(also remove the line to define cmd_fmt in your awk edit section):

awk '
 # create an associative array (key/value pairs) based on the file1
 NR==FNR for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next 

 # retrieve the first 7-char of each line in file2 as the key to test against the above hash
 k = substr($0, 1, 7) 

 # if find k, then print
 k in a print a[k] "t" $0 "t" l 

 # save prev line to 'l' which is supposed to be the URL
 l = $0 

' RS= file1 RS='n' file2 | while IFS=$'t' read -r base_dir sub_dir link; do
 echo "download [$link] to '$base_dir/$sub_dir'"
 # bash command lines to make sub-folders and download files
 mkdir -p "$base_dir/$sub_dir" 
 cd "$base_dir/$sub_dir"

 if curl -O -v -k -X GET "$link" -H "Content-Type:application/x-www-form-urlencoded" -H "Authorization:xxxx" >/dev/null 2>&1; then
 echo " + processing $link"
 # remove query_string from the link, since it might contains '/'
 filename="$link%?*"
 # remove path from filename and run `unzip`
 unzip "$filename##*/" 
 else
 echo " + error downloading: $link"
 fi

 # return to the base directory if it's a relative path
 # if all are absolute paths, then just comment out the following line
 cd ../..
done

Note: I did not test the curl line and dont know what the filenames could be for different links. filename="$link##*/" is to remove all chars before the last '/', which will leave filename and potential query_strings. "$filename%?*" is to remove the trailing query strings from filename. Actually filename downloaded by your curl command might be different which you will have to check and adjust from your end.