我有一个名为urls.list的 URL 列表：

https://target.com/?first=one
https://target.com/something/?first=one
http://target.com/dir/?first=summer
https://fake.com/?first=spring
https://example.com/about/?third=three
https://example.com/?third=three

并且我想根据它们的域使它们独一无二，例如https://target.com，这意味着每个域及其协议都会打印一次，并且避免使用下一个 URL。所以结果是：

https://target.com/?first=one
http://target.com/dir/?first=summer
https://fake.com/?first=spring
https://example.com/about/?third=three

这就是我试图做的：

cat urls.list | cut -d"/" -f1-3 | awk '!a[$0]++' >> host_unique.del

for urls in $(cat urls.list); do

    for hosts in $(cat host_unique.del); do
        if [[ $hosts == *"$urls"* ]]; then
            echo "$hosts"
        fi
    done
done

回答

awk -F'/' '!seen[$1,$3]++' urls.list

#!/usr/bin/env bash

declare -A uniq
mapfile -t urls < urls.list

for uniq_url in "${urls[@]}"; do
  IFS='/' read -ra url <<< "$uniq_url"
  if ((!uniq["${url[0]}","${url[2]}"]++)); then
    printf '%sn' "$uniq_url"
  fi
done