Why it becomes a problem

Three colours, five sizes, a stock status and a price band already make hundreds of combinations, and each is a page with the same products in a different cut. A crawler following links from one to the next can spend its whole visit there, and the server pays for every one. The usual sign is a shop that slows down at the moment Google visits.

A form, not links

Each value in a Siftwren set is a checkbox, a radio button or a dropdown option in a GET form. A crawler does not tick a box, so it does not walk from combination to combination on its own. The only links on a filtered page are the chips and Clear all, and they exist only once a filter is already on.

What the robots.txt lines are for

The form is the first line of defence and robots.txt the second, for addresses that reach a crawler some other way: a link from another site, a shared address, an old bookmark. These are the five patterns, and the tool applies them to addresses you paste:

Would robots.txt keep a crawler off this address?

The five lines Siftwren adds, applied to the addresses you paste. Nothing you type leaves this page.

What is left alone

Sorting is not a filter, and neither is the page number. ?orderby=price stays crawlable, because shoppers do link to it, and your plain category pages are the pages you do want crawled.

The count queries add load of their own, and we measured it: on a shop page with a five-filter set, fourteen database reads and no writes. A visit never writes to the database, so a crawler hitting filter addresses cannot fill your options table with cache entries, which is a complaint people have about other filters.