Hi,

I had a couple of people mention they were getting an error trying to access a link I posted in c/fountainpens. One of the users (@gemelen@mammut.moe) created a traffic dump of trying to access the link using curl. He posted his results on this Opensuse paste bin.

He added this:

I guess Cloudflare prevents some web clients (like embedded viewer for a Mastodon mobile app) and some networks (like mine Uruguayan Internet provider) - I could open the feddit.online and see your post there in a desktop browser, but being forbidden to access from a mobile or via curl.

George

  • Jerry on PieFed@feddit.onlineMEnglish
    1·
    3 months ago

    I tried recreating the issue using Summit on an Android while logged into piefed.social and I had no issues reading all your posts or seeing all your pictures. Some things to note:

    1. The correct URL for the post is not what was used in the curl. The correct URL is https://feddit.online/c/fountainpens@lemmy.world/p/1803832/why-fountain-pens. So the posted URL was destined to fail. Where did this URL come from?

    When Cloudflare tried to determine if it was a bot request, the process stopped when a 404 was returned by the server. Hence, Cloudflare’s attempt to prove the request was not from a bot failed due to this error, so a 403 was returned by Cloudflare.

    1. Curls will never work except to prove the firewall is working. Curl requests quack like a bot, they look like a bot, and they smell like a bot and are therefore identified as a bot and not allowed.

    2. Things work when the requests come from real web browsers, and they should work when the requests come from a real phone app. Was gemelen using either?

    3. There are no restrictions on Uruguay.

    If gemelen is trying to read the post through a real web browser or a real mobile app, the request should work. If he is using a real web browser or phone app, and using the correct URL, then please give me the details so I can check on why the requests were blocked.

    • gemelen@mammut.moe
      1·
      3 months ago

      @Jerry
      To clear this out: curl _may_ work even in that particular case with a provided user-agent and other things, if necessary. But the most importantly, it demonstrated that the Cloudflare is in front of your resource.

      Originally, the URL that was shared by @Unattributed is “https://feddit.online/c/fountainpens/p/1803832/why-fountain-pens”, so it was attempted to visit from the mobile (either Tusky or Pachli app - Mastodon clients for Android, and from mobile Vivaldi).
      Thus I (and accidentally other person) replied that it’s unreachable through these methods. On the other hand, it could be opened from a desktop browser (desktop version of Vivaldi specifically).

      Where that URL came? Likely, copied from the browser’s address bar, I would assume. But that’s a question to @Unattributed

      • Jerry on PieFed@feddit.onlineMEnglish
        1·
        3 months ago

        Yes, Cloudflare has tools to detect bot traffic, which include curl and similar utilities. It absolutely stands in front of the server.

        But if the correct URL cannot be opened by a mobile client, then there’s a problem. Then I’d need the details.

        The firewall is trying to only allow traffic from either web browsers, other ActivityPub servers, or mobile applications. It’s designed to block traffic from anywhere else.

    • Unattributed 𓂃✍︎@feddit.onlineOPEnglish
      1·
      3 months ago

      I’ve just asked gemelen if he can join us to answer your questions.

      I was thinking about what you said regarding curl (originally in your deleted post): would setting the user agent have had any effect? I seem to recall having done that in the past for some specific script I wrote (but that was over 15 years ago, so my memory may be shot).

      Makes sense that if the URL was wrong in the first place, then the whole thing would fail with a 404 or 403.

      gemelen did say that when he tried it from his desktop it worked — hence the reason he suspected it was an issue between cloudflare and his mobile carrier, as he mentions in the quote above.

      • Jerry on PieFed@feddit.onlineMEnglish
        1·
        3 months ago

        No, the agent string is not enough.

        I just checked. In the past 24 hours, the firewall has dumped 98K requests from AI bots that tried really hard to look like normal traffic and not like AI bots. These bots did a lot more than just set a user agent to try to look like real users, but it didn’t help them.

        • Unattributed 𓂃✍︎@feddit.onlineOPEnglish
          1·
          3 months ago

          Not surprising… I’m sure CloudFlare is working overtime to block these ridiculous bots. I’ve seen my analytics provider is blocking tons of bots every day, even low volume sites like mine are getting hit with hundreds of bot requests.

  • Admin Jerry@feddit.onlineMEnglish
    1·
    3 months ago

    When I suddenly began receiving over 750K requests per day from AI bots, I did significantly tighten firewall rules so the server could get back online (it was throwing gateway timeouts). This issue is probably collateral damage.

    The firewall does not block any ISPs in Uruguay, so I don’t believe Uruguay is a factor.

    There are several new layers of firewall rules. One of them is blocking the request.

    The information I need to adjust the firewall may be in the paste bin post (thank you for this!). But if there is anything more you can tell me about what is trying to read the message, it would be helpful. Which phone app, for example?

    I’ll have more time to look at this tomorrow. I may have more questions. Hopefully I can reproduce the issue.

    Thanks for reporting this!